Hot answers tagged

6

It will not scale! Use the parallel HDF5 functionality to read it in parallel with a relatively small number of readers, or do what your colleague suggests. There will be tradeoffs with both methods, and you will have to do some tuning. It depends pretty strongly on how big the file is, how many tasks you have reading it, what the underlying filesystem is (...


4

There is no faster way to storing things than to put them into memory. If your problem is such that you can't store all of the element matrix inverses in memory, then you have two options: (i) buy more memory for the machines you use, (ii) use more machines. There is also (iii) solve smaller problems. Trying to externalize the matrices to disk will make ...


3

HDF5 appears to have some JavaScript support via its HDF Server product. You might be able to use this, though it's a web API and not a native reader. Do you need to use this on the backend (like via node.js) on the JS side, or over the web? It might make a big difference in performance.


2

Here are two examples: http://math.nist.gov/MatrixMarket/formats.html http://tedlab.mit.edu/~dr/SVDLIBC/SVD_F_DB.html I don't claim either is a good format...


2

The problem with output from parallel processes is that they all go through ssh tunnels, and there is no guarantee in what order they will arrive. Even if you use Send/Recv to sequentialize them. You could do the following: mpirun -np 8 program_script where program_script: #!/bin/bash your_program > program.out$PMPI_RANK and then for i in `seq 1 8` ;...


2

One idea would be to store the low-rank approximation of the inverse of the matrix $A^{-1}\approx UV$ using SVD. If $A\in \mathbb{R}^{30\times 30}$, then $U\in \mathbb{R}^{30\times k}\ \& \ V\in \mathcal{R}^{k\times 30} $, where $k\approx 4,5$ is the rank. Use the $UV$ approximation as a preconditioner for solving the system $Ax=b$ using GMRES. You can ...


1

It's possible to use VTK library and its parallel IO built-in mechanism to write the files from each rank to different file and ParaView could combine them again to show you the visualization. Also, I should say I don't have a FORTRAN example and unfortunately VTK doesn't support FORTRAN officially. So, I'll show you a C++ example, which I'm not sure how ...


1

If your question was "Is it wise to distribute/replicate data among $N$ processors of a big cluster using a distributed file system" I would answer fore sure not! (As already pointed out by Bill in another answer.) GPFS, Lustre, or NFS are not meant for a substitute of efficient memory/communication organisation in a distributed memory/MPI programming ...


1

I would assume that the interfaces had changed and developed within the last 5 years. As of right now, the result can be achieved using the standard functionality of scipy (as requested in the comments). Use scipy.io.mmread to load the matrix from MatrixMarket file into sparse or dense scipy matrix. I would assume that your matrix is originally sparse, ...


Only top voted, non community-wiki answers of a minimum length are eligible