I'm working on a code that we eventually plan to scale up to 10's or 100's of thousands of processors (using MPI). I have a single HDF5 file (mesh nodes and connectivity, etc) that needs to be read and distributed to every process with identical data. This is only done once during the setup part of the code. The current solution is to have each process independently read the file, but my collaborator is worried this will not scale well and suggests having process 0 read the file then MPI distribute the data to the other processors. So I wanted to get an outside opinion from someone with more extensive HPC experience. Is this actually going to be a significant bottleneck?
$\begingroup$
$\endgroup$
1
-
1$\begingroup$ Most large machines have RAM in the range of 0.5-2GB per core. If your HDF mesh file is 2GB and each core needs the exact same copy of the mesh then you're seriously limiting the size of the problem you can solve. Unless this mesh of yours is a coarse version of the problem you eventually want to solve. $\endgroup$– staliCommented Jul 8, 2014 at 16:02
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
It will not scale! Use the parallel HDF5 functionality to read it in parallel with a relatively small number of readers, or do what your colleague suggests. There will be tradeoffs with both methods, and you will have to do some tuning. It depends pretty strongly on how big the file is, how many tasks you have reading it, what the underlying filesystem is (GPFS, Lustre, etc), how the file is striped, and what the underlying disk infrastructure is.
answered Jul 8, 2014 at 14:59
$\begingroup$
$\endgroup$
3
If your question was "Is it wise to distribute/replicate data among $N$ processors of a big cluster using a distributed file system" I would answer fore sure not! (As already pointed out by Bill in another answer.)
GPFS, Lustre, or NFS are not meant for a substitute of efficient memory/communication organisation in a distributed memory/MPI programming paradigm. But my practical answer is: why bother now? For sure the asymptotics are against re-reading the data on each processor, but if this becomes a real bottleneck can be determined only by experiment or a more thorough analysis, for which we do not have enough data from your post.
As pointed out by Stali in a comment, maybe you hit before a memory limit, or the data reading time remains negligible in your real-world application, where $N$ may be very big, but still bounded. Maybe it's the application that doesn't scale so well...
My 1 cent worth opinion: optimise only when your profiling/timing reveals that you have a real bottle neck.
Edit
Somehow, against my will, my answer has started a comment war. Just to be clear: I totally agree that it is foolish to let $N\approx 10^4 \div 10^5$ processes read the very same file at the very same time. But when the OP says "we eventually plan to scale up to 10's or 100's of thousands of processors" I understand that "eventually" means "at an unspecified later time". Before hitting these extreme numbers, there is a lot of work that can be done with more simple (albeit inefficient or even banausic) approaches. Just to put things in the right perspective: in the June 2014 top 500 list the median of the "total cores" column is 19280, so running with $N=10^5$ means that you have access to a significative portion of a really a big machine.
Tuning parallel I/O requires a lot of work, so my answers boils down to the very old advice: "avoid premature optimisation"... but here the risk of starting another war is even greater.
Final note: please, do not be afraid of down voting this post, if you think that it is pointless.
-
1$\begingroup$ You might bother now because 10k MPI tasks all reading the same file can, occasionally, bring down a large parallel filesystem or cause network issues. Better to put a simple, parallel HDF5 reader in place (he's already done 90% of the work) and save getting is account cut off at a center that has a large machine. Having to wait through the diagnosis and reactivation can put a cramp in your work. $\endgroup$ Commented Jul 9, 2014 at 2:13
-
1$\begingroup$ @BillBarth: agreed, $N=10^4$ MPI task should not rely on the filesystem for getting their data. My answer was just focusing on the $N=2^0 \dots 2^9$ range, where you can just do a little experimenting, and maybe, discover that your data-model has to be changed with respect to other issues. $\endgroup$ Commented Jul 9, 2014 at 12:29
-
1$\begingroup$ I disagree. Where else in a large scale system would the data come from? If your model relies on some input data, you're going to have to read it from the filesystem, most likely. In that case, you're going to have to do something to make the reading from the filesystem efficient. $\endgroup$ Commented Jul 9, 2014 at 14:21