I'm working on a code that we eventually plan to scale up to 10's or 100's of thousands of processors (using MPI). I have a single HDF5 file (mesh nodes and connectivity, etc) that needs to be read and distributed to every process with identical data. This is only done once during the setup part of the code. The current solution is to have each process independently read the file, but my collaborator is worried this will not scale well and suggests having process 0 read the file then MPI distribute the data to the other processors. So I wanted to get an outside opinion from someone with more extensive HPC experience. Is this actually going to be a significant bottleneck?
It will not scale! Use the parallel HDF5 functionality to read it in parallel with a relatively small number of readers, or do what your colleague suggests. There will be tradeoffs with both methods, and you will have to do some tuning. It depends pretty strongly on how big the file is, how many tasks you have reading it, what the underlying filesystem is (GPFS, Lustre, etc), how the file is striped, and what the underlying disk infrastructure is.
If your question was "Is it wise to distribute/replicate data among $N$ processors of a big cluster using a distributed file system" I would answer fore sure not! (As already pointed out by Bill in another answer.)
GPFS, Lustre, or NFS are not meant for a substitute of efficient memory/communication organisation in a distributed memory/MPI programming paradigm. But my practical answer is: why bother now? For sure the asymptotics are against re-reading the data on each processor, but if this becomes a real bottleneck can be determined only by experiment or a more thorough analysis, for which we do not have enough data from your post.
As pointed out by Stali in a comment, maybe you hit before a memory limit, or the data reading time remains negligible in your real-world application, where $N$ may be very big, but still bounded. Maybe it's the application that doesn't scale so well...
My 1 cent worth opinion: optimise only when your profiling/timing reveals that you have a real bottle neck.
Somehow, against my will, my answer has started a comment war. Just to be clear: I totally agree that it is foolish to let $N\approx 10^4 \div 10^5$ processes read the very same file at the very same time. But when the OP says "we eventually plan to scale up to 10's or 100's of thousands of processors" I understand that "eventually" means "at an unspecified later time". Before hitting these extreme numbers, there is a lot of work that can be done with more simple (albeit inefficient or even banausic) approaches. Just to put things in the right perspective: in the June 2014 top 500 list the median of the "total cores" column is 19280, so running with $N=10^5$ means that you have access to a significative portion of a really a big machine.
Tuning parallel I/O requires a lot of work, so my answers boils down to the very old advice: "avoid premature optimisation"... but here the risk of starting another war is even greater.
Final note: please, do not be afraid of down voting this post, if you think that it is pointless.