I have an application that can be trivially parallelized but its performance is to a large extent I/O bound. The application reads a single input array stored in a file that is typically 2-5 GB in size (but I expect this number to grow in the future). A typical computation applies the same operation to each row or column of that array. For CPU-heavy operations, I get very good scaling up to about 100 processors, but for slower operations I/O and the related communication (NFS access) dominate and I can't use more than a few processors efficiently.
What are efficient and portable (ideally portably efficient) options for such a situation? Parallel HDF5 seems promising. Does anyone have real-life experience with it?
Would MPI-I/O be something worth looking into? Can it work efficiently with a given file layout, or do I have to adapt everything?