21
$\begingroup$

I have an application that can be trivially parallelized but its performance is to a large extent I/O bound. The application reads a single input array stored in a file that is typically 2-5 GB in size (but I expect this number to grow in the future). A typical computation applies the same operation to each row or column of that array. For CPU-heavy operations, I get very good scaling up to about 100 processors, but for slower operations I/O and the related communication (NFS access) dominate and I can't use more than a few processors efficiently.

What are efficient and portable (ideally portably efficient) options for such a situation? Parallel HDF5 seems promising. Does anyone have real-life experience with it?

Would MPI-I/O be something worth looking into? Can it work efficiently with a given file layout, or do I have to adapt everything?

Improve this question
$\endgroup$
6
  • 4
    $\begingroup$ Great question. We have the same problem, and our crude solution is to write/read the domain decomposed array to/from N files, for N processors. I don't really like this, but it's simple. I'd be interested in seeing answers that also address the complexity of various library interfaces.... $\endgroup$
    – Yann
    Commented Dec 3, 2011 at 12:46
  • $\begingroup$ How are you distributing the array across the processors? What are you using for parallelism now? Are you writing to files over NFS as a form of communication? $\endgroup$
    – Dan
    Commented Dec 3, 2011 at 13:27
  • 2
    $\begingroup$ You might not have to rework your code very much; I had a problem like this once and was able to get a better speedup avoiding IO than optimizing it. $\endgroup$
    – Dan
    Commented Dec 3, 2011 at 13:30
  • 1
    $\begingroup$ Are you using a queue system like PBS or Torque? If so, there are commands to "stage in" a file to some directory when a job starts. I don't know if it would speed things up noticeably, but it might be worth a shot. $\endgroup$
    – Dan
    Commented Dec 13, 2011 at 0:18
  • 1
    $\begingroup$ @Dan: yes, I use PBS, and I can use it to put my file wherever I want. But since my cluster doesn't have node-local disks, there is nothing better than a shared NFS volume. $\endgroup$
    – khinsen
    Commented Dec 14, 2011 at 10:17

3 Answers 3

Reset to default
7
$\begingroup$

Parallel I/O can help you in this case, but if you're using (inherently pretty serial) NFS to serve your files, then it's not going to have the full effect you might want - there's going to be a serial bottleneck at the fileserver and having hundreds of processes making requests of the single server is not going to give you factors of hundreds speedup of doing it through a single process. Still, it could help to a point, especially since it sounds like the bottleneck is reading rather than writing, and it will be a big improvement if your system gets upgraded to a fully parallel filesystem.

MPI-IO is very low-level; it's worth understanding something about it to know what's going on "under the hood" with parallel HDF5, NetCDF4, or ADIOS, but using it yourself is really only well suited for raw binary data where the structure is well known at compile time. HDF5 and NetCDF4 are much more flexible.

Note that if your data is relatively simple -- eg, the big data structures are mainly n-dimensional arrays or vectors -- I recommend NetCDF4 (which is also parallel, and based on HDF5) rather than HDF5; it's siginificantly simpler to use. HDF5 is more complicated, and in exchange for that complexity allows very complicated data models. But if that's a feature you don't need, it's faster to get started in NetCDF4.

At our centre we have an afternoon-long and a day-long class on parallel I/O where we talk about the basic concepts, MPI-IO, HDF5, and NetCDF4; the slides can be found here.

Improve this answer
$\endgroup$
0
5
$\begingroup$

We get good scaling up to the entire XT6 at ORNL using MPI/IO to output vectors. Here is the code. The I/O subsystems for many machines are not designed for massive parallelism, so I think @Dan is correct that I would try to minimize IO by only writing every few steps, or some other agglomeration strategy.

As far as flexibly writing output in a scalable way, I have experience with XDMF, which can be effected by large parallel binary writes using HDF5 (like PETSc VecView) coupled with a small amount of XML code written in serial to describe the layout. This can be read by visualization packages like Paraview or MayaVi2. Another way to do this is use the VTK format with appended binary data, however this requires that you know everything you want to write up front.

Improve this answer
$\endgroup$
2
  • $\begingroup$ XDMF looks interesting, but it is about organizing data rather than about efficiently accessing what XDMF calls "heavy" data. What do you use for that aspect? $\endgroup$
    – khinsen
    Commented Dec 5, 2011 at 9:13
  • $\begingroup$ We just use the XDMF to point into HDF5. That way you can write all binary HDF5, but have it read by most visualization engines. $\endgroup$
    – Matt Knepley
    Commented Dec 5, 2011 at 16:38
1
$\begingroup$

I assume your scalability issue is related to output, and not to input. Parallel input is rather simple - what I do is each CPU opens the input NetCDF file and reads the part of the array that belongs to its tile (there might be a limit to how many readers can open the same NetCDF file but I am not sure). Parallel output is more problematic.

What I am doing currently is not quite optimal, but works for now. I gather the whole thing on one CPU and do the serial output. In the meantime, other players wait for the writer to finish. This worked well for me because I managed to keep the computation over output ratio quite high - so scalability would be good for well over 200 CPUs. But this is not the solution you are looking for.

Another solution is what Yann suggested - write serially to N files and have a drone CPU assemble the tiles to one piece - if it is RAM permitting.

Apart from parallel I/O libraries suggested in previous answers, you also may want to look into Parallel NetCDF http://trac.mcs.anl.gov/projects/parallel-netcdf, since you are already comfortable with NetCDF and MPI. I didn't use it in practice, but do plan to go in that direction when I hit the wall with gathering + serial I/O.

Improve this answer
$\endgroup$
3
  • $\begingroup$ It's input that creates my scalability issue. I suppose that all the incoming requests from many nodes overload the NFS server, but I have no idea how to verify this hypothesis. $\endgroup$
    – khinsen
    Commented Dec 14, 2011 at 10:20
  • $\begingroup$ @khinsen What you could do to test your hypothesis is read the file with a small number of CPUs, say between 1 and 8, and scatter the data to the rest. Do the profiling, see how much time you spend on I/O and how much on scatter. Vary the number of CPU readers and see what gives you best performance. $\endgroup$
    – milancurcic
    Commented Dec 14, 2011 at 22:42
  • $\begingroup$ Good suggestion! This will be some work because it means rewriting the code, but it's probably worth it. $\endgroup$
    – khinsen
    Commented Dec 20, 2011 at 9:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.