6
$\begingroup$

The cluster I work on has a filesystem labelled as "fast" with no further explanation. But the filesystem type is "lustre", which strongly hints at a Lustre filesystem. While general information about this file system type is easy to find, I have not been able to find any information about how to get the most performance from it, playing with the parameters that a programmer can work on: file layout and access patterns. Does anyone have experience with this?

Since I expect replies of the kind "just run benchmarks", that's of course what I did. I find that read access is sometimes dramatically faster then the NFS filesystem on the same machine, but sometimes clearly slower, although I don't see any significant difference between the two applications that gave these extreme results. That's why I'd like to understand the performance issues rather than just have numbers.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
6
$\begingroup$

This is kind of an open question. Is the performance difference always reproducible? It's possible someone else was really beating up the filesystem during one of your tests.

Lustre has some issues with locking that can cause performance problems if you're not careful. We'd need to know a lot more about the way the filesystem hardware and software are set up in order to begin to diagnose your problem. We'd also need to know a lot more about the two applications and their I/O patterns.

Presumably you've talked to the folks that run the computer you have access to, and they haven't been too helpful. If not, you'll really want to talk to them first.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ I did try the systems administrators first, but never got a reply. Is there any way for me to figure out more about the hardware and configuration of the machine by running some diagnostic tools? $\endgroup$
    – khinsen
    Dec 15, 2011 at 16:30
  • $\begingroup$ lfs df might tell you how many Object Storage Targets there are. You can use lfs getstripe <file> to understand how many of these OSTs your file has been striped across. Other than that, you really need to talk to the folks (syadmins or HPC consultants or whatever they have) to figure out what's going on. Is the behavior reproducible? $\endgroup$
    – Bill Barth
    Dec 15, 2011 at 19:52
  • $\begingroup$ Thanks, those commands look like good starting points! I'll see how far I can get with them. $\endgroup$
    – khinsen
    Dec 16, 2011 at 15:42
4
$\begingroup$

The problem is, there are many definitions of fast. Do you need high bandwidth, low latency or both? Do you need fast write only storage for off-line processing, or do you need immediate read-back capability on another machine on the network to process that data? Do you need to access a single file or many files?

Where I work, we use lustre extensively for some experimental stations. Our cluster can happily cope with streams of image data that saturate multiple Gigabit Ethernet links, but ask it for a directory listing in a directory with a thousand files and it crawls.

Since data might be distributed over many nodes, and those nodes cache aggressively, if you schedule things correctly then you can have multiple systems cooperate on processing data and each one have faster access to the data than if they had stored it locally. Certainly we have detectors here than can generate more data per second than a local disk can sustain. For these detectors, streaming to the lustre cluster is the only realistic option.

For our set-up, it seems that lustre is optimised for high bandwidth - fast access to large contiguous datasets, but as a single meta-data server is shared amongst all lustre nodes for any given filesystem, normal file system activities like ls and grep can really take a hit. This is apparently not an issue with the most recent version of lustre, but I haven't seen that in action.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.