Does IBM LSF facilitate parallel computing?

Question

I have been browsing through the documentation of the IBM Platform Load Sharing Facility, to find out if it supports parallel task computing.

Below is what I think is relevant for this question, Please correct me if my perception is wrong.

Focus seems to be on the parallel job execution, where a computation has to be split up and run as different processes.

The code we have, however, is optimized for parallel task execution. Here is some pseudo C# code:

for(timeStep=0; timeStep < MaxTimeSteps; timeStep++)
{
    Parallel.ForEach(Cube.Cells, currentCell =>
    {
        Compute(currentCell, Cube);
    }
}

This code will calculate a time series and perform some computation on each cell in every time step. The computation needs the state of other cells in the same time step, to do the computation. Note the Parallel.ForEach instruction.

This problem cannot easily be split into separately running processes which calculate a time step each. The state of each cell of the previous time step is needed in current time step. Splitting the problem further, into a job for each cell in each time step, would require synchronization and extensive exchange of data, be it inter process or through files or database systems.

On the other hand, the Parallel.ForEach instruction could benefit extremely from the availability of many CPU's.

So, could this problem, running as a single process, benefit from a grid running LSF? Would it be transparent for the process, so that it perceives one huge system, with many CPU's and one big memory address space?

Answer 1

LSF definitely does not provide transparent combination of nodes into a single shared memory environment for the single process running your multi-threaded program. There are tools that can do this (like ScaleMP's product), but that's not what LSF does. LSF is responsible for queuing up many jobs which wish to run on 1 to many nodes of a cluster and then executing those jobs in some order. It is the responsibility of the jobs themselves to deal with how the nodes in that job are utilized.

This is usually done with MPI which provides a portable interface for sending messages between programs or processes which are running independently on, perhaps, different computers using a variety of interconnect networks (including shared memory). Most HPC clusters use some sort of specialized network like Infiniband for low-latency ($\sim\!1\,{\rm \mu s}$) and high-bandwidth connectivity (56Gb/s).

MPI programs are increasingly, themselves, multi-threaded inside each process using such mechanisms as OpenMP or Pthreads. Sometimes this leads to better on-node performance and better exploitation of the shared memory architecture within the node.

LSF is one of many such resource manager and scheduling systems that provide queuing functionality for Linux clusters. Others include: SLURM, Moab/Torque, and PBSPro.

Answer 2

You haven't told us anything about how much work is involved in "Compute(currentCell,Cube)", so it's difficult to fully answer this question. Do you just need to execute a few lines of code per cell, or is it a very involved process that takes several minutes (or hours) of CPU time? What are the rough numbers of cells and time steps?

You also haven't said anything about how much data is shared in the cube and how it is accessed. Does every call to Compute require access to the entire cube or just to cells near the currentCell in a 3D grid? How much data is there per cell?

Depending on the answers to these questions, it might make sense to implement this with parallel threads on a single multiprocessor shared memory system (using e.g. OpenMP), or it might be reasonable to implement this using message passing on a distributed memory cluster (using e.g. MPI for the message passing.)

In any case, it's unlikely that LSF will be directly relevant to solving this problem. LSF is a job scheduler that works at a much higher level than would be appropriate for this. If you had (for example) a program that used MPI message passing between 100 processes that were working cooperatively, then you could use LSF to schedule a run of the program and LSF would schedule the execution of the program for a time when 100 processors with enough memory were available to run the job.

Answer 3

What LSF (or any batch scheduler) can offer is the ability to schedule your job on a machine with a large number of cores, so your process can maximize its potential parallelism.

There is an integration between LSF and ScaleMP, so you can submit a job that asks for more cores than any single machine in your grid has (for example, 1024 cores), and LSF will schedule the creation of a vSMP w/ that many cores. The integration requires that you state the exact number of cores. I think in your case it might be better to ask for something like "the most cores that are available now".

In regard to optimizing parallelism of a single process, OpenMP or MPI might be what you're looking for.