Computational Science Stack Exchange is a question and answer site for scientists using computers to solve scientific problems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have been browsing through the documentation of the IBM Platform Load Sharing Facility, to find out if it supports parallel task computing.

Below is what I think is relevant for this question, Please correct me if my perception is wrong.

Focus seems to be on the parallel job execution, where a computation has to be split up and run as different processes.

The code we have, however, is optimized for parallel task execution. Here is some pseudo C# code:

for(timeStep=0; timeStep < MaxTimeSteps; timeStep++)
    Parallel.ForEach(Cube.Cells, currentCell =>
        Compute(currentCell, Cube);

This code will calculate a time series and perform some computation on each cell in every time step. The computation needs the state of other cells in the same time step, to do the computation. Note the Parallel.ForEach instruction.

This problem cannot easily be split into separately running processes which calculate a time step each. The state of each cell of the previous time step is needed in current time step. Splitting the problem further, into a job for each cell in each time step, would require synchronization and extensive exchange of data, be it inter process or through files or database systems.

On the other hand, the Parallel.ForEach instruction could benefit extremely from the availability of many CPU's.

So, could this problem, running as a single process, benefit from a grid running LSF? Would it be transparent for the process, so that it perceives one huge system, with many CPU's and one big memory address space?

share|improve this question
up vote 5 down vote accepted

LSF definitely does not provide transparent combination of nodes into a single shared memory environment for the single process running your multi-threaded program. There are tools that can do this (like ScaleMP's product), but that's not what LSF does. LSF is responsible for queuing up many jobs which wish to run on 1 to many nodes of a cluster and then executing those jobs in some order. It is the responsibility of the jobs themselves to deal with how the nodes in that job are utilized.

This is usually done with MPI which provides a portable interface for sending messages between programs or processes which are running independently on, perhaps, different computers using a variety of interconnect networks (including shared memory). Most HPC clusters use some sort of specialized network like Infiniband for low-latency ($\sim\!1\,{\rm \mu s}$) and high-bandwidth connectivity (56Gb/s).

MPI programs are increasingly, themselves, multi-threaded inside each process using such mechanisms as OpenMP or Pthreads. Sometimes this leads to better on-node performance and better exploitation of the shared memory architecture within the node.

LSF is one of many such resource manager and scheduling systems that provide queuing functionality for Linux clusters. Others include: SLURM, Moab/Torque, and PBSPro.

share|improve this answer
Thank you for your clear statement about LSF. This is what I wanted to know. Naturally, IBM doesn't tell us what LSF can't do. – R. Schreurs Feb 11 '14 at 15:12

You haven't told us anything about how much work is involved in "Compute(currentCell,Cube)", so it's difficult to fully answer this question. Do you just need to execute a few lines of code per cell, or is it a very involved process that takes several minutes (or hours) of CPU time? What are the rough numbers of cells and time steps?

You also haven't said anything about how much data is shared in the cube and how it is accessed. Does every call to Compute require access to the entire cube or just to cells near the currentCell in a 3D grid? How much data is there per cell?

Depending on the answers to these questions, it might make sense to implement this with parallel threads on a single multiprocessor shared memory system (using e.g. OpenMP), or it might be reasonable to implement this using message passing on a distributed memory cluster (using e.g. MPI for the message passing.)

In any case, it's unlikely that LSF will be directly relevant to solving this problem. LSF is a job scheduler that works at a much higher level than would be appropriate for this. If you had (for example) a program that used MPI message passing between 100 processes that were working cooperatively, then you could use LSF to schedule a run of the program and LSF would schedule the execution of the program for a time when 100 processors with enough memory were available to run the job.

share|improve this answer
Thank you for your reply. I cannot answer your questions in detail, as I am only acting as an intermediate between the programmers and the administrator of the grid. The example was more or less hypothetical, trying to find out if running on the grid would extend task parallelism. I think the response I marked as answer, gives me enough information to go with. – R. Schreurs Feb 11 '14 at 15:26

What LSF (or any batch scheduler) can offer is the ability to schedule your job on a machine with a large number of cores, so your process can maximize its potential parallelism.

There is an integration between LSF and ScaleMP, so you can submit a job that asks for more cores than any single machine in your grid has (for example, 1024 cores), and LSF will schedule the creation of a vSMP w/ that many cores. The integration requires that you state the exact number of cores. I think in your case it might be better to ask for something like "the most cores that are available now".

In regard to optimizing parallelism of a single process, OpenMP or MPI might be what you're looking for.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.