1
$\begingroup$

I have two slightly different but getting the same results MPI code.

The first one is from an open-source package having several data exchange steps in between:

int main ( int argc, char **argv )
{
    int i,j,nx=600,nz=300,NP, MYID;
    int idum[2];
    float v[420][720];

    for (i=0;i<420;i++){
      for (j=0;j<720;j++){
        if(i<161) { v[i][j] = 2800.0; }
        else { v[i][j] = 5200.0; }
      }
    }

    MPI_Init ( &argc, &argv );
    MPI_Comm_size ( MPI_COMM_WORLD, &NP );
    MPI_Comm_rank ( MPI_COMM_WORLD, &MYID );

    if(MYID==0){
        idum[0] = nx;
        idum[1] = nz;
    }
    MPI_Barrier(MPI_COMM_WORLD);
    MPI_Bcast(&idum,5,MPI_INT,0,MPI_COMM_WORLD);
    MPI_Bcast(&v,420*720,MPI_FLOAT,0,MPI_COMM_WORLD);
    MPI_Barrier(MPI_COMM_WORLD);

    nx = idum[0];
    nz = idum[1];

    for (i=0;i<5;i++){
      printf("id=%d,v[%d][350]=%f,\n",MYID,i*100+19,v[i*100+19][350]);
    }
    printf("nx=%d,nz=%d\n",nx,nz);
    MPI_Finalize();
    exit(0);
}

I run the code using mpirun with 4 cores. The results are:

id=0,v[19][350]=2800.000000,
id=0,v[119][350]=2800.000000,
id=0,v[219][350]=5200.000000,
id=0,v[319][350]=5200.000000,
id=0,v[419][350]=5200.000000,
nx=600,nz=300
id=0,v[19][350]=2800.000000,
id=0,v[119][350]=2800.000000,
id=0,v[219][350]=5200.000000,
id=0,v[319][350]=5200.000000,
id=0,v[419][350]=5200.000000,
nx=600,nz=300
id=0,v[19][350]=2800.000000,
id=0,v[119][350]=2800.000000,
id=0,v[219][350]=5200.000000,
id=0,v[319][350]=5200.000000,
id=0,v[419][350]=5200.000000,
nx=600,nz=300
id=0,v[19][350]=2800.000000,
id=0,v[119][350]=2800.000000,
id=0,v[219][350]=5200.000000,
id=0,v[319][350]=5200.000000,
id=0,v[419][350]=5200.000000,
nx=600,nz=300

But I think the data exchange part is a little "redundant?", so I simplify the above code as:

int main ( int argc, char **argv )
{
    int i,j,nx=600, nz=300, NP=0, MYID;
    float v[420][720];

    for (i=0;i<420;i++){
      for (j=0;j<720;j++){
        if(i<161) { v[i][j] = 2800.0; }
        else { v[i][j] = 5200.0; }
      }
    }

    MPI_Init ( &argc, &argv );
    MPI_Comm_size ( MPI_COMM_WORLD, &NP );
    MPI_Comm_rank ( MPI_COMM_WORLD, &MYID );

    for (i=0;i<5;i++){
      printf("id=%d,v[%d][350]=%f,\n",MYID,i*100+19,v[i*100+19][350]);
    }

    printf("nx=%d,nz=%d\n",nx,nz);
    MPI_Finalize();
    exit(0);
}

I get the same results as the first code.

Which one of the two codes is correct? If both are correct, which one is better? Why do we need to have the data exchange lines in the first code, or don't have to?

$\endgroup$
  • $\begingroup$ Also available as stackoverflow.com/questions/51343480/… $\endgroup$ – High Performance Mark Jul 15 '18 at 17:39
  • 1
    $\begingroup$ Your idum is of length two but you're broadcasting it as length 5. Your barriers serve no function. And yes, the broadcasts themselves serve no purpose. $\endgroup$ – Victor Eijkhout Jul 15 '18 at 21:17
  • $\begingroup$ To expand on Victor’s observation, you should first fix the out of bounds memory writes before fixing MPI calls. It’s a very serious class of bugs. Try compiling with a memory sanitizer and running again. $\endgroup$ – Kirill Dec 13 '18 at 14:54
3
$\begingroup$

The question I think you're asking, based on your code, is:

A. Better to have each node do redundant work

B. Better to have one node do the work distribute the results.

The answer is usually A.

To see why, consider these Latency Numbers Every Programmer Should Know:

Latency Numbers Every Programmer Should Know]

Note that CPU operations take essentially no time (~1ns) and that doing a decently complex operation on 1KB of local memory takes 3 microseconds. On the other hand, sending 1KB of data over a network takes 10 microseconds. Therefore, you're moving (at least) three times faster by doing the operation locally.

But it's even better than that. In parallel computing we call the "critical path" the longest series of serial operations we are obliged to do. In a program where we calculate then broadcast, the total time is the sum of these two operations. In this example, that's 13 microseconds.

Communication also introduces unpredictability: message transmission time can depend on message length in nonlinear ways:

MPI packet size versus bandwidth

Communication bandwidth is also a function of the number of processes communicating:

MPI bandwidth versus number of tasks

So the question is: do you want to delegate the speed of your computation to a slow network with unknown timing properties? Or do you want to do redundant computation locally, where you know it will be fast?

And if you compare trends in network versus memory bandwidth, the push to avoid communication is only increasing:

Network versus memory bandwidth

This is why communication avoiding algorithms are a bit of a hot topic. Where that means avoiding not just communication between network nodes, but also between RAM and CPU cache, and even different levels of the CPU cache. The speeds of various parts of computers are diverging from each other exponentially, so it is more and more important to avoid slow parts of your system as much as possible.

$\endgroup$
1
$\begingroup$

A couple of things:

  • It is certainly a bit strange to start the computations before MPI_INIT. Now, the fill of the array v is performed before the MPI execution environment is initialized. At least according to MPICH, one should do as little as possible before MPI_INIT and after MPI_FINILIZE, because the MPI standard does not specify what is actually allowed.
  • I would assume, that the code was supposed to:
    1. Initialize MPI
    2. Fill the array v on the master node (ID=0)
    3. Broadcast the info about the idum and v from ID=0 to other nodes in the communicator (MPI_COMM_WORLD).
    4. Print out some elements of the array on all the nodes.

Now a lot of the things go wrong here (if at least some of the assumptions I make are correct)

  • I doubt that this code is actually launched (properly) using MPI because according to your output, every processor outputs his ID=0.
  • If the arrays v and idum is supposed to be filled on ALL processors (not only the master one), of course, those broadcasts are useless. But then the MPI parallelization is useless.
  • Your "fixed version" of the code is not practically using MPI at all.

My suggestions:

  1. Fix the initialization of the MPI and move it as close to the top of the main function as possible
  2. Determine, where do you want to fill array v: on all the nodes or only on the master
  3. Based on that, decide on the broadcasts if necessary.
  4. Make sure you are running it using proper MPI command and your ids in the output are actually not all zeroes.
$\endgroup$
  • $\begingroup$ I would contest the need to move MPI_Init to the top of main. From my experience there is no harm done if it is not one of the first functions executed - unless I/O is performed on shared resources. It must be ensured that it is executed by all processes - same as MPI_Finalize $\endgroup$ – Nox Nov 12 '18 at 19:45
  • $\begingroup$ @Nox the fact that the standard does not talk about what a program can do before initialization (see the link in the answer), inspires to move MPI_Init() as far above as possible. The fact that it might not influence the workflow, just says it is a good justified practice, not the source of error. $\endgroup$ – Anton Menshov Nov 12 '18 at 19:52

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.