MPI data broadcast or not in C

Question

I have two slightly different but getting the same results MPI code.

The first one is from an open-source package having several data exchange steps in between:

int main ( int argc, char **argv )
{
    int i,j,nx=600,nz=300,NP, MYID;
    int idum[2];
    float v[420][720];

    for (i=0;i<420;i++){
      for (j=0;j<720;j++){
        if(i<161) { v[i][j] = 2800.0; }
        else { v[i][j] = 5200.0; }
      }
    }

    MPI_Init ( &argc, &argv );
    MPI_Comm_size ( MPI_COMM_WORLD, &NP );
    MPI_Comm_rank ( MPI_COMM_WORLD, &MYID );

    if(MYID==0){
        idum[0] = nx;
        idum[1] = nz;
    }
    MPI_Barrier(MPI_COMM_WORLD);
    MPI_Bcast(&idum,5,MPI_INT,0,MPI_COMM_WORLD);
    MPI_Bcast(&v,420*720,MPI_FLOAT,0,MPI_COMM_WORLD);
    MPI_Barrier(MPI_COMM_WORLD);

    nx = idum[0];
    nz = idum[1];

    for (i=0;i<5;i++){
      printf("id=%d,v[%d][350]=%f,\n",MYID,i*100+19,v[i*100+19][350]);
    }
    printf("nx=%d,nz=%d\n",nx,nz);
    MPI_Finalize();
    exit(0);
}

I run the code using mpirun with 4 cores. The results are:

id=0,v[19][350]=2800.000000,
id=0,v[119][350]=2800.000000,
id=0,v[219][350]=5200.000000,
id=0,v[319][350]=5200.000000,
id=0,v[419][350]=5200.000000,
nx=600,nz=300
id=0,v[19][350]=2800.000000,
id=0,v[119][350]=2800.000000,
id=0,v[219][350]=5200.000000,
id=0,v[319][350]=5200.000000,
id=0,v[419][350]=5200.000000,
nx=600,nz=300
id=0,v[19][350]=2800.000000,
id=0,v[119][350]=2800.000000,
id=0,v[219][350]=5200.000000,
id=0,v[319][350]=5200.000000,
id=0,v[419][350]=5200.000000,
nx=600,nz=300
id=0,v[19][350]=2800.000000,
id=0,v[119][350]=2800.000000,
id=0,v[219][350]=5200.000000,
id=0,v[319][350]=5200.000000,
id=0,v[419][350]=5200.000000,
nx=600,nz=300

But I think the data exchange part is a little "redundant?", so I simplify the above code as:

int main ( int argc, char **argv )
{
    int i,j,nx=600, nz=300, NP=0, MYID;
    float v[420][720];

    for (i=0;i<420;i++){
      for (j=0;j<720;j++){
        if(i<161) { v[i][j] = 2800.0; }
        else { v[i][j] = 5200.0; }
      }
    }

    MPI_Init ( &argc, &argv );
    MPI_Comm_size ( MPI_COMM_WORLD, &NP );
    MPI_Comm_rank ( MPI_COMM_WORLD, &MYID );

    for (i=0;i<5;i++){
      printf("id=%d,v[%d][350]=%f,\n",MYID,i*100+19,v[i*100+19][350]);
    }

    printf("nx=%d,nz=%d\n",nx,nz);
    MPI_Finalize();
    exit(0);
}

I get the same results as the first code.

Which one of the two codes is correct? If both are correct, which one is better? Why do we need to have the data exchange lines in the first code, or don't have to?

Answer 1

The question I think you're asking, based on your code, is:

A. Better to have each node do redundant work

B. Better to have one node do the work distribute the results.

The answer is usually A.

To see why, consider these Latency Numbers Every Programmer Should Know:

Note that CPU operations take essentially no time (~1ns) and that doing a decently complex operation on 1KB of local memory takes 3 microseconds. On the other hand, sending 1KB of data over a network takes 10 microseconds. Therefore, you're moving (at least) three times faster by doing the operation locally.

But it's even better than that. In parallel computing we call the "critical path" the longest series of serial operations we are obliged to do. In a program where we calculate then broadcast, the total time is the sum of these two operations. In this example, that's 13 microseconds.

Communication also introduces unpredictability: message transmission time can depend on message length in nonlinear ways:

Communication bandwidth is also a function of the number of processes communicating:

So the question is: do you want to delegate the speed of your computation to a slow network with unknown timing properties? Or do you want to do redundant computation locally, where you know it will be fast?

And if you compare trends in network versus memory bandwidth, the push to avoid communication is only increasing:

This is why communication avoiding algorithms are a bit of a hot topic. Where that means avoiding not just communication between network nodes, but also between RAM and CPU cache, and even different levels of the CPU cache. The speeds of various parts of computers are diverging from each other exponentially, so it is more and more important to avoid slow parts of your system as much as possible.

Answer 2

A couple of things:

• It is certainly a bit strange to start the computations before MPI_INIT. Now, the fill of the array v is performed before the MPI execution environment is initialized. At least according to MPICH, one should do as little as possible before MPI_INIT and after MPI_FINILIZE, because the MPI standard does not specify what is actually allowed.
• I would assume, that the code was supposed to:
  1. Initialize MPI
  2. Fill the array v on the master node (ID=0)
  3. Broadcast the info about the idum and v from ID=0 to other nodes in the communicator (MPI_COMM_WORLD).
  4. Print out some elements of the array on all the nodes.

Now a lot of the things go wrong here (if at least some of the assumptions I make are correct)

• I doubt that this code is actually launched (properly) using MPI because according to your output, every processor outputs his ID=0.
• If the arrays v and idum is supposed to be filled on ALL processors (not only the master one), of course, those broadcasts are useless. But then the MPI parallelization is useless.
• Your "fixed version" of the code is not practically using MPI at all.

My suggestions:

1. Fix the initialization of the MPI and move it as close to the top of the main function as possible
2. Determine, where do you want to fill array v: on all the nodes or only on the master
3. Based on that, decide on the broadcasts if necessary.
4. Make sure you are running it using proper MPI command and your ids in the output are actually not all zeroes.