Say, I have this code

double start = MPI_Wtime();
double end = MPI_Wtime();

How do I interpret the numbers I am getting. For instance, with 1 process I get something like


For two processes I get


What can I say about this? Do I take the average of these results?


That depends on your goal. For one process it's obvious that $t$ is the wall-time of the entire program launch.

For two processes, you got $t_1$ and $t_2$ which correspond for the walltime on processors 1 and 2.

Usually, one is interested in the total time it took to solve the problem - aka overall walltime, which will be $t=\max(t_1,t_2)$. However, I would recommend some additional measures to do a proper job:

Put an MPI_BARRIER before and after the computation is done. Measure the walltime only on the master node, which better be done right before and after the barrier. Then, you will obtain the accurate walltime of the ParallelMatrixVectorProductRotuine(...); for $N$ processors.

  • $\begingroup$ How do I measure the time on a single node? I will update my post to show what I have tried. $\endgroup$
    Jul 17 '18 at 17:27
  • $\begingroup$ if (my_id == master_id), where you get my_id from MPI and master_id set to $0$. MPI_COMM_RANK is the command. Take a look at this page, for example, for basic MPI commands to use. $\endgroup$
    – Anton Menshov
    Jul 17 '18 at 17:27
  • $\begingroup$ make sure to have the MPI_Barriers in this case. $\endgroup$
    – Anton Menshov
    Jul 17 '18 at 17:28
  • $\begingroup$ Ahh yes I have. Sorry about that. Very silly question. Is it unusual to go from 1.7 seconds to 1.0 seconds by increasing the processor count by 1 (from 1 to 2)? I have followed your advice exactly. $\endgroup$
    Jul 17 '18 at 17:30
  • $\begingroup$ Depends on the implementation of parallel matrix-vector product (MVP) and the size of the matrix you are dealing with. But for small matrices (say, $1000\times 1000$) and custom coded MVP such inefficiency could be expected. $\endgroup$
    – Anton Menshov
    Jul 17 '18 at 17:31

There can be any number of reasons for different times on different processes.

  1. If one process has lots of local work after the last MPI communication, it will report a longer running time.
  2. If one process concludes with an eager send, it will report a shorter running time than the other process, which has to perform the receive.

So the only thing you can say is that the processes are somehow unbalanced. What should you report as the objective overall time for the operation? Does such a concept make sense?

  • Maybe your code will go on to do local work, in which case your product seamlessly transitions into the next operation, and the concept of an overall time makes no sense.
  • Maybe the next operation is something collective like an inner product, and the processor that finished early has idle time.

If you really want to assign an objective global time, then I would use barriers, and put the timers immediately after the barriers. Even though a barrier is not a time synchronization semantically, in practice it often is.

But bear in mind that your production code will be faster if you then remove the barriers again, meaning that the concept of the time was ill-defined to begin with.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.