There is no way for anyone to tell you where the issue comes from without access to the actual code. But here are common reasons why the output (legitimately) differs for different numbers of processes:
If a code has to enumerate objects to be worked on (for example in "embarrassingly parallel" problems, but also degrees of freedom in mesh-based ...
The basic model of MPI is "two-sided communication": you have a sender who knows where to send, and a receiver who knows from where to expect something. In your description that is not the case: the sender sends to a randomly generated receiver. You could do this with one-sided communication in MPI which will be a bit of a learning curve. There the ...
MPI Q1: In a naive implementation yes, you will end up with a copy of the mesh on each process. You can reduce this somewhat by only duplicating the parts of the mesh which contain the surface, since you are only doing a surface integral and not a volume integral. This in theory will reduce how much memory is duplicated per process ($O(n^2)$ vs. $O(n^3)$). I ...
One sided communication is certainly one route. Another mechanism you could use to tackle this is use non-blocking two-sided communication, things like MPI_ISend and MPI_IRecv. If you fix some MPI process, you could set it up to have two main tasks:
Process requests from other processes asking for array data local to this process
When this process gets a ...
MPI is designed so that the same executable is started on a number of machines, none of which may be the one on which the mpirun program is called. Furthermore, MPI jobs are typically put into a queue, and the actual job may run hours or days later.
As a consequence, MPI jobs are not meant to be used in an interactive mode, but all input should be provided ...
Input from the console may work on process zero, if that runs locally, but will certainly not work on other ranks. However, I wouldn't even do interactive input on process zero.
if [ $PMI_RANK -eq 0 ] ; then
( echo 5 ; echo 1 ; echo foo ) | $*
Put that in a script and start that with mpirun:
mpirun myscript.sh myprogram
MPI_Test is a local operation. Do MPI_Iprobe instead. It can also be that your MPI allows you to control asynchronous progress with enviroment variables. Intel MPI does that, I don't know about others.
Things may also depend on your network cards.