# Problems running a PETSc example in parallel

After configuring and building PETSc, I have successfully been able to run several examples. In particular, I am working with this example. I have been able to run the program using the following commands:

make ex2 mpiexec -n 4 ./ex2 -m 40 -n 40

which produces the following output:

Norm of error 0.000642883 iterations 26
Norm of error 0.000642883 iterations 26
Norm of error 0.000642883 iterations 26
Norm of error 0.000642883 iterations 26


This seems to tell me that the same problem was solved 4 times, rather than once by four processors in parallel. Suspicious, I ran the program again using

mpiexec -n 4 ./ex2 -m 40 -n 40 -log_summary

which produced the following output (note that it says that ./ex2 was run with only 1 processor:

Norm of error 0.000642883 iterations 26
************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

./ex2 on a petsc-arc named utepgeon01.utep.edu with 1 processor, by pmdelgado2 Tue Jan 24 22:16:23 2012
Using Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012

Max       Max/Min        Avg      Total
Time (sec):           6.203e-02      1.00000   6.203e-02
Objects:              4.500e+01      1.00000   4.500e+01
Flops:                3.230e+06      1.00000   3.230e+06  3.230e+06
Flops/sec:            5.207e+07      1.00000   5.207e+07  5.207e+07
Memory:               7.996e+05      1.00000              7.996e+05
MPI Messages:         0.000e+00      0.00000   0.000e+00  0.000e+00
MPI Message Lengths:  0.000e+00      0.00000   0.000e+00  0.000e+00
MPI Reductions:       4.730e+02      1.00000


I know that the computer I'm using has 4 processors (after typing grep processor /proc/cpuinfo). After checking my reconfigure-petsc-arch.py file, I see the following:

#!/usr/bin/python
if __name__ == '__main__':
import sys
import os
sys.path.insert(0, os.path.abspath('config'))
import configure
configure_options = [
'--with-cc=gcc',
'--with-fc=gfortran',
'PETSC_ARCH=petsc-arch',
]
configure.petsc_configure(configure_options)


Is there something else I need to do to distribute the work of solving the linear system in parallel?

• This sort of error frequently occurs when the MPI used to compile is not the same MPI used to execute the code. The MPI launcher ends up launching 4 independent processes in this case which do not share a communicator. What are the details of your MPI installation and the PETSc configuration options? – Aron Ahmadia Jan 25 '12 at 5:47
• @AronAhmadia: Where would I find this information? – Paul Jan 25 '12 at 6:09
• How did you install PETSc on to the machine? The information is also located in \$PETSC_DIR/\$PETSC_ARCH/conf/reconfigure.py in only slightly convoluted Python :) I usually verify that the mpirun is hooked up properly by running a simple hello world program that reports the world communicator size and process rank. – Aron Ahmadia Jan 25 '12 at 6:19
• This goes to show that there are in fact some things that petsc-maint@mcs.anl.gov is better for than stackexchange... :) – Peter Brune Jan 25 '12 at 14:46
• @Paul, petsc-maint is a closed mailing list, meaning that solutions found there are not accessible to the web. It would be valuable to scicomp if you posted an answer detailing the solution to this problem (you are allowed to answer your own question). Thanks! – Aron Ahmadia Jan 25 '12 at 19:07

After consulting the resources suggested, I found out that the computer that I was working with already had a different version of MPI installed on it, which PETSc was trying to use everytime I called the 'mpiexec' commmand. When I built PETSc, I asked it to install MPI as well. So, whenever I want to run a PETSc program in parallel, I needed to use the version that was installed with PETSc, not the one that was installed previously.

To do this, I looked in my $PETSC_DIR/$PETSC_ARCH/bin and found mpiexec in this directory. So, I could run PETSc in parallel if I called it from the command prompt as:

$PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 4 ./ex2 -m 40 -n 40 -log_summary


where $PETSC_DIR is the main directory where all the PETSc files were installed, and $PETSC_ARCH is the directory where all the PETSc architecture configuration files are installed. However, to avoid having to type so much at the command prompt, I edited my .bashrc file (in my home directory in linux) with the following alias:

alias petsc="~/petsc-3.2-p6/petsch-arch/bin/mpiexec"


(note: the ~ is there because I have it installed in my home directory) This way, anytime I want to run a petsc file in parallel I could type something like this:

petsc -n 4 ./ex2 -m 40 -n 40 -log_summary

• In addition, you can use the Makefile to run examples using the launcher corresponding to your MPI installation. The prefix for this is "run". For example, you could run make runex2 to have PETSc launch your job (though you will need to edit the Makefile to specify your arguments or to accept them as an argument from the command line). – Aron Ahmadia Jan 26 '12 at 9:18