After configuring and building PETSc, I have successfully been able to run several examples. In particular, I am working with this example. I have been able to run the program using the following commands:

make ex2 mpiexec -n 4 ./ex2 -m 40 -n 40

which produces the following output:

Norm of error 0.000642883 iterations 26
Norm of error 0.000642883 iterations 26
Norm of error 0.000642883 iterations 26
Norm of error 0.000642883 iterations 26 

This seems to tell me that the same problem was solved 4 times, rather than once by four processors in parallel. Suspicious, I ran the program again using

mpiexec -n 4 ./ex2 -m 40 -n 40 -log_summary

which produced the following output (note that it says that ./ex2 was run with only 1 processor:

Norm of error 0.000642883 iterations 26
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

./ex2 on a petsc-arc named utepgeon01.utep.edu with 1 processor, by pmdelgado2 Tue Jan 24 22:16:23 2012
Using Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 

                         Max       Max/Min        Avg      Total 
Time (sec):           6.203e-02      1.00000   6.203e-02
Objects:              4.500e+01      1.00000   4.500e+01
Flops:                3.230e+06      1.00000   3.230e+06  3.230e+06
Flops/sec:            5.207e+07      1.00000   5.207e+07  5.207e+07
Memory:               7.996e+05      1.00000              7.996e+05
MPI Messages:         0.000e+00      0.00000   0.000e+00  0.000e+00
MPI Message Lengths:  0.000e+00      0.00000   0.000e+00  0.000e+00
MPI Reductions:       4.730e+02      1.00000

I know that the computer I'm using has 4 processors (after typing grep processor /proc/cpuinfo). After checking my reconfigure-petsc-arch.py file, I see the following:

if __name__ == '__main__':
  import sys
  import os
  sys.path.insert(0, os.path.abspath('config'))
  import configure
  configure_options = [

Is there something else I need to do to distribute the work of solving the linear system in parallel?

  • 3
    $\begingroup$ This sort of error frequently occurs when the MPI used to compile is not the same MPI used to execute the code. The MPI launcher ends up launching 4 independent processes in this case which do not share a communicator. What are the details of your MPI installation and the PETSc configuration options? $\endgroup$ Commented Jan 25, 2012 at 5:47
  • $\begingroup$ @AronAhmadia: Where would I find this information? $\endgroup$
    – Paul
    Commented Jan 25, 2012 at 6:09
  • 4
    $\begingroup$ How did you install PETSc on to the machine? The information is also located in \$PETSC_DIR/\$PETSC_ARCH/conf/reconfigure.py in only slightly convoluted Python :) I usually verify that the mpirun is hooked up properly by running a simple hello world program that reports the world communicator size and process rank. $\endgroup$ Commented Jan 25, 2012 at 6:19
  • 2
    $\begingroup$ This goes to show that there are in fact some things that [email protected] is better for than stackexchange... :) $\endgroup$ Commented Jan 25, 2012 at 14:46
  • 1
    $\begingroup$ @Paul, petsc-maint is a closed mailing list, meaning that solutions found there are not accessible to the web. It would be valuable to scicomp if you posted an answer detailing the solution to this problem (you are allowed to answer your own question). Thanks! $\endgroup$ Commented Jan 25, 2012 at 19:07

1 Answer 1


After consulting the resources suggested, I found out that the computer that I was working with already had a different version of MPI installed on it, which PETSc was trying to use everytime I called the 'mpiexec' commmand. When I built PETSc, I asked it to install MPI as well. So, whenever I want to run a PETSc program in parallel, I needed to use the version that was installed with PETSc, not the one that was installed previously.

To do this, I looked in my $PETSC_DIR/$PETSC_ARCH/bin and found mpiexec in this directory. So, I could run PETSc in parallel if I called it from the command prompt as:

$PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 4 ./ex2 -m 40 -n 40 -log_summary

where $PETSC_DIR is the main directory where all the PETSc files were installed, and $PETSC_ARCH is the directory where all the PETSc architecture configuration files are installed. However, to avoid having to type so much at the command prompt, I edited my .bashrc file (in my home directory in linux) with the following alias:

alias petsc="~/petsc-3.2-p6/petsch-arch/bin/mpiexec"  

(note: the ~ is there because I have it installed in my home directory) This way, anytime I want to run a petsc file in parallel I could type something like this:

petsc -n 4 ./ex2 -m 40 -n 40 -log_summary
  • $\begingroup$ In addition, you can use the Makefile to run examples using the launcher corresponding to your MPI installation. The prefix for this is "run". For example, you could run make runex2 to have PETSc launch your job (though you will need to edit the Makefile to specify your arguments or to accept them as an argument from the command line). $\endgroup$ Commented Jan 26, 2012 at 9:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.