Skip to main content
58 votes

How mature is the "Julia" scientific computing language project?

Julia, at this point (May 2019, Julia v1.1 with v1.2 about to come out) is quite mature for scientific computing. The v1.0 release signified an end to yearly code breakage. With that, a lot of ...
Chris Rackauckas's user avatar
32 votes
Accepted

Why would you need frameworks like MPI when you can multi-task using threads?

There is one real and one practical reason. First, MPI was developed at a time when machines had exactly one processor core and when we wanted to couple different machines. It is today used on ...
Wolfgang Bangerth's user avatar
19 votes

Are there any embarrassingly parallel tasks that require a CPU rather than GPU?

The simple example from electromagnetics (EM) would be performing a parallel frequency sweep for a frequency-domain simulation, say, full-wave extraction of network parameters (S, Y, Z, etc) for a ...
Anton Menshov's user avatar
  • 8,702
14 votes
Accepted

Under what circumstances is parallel scaling of the finite element method not "solved"?

There are multiple questions in the post, so let me address these separately: Scaling: Every parallel program is composed of sequential and parallel tasks, and Amdahl's law then guarantees that there ...
Wolfgang Bangerth's user avatar
13 votes

How do I reliably generate random numbers in Python distributed across multiple nodes?

To the best of my knowledge, Numpy does not support independent streams. Indeed, getting independent streams from the Mersenne Twister (Pythons RNG) is notoriously difficult although it can be done. ...
LKlevin's user avatar
  • 2,503
12 votes
Accepted

Solving linear system of the form $ABx=b$

Defining the auxiliary variable $y=Bx$ yields the following algebraically equivalent expanded system, $$\underbrace{\begin{bmatrix} 0 & A \\ B & -I \end{bmatrix}}_{K} \underbrace{\begin{...
Nick Alger's user avatar
  • 3,143
12 votes

Iteration counts of AMG solver changes in parallel

This is something that can happen in almost any numerical algorithm running in parallel. It's important to know that floating-point addition is not associative due to round-off errors. Thus you can't ...
Brian Borchers's user avatar
11 votes

Parallelizing a for-loop in Python

Without assuming something special on my_function choosing multiprocessing.Pool().map() is a good guess for parallelizing such ...
Xavier Combelle's user avatar
11 votes

Are there any embarrassingly parallel tasks that require a CPU rather than GPU?

High-quality video encoding is something like this. The search space is so huge that it requires branching to prune it rapidly, but GPUs are terrible at that. Modern CPU short-vector SIMD works well ...
Peter Cordes's user avatar
9 votes
Accepted

C standard for computational science

In theory, as the original authors, you're free to pick and name a standard, then expect others to follow it. In practise, if you're supporting an HPC system, then your choice is likely to be ...
origimbo's user avatar
  • 2,259
9 votes
Accepted

How to choose a python parallelization library?

Dask schedules tasks across processes and across nodes, so it is appropriate for use on a single computer, supercomputer, or cloud. Dask also provides specialized data structures to aid in this. ...
Richard's user avatar
  • 3,981
9 votes

Why would you need frameworks like MPI when you can multi-task using threads?

Wolfgang Bangerth's answer is totally correct, and I only want to add one practical aspect. Portability across hardware Let's say you write a research code from scratch. You have a powerful multi-...
MPIchael's user avatar
  • 3,005
9 votes

Are there any embarrassingly parallel tasks that require a CPU rather than GPU?

GPUs work with the model SIMD (single instruction multiple data) i.e. they execute an instruction over multiple data. To give an idea: under CUDA technology when you have got an if-then-else ...
Mauro Vanzetto's user avatar
9 votes
Accepted

Searching for recent code source for "Parallel scientific computing in C++ and MPI "

First of all, the book you mention is very old. In fact, it misses the last two MPI standards (3 and 4), and every C++ standard from C++11 on. Secondly, know that MPI has officially no C++ bindings, ...
Victor Eijkhout's user avatar
7 votes

Speeding up a linear transform using Python

Assuming that your kernel is somewhat smooth, use low-rank approximation. Here's a naive example: ...
cfh's user avatar
  • 586
7 votes
Accepted

A good, simple book/resource on Parallel Programming in C++ for scientific computing

One of the first things that you need to understand about parallel programming is the difference between shared memory multiprocessor computer systems and distributed memory clusters. A shared ...
Brian Borchers's user avatar
7 votes

Parallel decomposition of 2D grid with minimal communication

The basic approach that I am using is weighted orthogonal recursive bisection (weighted ORB). (which is usually applied to unstructured meshes or N-body simulations; however, it is still naturally ...
Anton Menshov's user avatar
  • 8,702
7 votes

C standard for computational science

You should definitely jump to C99, or newer(!). The C99 standard introduced the restrict keyword. Loosely speaking, with this keyword you can inform the compiler ...
wim's user avatar
  • 571
6 votes
Accepted

Efficiency of parallel direct linear solver

Your problem is too small. You have to consider that to get good efficiency, each processor has to have enough work to offset the cost of communication. In other words, there is a threshold how many ...
Wolfgang Bangerth's user avatar
6 votes

Parallel integration of dynamical systems

First of all, don't even consider "optimizing" before you're using the right integration method. Chucking more computers at the problem may sound like the easiest way to solve it, but in ...
Chris Rackauckas's user avatar
6 votes

Why my parallel code using MPI is much slower than the serial one?

The first thing you need to ask yourself: is your problem big enough that the overhead of MPI messaging is less than the work that you save. Your problem size is 10k which is small, but on the other ...
Victor Eijkhout's user avatar
6 votes

Are there any embarrassingly parallel tasks that require a CPU rather than GPU?

When it comes to playing chess and other complex turn-based games using the MiniMax algorithm, then GPU acceleration is either not viable or only viable for a couple minor sub-problems. Chess engines ...
Philipp's user avatar
  • 161
6 votes

Solving PDEs in parallel

Domain decomposition was developed in the late 1990s and early 2000s because it allowed the re-use of sequential PDE solvers: You only have to write a wrapper around it that sends the computed ...
Wolfgang Bangerth's user avatar
6 votes
Accepted

Multi threaded finite element assembly implementation

It turns out that I have just the right paper for you on this subject: https://www.math.colostate.edu/~bangerth/publications/2013-pattern.pdf
Wolfgang Bangerth's user avatar
6 votes

Solve a large-scale linear system of equations with millions of unknowns

You're going to need a large cluster or a supercomputer to solve this. Memory usage is like arc_lupus commented, a double-precision float takes 8 bytes and there will be 1e-6^2 entries. We store just ...
Neil Lindquist's user avatar
6 votes

PhD in scientific computing to be a scientific programmer

I think you are asking the wrong question. You are asking "do I need a PhD for folks to hire me as a scientific programmer?" This is a kind of hypothetical. You're asking and getting ...
Richard's user avatar
  • 3,981
5 votes

How do I reliably generate random numbers in Python distributed across multiple nodes?

It is not a problem if one processor generates a number that has already appeared on a different processor. It would, however, be a problem if the two generated whole sequences that are similar. This ...
Wolfgang Bangerth's user avatar
5 votes

CPU usage when a MPI rank waits during a blocking communication

It depends on the communication settings you use for MPI. In blocking communication, MPI has three wait modes. Aggressive busy wait. This is a kind of default mode. Open MPI, at least, uses this when ...
Richard's user avatar
  • 3,981
5 votes

FEM and High Performance Computing

The correct answer to your question, IMHO, is "depends on your target and your problem at hand". 1.) If your target is to simulate a large-scale problem on HPC and if you know of an existing ...
Chenna K's user avatar
  • 964
5 votes

Parallelize Scipy iterative methods for linear equation systems(bicgstab) in Python

Take a look at the PETSc library, which has a python interface. That allows you to use MPI parallelism which scales up arbitrarily. https://www.mcs.anl.gov/petsc/ It has several variants of the ...
Victor Eijkhout's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible