59 votes

How mature is the "Julia" scientific computing language project?

Julia, at this point (May 2019, Julia v1.1 with v1.2 about to come out) is quite mature for scientific computing. The v1.0 release signified an end to yearly code breakage. With that, a lot of ...
Chris Rackauckas's user avatar
32 votes
Accepted

Why would you need frameworks like MPI when you can multi-task using threads?

There is one real and one practical reason. First, MPI was developed at a time when machines had exactly one processor core and when we wanted to couple different machines. It is today used on ...
Wolfgang Bangerth's user avatar
24 votes

Parallelizing a for-loop in Python

Joblib does what you want. The basic usage pattern is: ...
Daniel Mahler's user avatar
19 votes

Are there any embarrassingly parallel tasks that require a CPU rather than GPU?

The simple example from electromagnetics (EM) would be performing a parallel frequency sweep for a frequency-domain simulation, say, full-wave extraction of network parameters (S, Y, Z, etc) for a ...
Anton Menshov's user avatar
  • 8,652
16 votes

Parallelizing a for-loop in Python

What you're looking for is Numba, which can auto parallelize a for loop. From their documentation ...
LKlevin's user avatar
  • 2,493
13 votes

How do I reliably generate random numbers in Python distributed across multiple nodes?

To the best of my knowledge, Numpy does not support independent streams. Indeed, getting independent streams from the Mersenne Twister (Pythons RNG) is notoriously difficult although it can be done. ...
LKlevin's user avatar
  • 2,493
13 votes
Accepted

Under what circumstances is parallel scaling of the finite element method not "solved"?

There are multiple questions in the post, so let me address these separately: Scaling: Every parallel program is composed of sequential and parallel tasks, and Amdahl's law then guarantees that there ...
Wolfgang Bangerth's user avatar
12 votes
Accepted

What is "good" parallel scaling?

Good is a relative term, and it will depend on the nature of the problem, the nature of the algorithm, and properties of the hardware involved. The only absolute reference point is ideal scaling (100%...
David Ketcheson's user avatar
12 votes
Accepted

Solving linear system of the form $ABx=b$

Defining the auxiliary variable $y=Bx$ yields the following algebraically equivalent expanded system, $$\underbrace{\begin{bmatrix} 0 & A \\ B & -I \end{bmatrix}}_{K} \underbrace{\begin{...
Nick Alger's user avatar
  • 3,143
12 votes

Iteration counts of AMG solver changes in parallel

This is something that can happen in almost any numerical algorithm running in parallel. It's important to know that floating-point addition is not associative due to round-off errors. Thus you can't ...
Brian Borchers's user avatar
11 votes

Parallelizing a for-loop in Python

Without assuming something special on my_function choosing multiprocessing.Pool().map() is a good guess for parallelizing such ...
Xavier Combelle's user avatar
11 votes
Accepted

What is the state of the art algorithm for diagonalizing real symmetric matrices?

The QR/Francis algorithm is the go-to choice for dense eigenproblems, but there are a few competitors around: The Jacobi algorithm (like QR, another algorithm with an unfortunate name, which can be ...
Federico Poloni's user avatar
11 votes

Are there any embarrassingly parallel tasks that require a CPU rather than GPU?

High-quality video encoding is something like this. The search space is so huge that it requires branching to prune it rapidly, but GPUs are terrible at that. Modern CPU short-vector SIMD works well ...
Peter Cordes's user avatar
10 votes
Accepted

Parallel Mersenne Twister for Monte Carlo

Like you say, using the Mersenne Twister for parallel computations is almost always done incorrectly, as the correct method is tricky to implement. By far the easiest and best answer would be to move ...
LKlevin's user avatar
  • 2,493
9 votes
Accepted

C standard for computational science

In theory, as the original authors, you're free to pick and name a standard, then expect others to follow it. In practise, if you're supporting an HPC system, then your choice is likely to be ...
origimbo's user avatar
  • 2,239
9 votes
Accepted

How to choose a python parallelization library?

Dask schedules tasks across processes and across nodes, so it is appropriate for use on a single computer, supercomputer, or cloud. Dask also provides specialized data structures to aid in this. ...
Richard's user avatar
  • 3,961
9 votes

Why would you need frameworks like MPI when you can multi-task using threads?

Wolfgang Bangerth's answer is totally correct, and I only want to add one practical aspect. Portability across hardware Let's say you write a research code from scratch. You have a powerful multi-...
MPIchael's user avatar
  • 2,842
9 votes

Are there any embarrassingly parallel tasks that require a CPU rather than GPU?

GPUs work with the model SIMD (single instruction multiple data) i.e. they execute an instruction over multiple data. To give an idea: under CUDA technology when you have got an if-then-else ...
Mauro Vanzetto's user avatar
9 votes
Accepted

Searching for recent code source for "Parallel scientific computing in C++ and MPI "

First of all, the book you mention is very old. In fact, it misses the last two MPI standards (3 and 4), and every C++ standard from C++11 on. Secondly, know that MPI has officially no C++ bindings, ...
Victor Eijkhout's user avatar
8 votes

How mature is the "Julia" scientific computing language project?

From my experience Julia isn't ready for (scientific) everyday use yet (I'm talking about the stabilized version 0.4 of march 2016). The language itself is nice, feature rich and consistent; a ...
johannes_lalala's user avatar
8 votes
Accepted

Intel Knights Landing work loads vs NVIDIA GeForce

Like Brian said, the Xeon Phi cores are not at all comparable to the CUDA ones. The problem with the Phi is that it's somewhere between two horses. If you are doing highly parallel floating point ...
LKlevin's user avatar
  • 2,493
7 votes

Intel Knights Landing work loads vs NVIDIA GeForce

CUDA cores aren't at all comparable to the separate processor cores in the Xeon Phi coprocessors. The Phi coprocessor cores are full fledged processors that can have their own loops, branching, etc. ...
Brian Borchers's user avatar
7 votes

Linear Systems with Multiple Right Hand sides

It's possible that your performance is limited by the memory bandwidth of your system. It's not at all uncommon to have a situation where two or three of your cores can use all of the available ...
Brian Borchers's user avatar
7 votes

Speeding up a linear transform using Python

Assuming that your kernel is somewhat smooth, use low-rank approximation. Here's a naive example: ...
cfh's user avatar
  • 586
7 votes
Accepted

A good, simple book/resource on Parallel Programming in C++ for scientific computing

One of the first things that you need to understand about parallel programming is the difference between shared memory multiprocessor computer systems and distributed memory clusters. A shared ...
Brian Borchers's user avatar
6 votes

MPI+OpenMP Scalability

No. You need to test it to that scale, especially if you have MPI calls within OpenMP regions.
Bill Barth's user avatar
  • 10.9k
6 votes
Accepted

Efficiency of parallel direct linear solver

Your problem is too small. You have to consider that to get good efficiency, each processor has to have enough work to offset the cost of communication. In other words, there is a threshold how many ...
Wolfgang Bangerth's user avatar
6 votes

Parallel integration of dynamical systems

First of all, don't even consider "optimizing" before you're using the right integration method. Chucking more computers at the problem may sound like the easiest way to solve it, but in ...
Chris Rackauckas's user avatar
6 votes

Parallel decomposition of 2D grid with minimal communication

The basic approach that I am using is weighted orthogonal recursive bisection (weighted ORB). (which is usually applied to unstructured meshes or N-body simulations; however, it is still naturally ...
Anton Menshov's user avatar
  • 8,652
6 votes

C standard for computational science

You should definitely jump to C99, or newer(!). The C99 standard introduced the restrict keyword. Loosely speaking, with this keyword you can inform the compiler ...
wim's user avatar
  • 561

Only top scored, non community-wiki answers of a minimum length are eligible