# How to simulate over 1 billion particles?

I want to simulate human erythrocytes in capillaries. I calculated, that for a 1 meter long and 1 mm in diameter capillary there are about 3 billion blood cells.

Erythrocytes are actually discs, but let’s assume that it’s a simple cuboid. What do we have

Erythrocyte -> vertex[8] -> double x

double y

double z


In C++ double weights 8 bytes. 1 vertex=24 bytes, 8 vertexes = 24 bytes = data size of an erythrocyte. 24*7 billions bytes = 168 GB. I have only 4 Gb RAM, unfortunately. What should I do? I have never been up in computing.

• Try to decrease calculations?

Actually, there is a wide distribution of vessels diameter, for example, the largest is the aorta (as far as I know) - few cm, and the shortest could have diameter about single erythrocyte size ~ 10um, by the way, 10um it’s only 100th part of mm, so there is no sense to decreasing vessel diameter.

Decrease vessel’s width? Okay, let it be 0.3 m = 30 cm. Recalculate:

Assume vessel is a cylinder, with diameter=0.001 m and height=0.3 m. The volume equals $$2.35619449\cdot 10^{11} \mu m^3$$. Average erythrocyte volume $$90 \mu m^3$$.

We have 2.6 billion erythrocytes and 2.6 billion * 24 bytes of data. I still need some additional power computing solutions. Don’t I?

• Do you think it's reasonable to simulate a "1 meter" long capillary?! What's different along this "1 meter"? If it's just really a giant pipe that nothing will be changed along that "1 meter", you could just ignore that and simulate it in a pipe with 10 mm long and 1 mm in diameter. To step into exascale (billion scale), 4GB or even 168GB RAM are not relevant at all. If somebody wants to do this exascale simulation, he or she needs a really scalable parallel code with a powerful supercomputer. Jan 16 '20 at 17:20
• @AloneProgrammer I know understand that you answer is to solve my problem "not directly" - by decreasing calculations. Okay, I edit my question, even with calculation decreasing it’s still too much for even powerful pc Jan 16 '20 at 20:26
• What @AloneProgrammer is (correctly) saying is that brute force does not always yields more insight. In the smallest vessels, flow is laminar and there is no point in large-scale computations. In the largest vessels, flow is turbulent, and there large-scale computations make sense. Jan 16 '20 at 21:40
• "it’s only 100th part of mm, so there is no sense to decreasing vessel diameter." Even decreasing the dimension of computational domain by just 1 part, still it worth to do it cause exascale computing is really costly these days. You are trying to simulate two extremely different length scales (your tube in the macroscopic world but red blood cells in the microscopic world) which doesn't seem reasonable. At least, as a beginner, it doesn't seem reasonable to start with something in exascale. Have you looked at this: bit.ly/3adph3f Jan 16 '20 at 22:22
• This is the reason that people use continuum models. Do some reading and see what models people in your field are using then try to simulate those Jan 19 '20 at 18:48

A first step, if you "have never been up in computing", is to read the literature and see what others are doing and have done.

The second step is that you will likely learn that what you want to do is not possible today -- at least unless you have access to supercomputers. I suspect that 3 billion particles is possible today, but only if you have access to a few thousand processor cores at the same time. And they will only let you use those after you have demonstrated experience in programming and using 1, then 10, then 100 processor cores at a time.

So start reading and learning/practicing parallel computing!

• Well, I actually know that IBM provides access to their supercomputer. If it is not too expensive, I pay, though. Do You know about prices? Jan 16 '20 at 20:22
• To within a factor of 3 or so, a CPU hour costs 10 US cents. So to run computations on 1,000 cores, you'll have to pony up about \$100 per hour. You will likely need dozens of hours even if you have a fully tested and optimized code (which I take you neither have, nor have the experience to write). So in the best of cases, you'll need many thousand dollars. Jan 16 '20 at 21:35
• And bear in mind that many public computing centers are subsidized, so they must show their funders that they're producing Science. That means you'll have to be able to convincingly demonstrate that your code scales well and that you need access to large amounts of resources. Jan 16 '20 at 22:33

From a technical standpoint, your simulation is possible, if highly impractical. The operating system uses virtual memory to abstract resource allocation away from calling processes, but the good thing about that is that your 4Gb of RAM can seem infinite to your simulation program because the operating system will simply use disk space for all the RAM memory requests that don't fit in the active page table. If you're using linux, you can probably see your current system limits configuration using cat /proc/self/limits. Some systems set a default maximum, in which case calling malloc will fail with ENOMEM, but you can configure this on your own computer to whatever you want, including removing the limit altogether.

That being said, disk access is millions of times slower than reading from RAM, even if we assume the unrealistic, best-case scenario where you're mostly doing large sequential reads, which is not the point of random access memory at all.

I think the first thing you should figure out is what exactly you're trying to model about these particles. If you're trying to explore the physics of their interaction with the environment, do you really need to model all of them at once? The biggest hurdle in these kinds of simulations is the inter-dependence of state variables. After all, you're trying to model interactions, otherwise you could simply parallelize this on a supercomputer. If you were to reduce the number of particles being simulated at any one time, you could use put those resources towards the computation of the simulation data. Modeling viscosity, flow rate, etc., takes a ton of computational resources, and the more resources you have at your disposal, the more realistically you can model the interactions between the particles themselves.

I interned in a computational physics lab doing solid-state simulations, and the biggest hurdle in those projects was the amount of interaction between each molecule. The computational complexity of the computations involved could reach n!, so if you're plan is to start out by modeling every blood cell in the human body from the beginning, my advice would be to either design the model first, prototype the necessary algorithms, run small simulations with a low iteration count, repeat as necessary while you fine-tune the model, and then gradually increase either the scope or the depth (but not both) of the investigation, or cryogenically freeze yourself until a Kardashev tier 2 civilization shows up with the tech you need.

• Suggesting that anything useful can be done if you need to swap memory is just plain silly. Swapping is useful if there are parts of the operating data of a machine that are not used for long periods -- think about having multiple programs open, but only one being interactive at any given time. But if you can't fit the data of a single program into memory, and you do loops over this data as you would for particle algorithms, you have to swap out data all the time. Your program will not terminate in any reasonable time frame. It's silly to suggest that anything useful can be done this way. Jan 21 '20 at 22:00
• My intent was to point out the computational infeasibility of the simulation as described, so if the impression upon reading is the opposite, something went horribly wrong Jan 22 '20 at 14:23
• Well, it's really the first sentence that suggests that such simulations are "possible, if highly impractical". I would argue that if you need to swap data in and out all the time, then this is so much slower that you're just not going to ever crank through all of your particles sufficiently quickly to do anything useful. I mean, think about the fact that just reading 1 GB of data from disk takes on the order of seconds. Writing it the same amount. So you'll end up with on the order of a hours for the 168 GB of data quoted in the question. You can't do time stepping that way. Jan 23 '20 at 0:20
• I can see how my introduction might inadvertently convey a sense of false hope, and I agree; the amount of disk swapping required would be prohibitive Jan 24 '20 at 15:56