0
$\begingroup$

I want to do a huge simulation that requires ~ 1 TB of data to describe a bunch of interacting particles (each has different interactions). Is it possible to store this data in an array in C++, so that I can access it during the simulation?

I have access to a 60 node cluster. Each node has 2 CPUs with 48x16GB DDR4. So that's a total of 192 GB per node, or 11520 GB = 11 TB total on the cluster. How would you dynamically allocate a 1 TB array on this cluster?

$\endgroup$
4
$\begingroup$

You could try using UPC++, which sets up a globally accessible address space distributed across your nodes.

A more standard approach would be to learn how to use MPI.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I'm pretty familiar with MPI. What function is used to distribute memory across processors? $\endgroup$ – Drew Apr 11 at 12:47
  • $\begingroup$ @Drew: There is no distribution function. You'd have to come up with a system for distributing your calculations or rotating the memory between nodes somehow. $\endgroup$ – Richard Apr 11 at 18:48
2
$\begingroup$

Really the only way to run simulations on a multi-node cluster is to use MPI. And you don't distribute data because that would mean it would start out in one place. With MPI everything is distributed all the time, including being created distributed.

It also means you don't create one global array: in your case you have 120 processes that each can store a local array. Each element of such an array has a global number, but you don't use that for indexing. If node 10 has elements 1million through 2million, then local index 27 correspond to global index 1,000,027. So you use the 27 for indexing, but 1000027 for determining what space coordinate you are in your simulation.

In other other words: you have a bunch of local arrays, but conceptually they come together into one big global array.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

There are packages for handling distributed arrays, such as Global Arrays: https://hpc.pnl.gov/globalarrays/

If that doesn't suite your needs, search for PGAS - Partitioned Global Address Space.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.