When memory bandwidth limited computations are performed in shared memory environments (e.g. threaded via OpenMP, Pthreads, or TBB), there is a dilemma of how to ensure that the memory is correctly distributed across physical memory, such that each thread mostly accesses memory on a "local" memory bus. Although the interfaces are not portable, most operating systems have ways to set thread affinity (e.g.
pthread_setaffinity_np() on many POSIX systems,
sched_setaffinity() on Linux,
SetThreadAffinityMask() on Windows). There are also libraries such as hwloc for determining the memory hierarchy, but unfortunately, most operating systems do not yet provide ways to set NUMA memory policies. Linux is a notable exception, with libnuma allowing the application to manipulate memory policy and page migration at page granularity (in mainline since 2004, thus widely available). Other operating systems expect users to observe an implicit "first touch" policy.
Working with a "first touch" policy means that the caller should create and distribute threads with whatever affinity they plan to use later when first writing to the freshly allocated memory. (Very few systems are configured such that
malloc() actually finds pages, it just promises to find them when they are actually faulted, perhaps by different threads.) This implies that allocation using
calloc() or immediately initializing memory after allocation using
memset() is harmful since it will tend to fault all the memory onto the memory bus of the core running the allocating thread, leading to worst-case memory bandwidth when the memory is accessed from multiple threads. The same applies to the C++
new operator which insists on initializing many new allocations (e.g.
std::complex). Some observations about this environment:
- Allocation can be made "thread collective", but now allocation becomes mixed into the threading model which is undesirable for libraries which may have to interact with clients using different threading models (perhaps each with their own thread pools).
- RAII is considered to be an important part of idiomatic C++, but it seems to be actively harmful for memory performance in a NUMA environment. Placement
newcan be used with memory allocated via
malloc()or routines from
libnuma, but this changes the allocation process (which I believe is necessary).
- EDIT: My earlier statement about operator
newwas incorrect, it can support multiple arguments, see Chetan's reply. I believe there is still a concern of getting libraries or STL containers to use specified affinity. Multiple fields may be packed and it may be inconvenient to ensure that, e.g., an
std::vectorreallocates with the correct context manager active.
- Each thread can allocate and fault its own private memory, but then indexing into neighboring regions is more complicated. (Consider a sparse matrix-vector product $y \gets A x$ with a row partition of the matrix and vectors; indexing the unowned part of $x$ requires a more complicated data structure when $x$ is not contiguous in virtual memory.)
Are any solutions to NUMA allocation/initialization considered idiomatic? Have I left out other critical gotchas?
(I don't mean for my C++ examples to imply an emphasis on that language, however the C++ language encodes some decisions about memory management that a language like C does not, thus there tends to be more resistance when suggesting that C++ programmers do those things differently.)