# total memory usage of MPI shared memory

I am trying to use the MPI share memory feature. I have several SMP nodes, and each of them has four cores. I need an array of size N for each node that should be accessed by all four cores in each node. My plan is to construct a shared window of size N/4 using MPI_Win_allocate_shared, and I expect that memory usage of each node would be N. In the example below, N is 4X10^9 bytes, but the memory usage of each node is not 4GB but 16GB. Am I missing something?

#include <iostream>
#include <mpi.h>

int main(int argc, char** argv) {
MPI_Init(&argc, &argv);

int rank_all;
int rank_sm;
int size_sm;

// all communicator
MPI_Comm comm_sm;
MPI_Comm_rank(MPI_COMM_WORLD, &rank_all);

// shared memory communicator
MPI_Comm_split_type(MPI_COMM_WORLD, MPI_COMM_TYPE_SHARED, 0, MPI_INFO_NULL, &comm_sm);
MPI_Comm_rank(comm_sm, &rank_sm);
MPI_Comm_size(comm_sm, &size_sm);

std::size_t local_window_count(1000000000);

char* base_ptr;
MPI_Win win_sm;
int disp_unit(sizeof(char));
MPI_Win_allocate_shared(local_window_count * disp_unit, disp_unit, MPI_INFO_NULL, comm_sm, &base_ptr, &win_sm);

// write
char buffer;
if (rank_sm == 0) {
buffer = 'A';
}
else if (rank_sm == 1) {
buffer = 'C';
}
else if (rank_sm == 2) {
buffer = 'G';
}
else {
buffer = 'T';
}

MPI_Win_fence(0, win_sm);

for (std::size_t it = 0; it < local_window_count; it++) {
base_ptr[it] = buffer;
}

MPI_Win_fence(0, win_sm);

long long int index_start(-1 * rank_sm * local_window_count);
long long int index_end((size_sm - rank_sm) * local_window_count - 1);

for (long long int it_rel = index_start; it_rel < index_end; it_rel++) {
buffer = base_ptr[it_rel];

if (it_rel == index_start) {
std::cout << rank_sm << " start: " << buffer << std::endl;
}
else if (it_rel == (index_end - 1)) {
std::cout << rank_sm << " end: " << buffer << std::endl;
}
}

MPI_Finalize();

return 0;
}

• How are you measuring memory usage? It's possible you are being deceived by how Linux counts virtual memory in the context of shared memory. – Jeff May 12 '15 at 2:48
• The values were reported by Torque. Torque reports virtual memory usage and physical memory usage separately. – user4147776 May 12 '15 at 17:38
• Either your MPI implementation is buggy or Torque is over-counting the VM usage for some reason. I don't know enough about Torque to know how likely this is, but it seems rather unlikely that an MPICH implementation is buggy in this case. – Jeff May 13 '15 at 13:50
• I see. I will test it with the latest version of OpenMPI. Can you recommend a decent way of measuring memory usage? – user4147776 May 13 '15 at 18:41
• Not for shared memory allocation. For glibc heap, yeah, but that's not what will be used here. I know how to instrument MPICH though. You should try it and see if behavior changes, as validation of the OpenMPI implementation. – Jeff May 13 '15 at 23:31