Questions tagged [memory-management]
For questions about properly allocating memory for data structures used in a calculation. This tag is often used when dealing with problems that require very large memory or specific constraints on memory usage.
41
questions
1
vote
1
answer
94
views
Space complexity of a semidefinite program
What is the space complexity of a semidefinite program (SDP)?
What is the answer to the same question for convex optimization problems in general?
1
vote
3
answers
180
views
Inefficient comparisons of custom data type C++
I've got some code that I need to squeeze every bit of both time and space out of. I'm looking for a better solution to the following problem.
For reasons outside of the scope of this question, I ...
- 113
1
vote
1
answer
81
views
Memory issues with iterative solvers
Was trying to implement a poisson 2d solver using Conjugate Gradient Method, so from 10x10 grid the matrix becomes 100x100 (since we have 100 nodes to find the values at), 100x100 grid goes to ...
- 354
1
vote
0
answers
161
views
Why does Eigen allocate a temporary to evaluate A.noalias() = B.transpose()*C in parallel?
I wrote a program which iteratively transforms data using matrix multiplications. To minimize the number of large memory allocations, I use two roughly equal-sized ...
- 21
1
vote
3
answers
360
views
How can we solve the normal equations with limited memory?
I was asked this open ended question in an interview once:
How would you find a solution to the normal equations with limited memory?
Unlike Solving sparse least squares system with limited memory, ...
- 143
11
votes
1
answer
5k
views
How to compute Singular value decomposition of a large matrix with Python
Language: Python3
Problem: I have a matrix Q of shape [51200 rows x 51200 cols] stored in a binary file, each of the element in this matrix has a data type of complex64. To load the data into memory I ...
- 111
2
votes
1
answer
242
views
Methods to improve the efficiency and the memory requirement of LU factorization for complex symmetric system matrix
I want to solve a linear set of equations (Ax=b) using LU decomposition. My "A" matrix is a complex matrix which is ...
- 33
2
votes
1
answer
70
views
Understading memory sharing in a GPGPU using a lattice example
I am new in the GPUs world, I used them in Matlab ambient so I didn't need to appreciate the subtleties of these devices.
I know that a GPU can be divided into multiprocessors (also called Streaming ...
- 23
2
votes
0
answers
73
views
OpenMP inconsistently segfaulting
I parallelized my code with openMP, and now have this bug in my code that's really odd. If it's the first time I run on a computer, it segfaults, but if I run the program again it runs fine. I've ...
- 2,059
0
votes
1
answer
87
views
Efficient way to store array where elements are added and removed frequently
At any point in time I have a list of N points, points will be added and removed making the number of total points change over time. If I just use an array to store this information, I'm concerned ...
- 525
0
votes
3
answers
170
views
How to store a TB size array in C++ on a cluster
I want to do a huge simulation that requires ~ 1 TB of data to describe a bunch of interacting particles (each has different interactions). Is it possible to store this data in an array in C++, so ...
- 109
0
votes
0
answers
107
views
Petsc matrix with DM objects takes too much memory
I have been learning petsc recently and made a simple solver for stokes equation on 3D staggered grid.
My code is running on server with 200GB ram, but if I run a solver on 256^3 grid, memory used by ...
4
votes
2
answers
2k
views
2
votes
0
answers
82
views
Processing time steps in chunks with Fortran [closed]
My PDE simulation program written in Fortran has to make about 2 million variable time steps. But with each time step it slows down more and more, so that if it initially makes 1000 time steps per ...
- 216
1
vote
1
answer
860
views
Doing computations on a very large numpy array: streaming the calculation vs out-of-core memory
I am trying to perform a calculation in numpy that depends on several parameters, and involved the creation of many intermediate arrays. These intermediate steps involve integrals over more parameters....
- 133
0
votes
0
answers
19
views
Cost functions to judge time/memory/accuracy tradeoffs
I am working on an interesting algorithm: Its absolute error is exponential in a parameter $j \in \mathbb{N}$, and for a given $j$, I have complete freedom to choose between an $\mathcal{O}(1)$ time-...
- 2,055
1
vote
1
answer
92
views
Minimizing the used memory in diffusion simulation using Python
I am recently dealing with a diffusion simulation project and I have come up with the following code:
...
- 27
1
vote
1
answer
258
views
Finite difference - Explicit / Implicit / Crank Nicolson - Does the implicit method require the least memory?
Examine a dynamic 2D heat equation $\dot{u} = \Delta u$ with zero boundary temperature. A standard finite difference approach is used on a rectangle using a $n\times n$ grid. For the resulting linear ...
- 11
1
vote
0
answers
242
views
Why is the method of im2col with GEMM is more efficient than the method of direction implementation with SIMD in CNN
The convolutional layers are most computationally intense parts of Convolutional neural networks (CNNs).Currently the common approach to impement convolutional layers is to expand the image into a ...
- 193
1
vote
2
answers
89
views
Access optimized data structure for representing integer lattice
Consider the integer lattice in $2d$, namely the set $\mathbb{Z}^2 = \{(x,y): x,y\in \mathbb{Z}\}$, and let $u:\mathbb{Z}^2 \to \mathbb{R} $ be a function defined on some bounded subset of $\mathbb{Z}^...
- 111
0
votes
1
answer
136
views
Use of GPU with respect to CPU
I have research work where I need to compute a matrix inversion. The matrix has a size $31300\times31300$. I am using a universal java matrix package to invert this matrix. But as the dimension of the ...
- 35
0
votes
1
answer
182
views
Compare 64gb dd3 ram with 2gb Quadro M1000M gpu on Lenovo P50
I frequently work with Keras and tensorflow-gpu on my Lenovo P50 workstation which has 64GB ram and a 2GB Nvidia QUadro M1000M GPU.
Today while training my model on GPU, it ran into memory error and ...
- 103
1
vote
1
answer
192
views
Reordering algorithm for minimization of ram usage of a skyline matrix
The stiffness matrix of $Ax=B$ system of linear equations, where $A$ is an $n\times n$ symmetric matrix stored in the form of symmetric skyline matrix, that is associated with a finite element model ...
- 13
0
votes
1
answer
287
views
Why does multithreaded code Segfault in 64-bit but not 32-bit [closed]
I have written some multithreaded code using PyOpenCL, which works fine under the following conditions:
GPU 32-bit floating point values (Intel Iris GPU can only handle single precision values)
CPU ...
- 117
1
vote
1
answer
436
views
Why does PETSc matrix memory allocation improve performance so much?
Context
In the Portable, Extensible Toolkit for Scientific Computing (PETSc), the user often creates matrices and vectors. These objects are then used as input for other routines like iterative ...
1
vote
0
answers
472
views
Memory requirement to find eigenvalues and -vectors of large sparse matrix
How can I estimate how much memory will be needed to find eigenvalues and eigenvectors of a given large sparse matrix?
I have a real symmetric matrix with roughly $5 \times 10^4$ rows and columns, ...
- 111
0
votes
1
answer
233
views
bifurcation diagram MATLAB runs out of memory
I have a 3000 x 2301 matrix 'xfinfin' in MATLAB that I want to plot in the following way:
...
- 59
2
votes
1
answer
217
views
Solving sparse linear equations with an iterative, out-of-core algorithm
Is there an iterative sparse parallel linear equation solver with out-of-core capabilities?
I need to solve a very large system of equations. I have implemented direct sparse parallel solvers in-core ...
- 139
2
votes
0
answers
87
views
How to fix time intervals to store data in a stochastic simulation (continous time markov chain)
I am using FORTRAN to implement Gillespie's stochastic simulation algorithm. I would be running many simulations in parallel (both parallel instances with different seed and parallel functions); if I ...
- 143
3
votes
0
answers
1k
views
total memory usage of MPI shared memory
I am trying to use the MPI share memory feature. I have several SMP nodes, and each of them has four cores. I need an array of size N for each node that should be accessed by all four cores in each ...
- 31
2
votes
2
answers
2k
views
How best to use scipy.integrate's ODE solvers when state is not naturally stored as a vector?
I have a large system of ODEs. For various reasons, it is natural to store the values of the dependent variables in a multidimensional array. For example, these values might represent the solution ...
- 659
3
votes
1
answer
595
views
Not enough memory for GMRES
After realizing that Gauss-Seidel is extremely slow for my simulation, i wanted to try GMRES and luckily found the C++ code here without diving into the theory. The size of the matrix in my case is <...
- 293
3
votes
1
answer
5k
views
How to change the dimensions of an Eigen Matrix in a loop?
I have a while loop, in which I use a Matrix A, vectors B and x with varying dimensions:
<...
- 719
1
vote
1
answer
133
views
How to allocate memory for successive iterative solutions with potentially different non-zero structure?
Background
I am solving the unsteady heat equation in 3D using an alternating direction implicit (ADI) method. This means that I am solving three different tridiagonal systems within a single timestep ...
- 469
3
votes
2
answers
2k
views
How Do I solve large systems given UMFPACK memory limitations?
I am trying to solve a system of equations (A x = b) for 3D heat diffusion (i.e. each equation has at most 7 terms not including the constant "b" term) using UMFPACK with boost numeric bindings to C++....
- 469
3
votes
2
answers
975
views
Memory management for solving large sparse systems with UMFPACK
I'm using umfpack for solving large systems of equation. However, I'm constantly getting out of memory issues for even modest size problems as pre2, torso3, ohne2 Hamrle3 (all from Tim Devis's ...
- 455
3
votes
1
answer
378
views
LU Decomposition with memory-mapped matrices
I have a ~4.12 Tb structured relatively-sparse matrix dataset (about 8% of the matrix entries are non-zero) that i want to apply an LU decomposition, however, given the size of it, loading it in ...
- 249
2
votes
1
answer
246
views
Memory management in time-domain simulation
I develop a parallel time-domain DAE simulation software using OpenMP and Fortran 2003. The main algorithm is:
...
- 343
1
vote
0
answers
304
views
Memory allocation error with GSL ODE solver applied to system of 4 ODEs
I am trying to solve a (large) system of ODEs with GSL solvers. When I use driver method I get an error message of could not allocate space for gsl_interp_accel, ...
- 111
18
votes
4
answers
5k
views
Portable multicore/NUMA memory allocation/initialization best practices
When memory bandwidth limited computations are performed in shared memory environments (e.g. threaded via OpenMP, Pthreads, or TBB), there is a dilemma of how to ensure that the memory is correctly ...
- 25.4k
8
votes
4
answers
381
views
(How) do you take into account memory fragmentation?
I use an example from finite element theory, but anybody who maintains a large datastructure and successively extends it will find something similar.
Suppose I have an unstructured mesh of points and ...
- 3,570