# How to debug segmentation faults in large problems?

I am sorry if this question seems like off topic or opinion based, but I was not sure how to go about it.

I am currently working on a 100k x 100k positive definite linear system and trying to solve it using Cholesky factorization from Scalapack 2.1.0 (pdpotrf). My problem is that when I run the same system for 50kx50k matrix (i.e. A(1:50000,1:50000)) my program runs fine. However it is giving segmentation fault in 100k run:

Backtrace for this error:
#0  0x7f8126942b9a in ???
#1  0x7f8126941dc3 in ???
#2  0x7f8125e4824f in ???
#3  0x45409d in ???
#4  0x44382c in pdpotf2_
at /home/user/sclpk_solver/scalapack-2.1.0/SRC/pdpotf2.f:250
#5  0x414502 in pdpotrf_
at /home/user/sclpk_solver/scalapack-2.1.0/SRC/pdpotrf.f:262


Normally I would like to use record-replay or such debugging tools to go about it, but given the size of the problem and the fact that it takes it about 10 hours before I get the failure, I am not sure how to even start debugging it. The exact same code and exact same data works fine on intel MKL(ver 2018 service pack 3). But GNU compilers are giving issue (version 4.8 and 7.1).

UPDATE: I just figured out an important detail: my program works for cores greater then 4. its <=4 when it gives the segmentation fault. I think its because for 100k, 4 or less cores mean each core will have > 50k*50k elements. so there is somewhere integer overflow error. It can be strengthened by the fact that when compiled with intel MKL, it needs -i8 flag for numroc function to work properly.

Also I tried compiling scalapack with "-g -Og" flag, and this time program failed with following output:

{*****,    0}:  On entry to PDPOTRF parameter number    1 had an illegal value
<< Warning >> pdpotrf FAILED      -1
{*****,    0}:  On entry to PDPOTRF parameter number    1 had an illegal value
<< Warning >> pdpotrf FAILED      -1
{    0,    0}:  On entry to PDPOTRS parameter number    1 had an illegal value
<< Warning >> pdpotrs FAILED
��������
Solved! predicting test values
{    1,    0}:  On entry to PDPOTRS parameter number    1 had an illegal value
<< Warning >> pdpotrs FAILED
��������
TIme for solving (sec)   9.0224840000000008E-003
{    0,    0}:  On entry to PDPOTRS parameter number    1 had an illegal value
<< Warning >> pdpotrs FAILED
��������
Solved! delta test values
TIme for solving (sec)   9.1293109999999993E-003
{    1,    0}:  On entry to PDPOTRS parameter number    1 had an illegal value
<< Warning >> pdpotrs FAILED
��������
~

PBLAS ERROR 'Illegal DESCA[INB_] = 0, DESCA[INB_] must be at least 1'
from {0,0}, pnum=0, Contxt=0, in routine 'PDGEMV'.

PBLAS ERROR 'Illegal DESCA[NB_] = 0, DESCA[NB_] must be at least 1'
from {0,0}, pnum=0, Contxt=0, in routine 'PDGEMV'.

PBLAS ERROR 'Illegal DESCA[RSRC_] = 100000, DESCA[RSRC_] must be either -1, or >= 0 and < 2'
from {0,0}, pnum=0, Contxt=0, in routine 'PDGEMV'.

PBLAS ERROR 'Illegal DESCA[M_] = 0, it must be at least 1'
from {0,0}, pnum=0, Contxt=0, in routine 'PDGEMV'.

PBLAS ERROR 'Illegal DESCX[INB_] = 0, DESCX[INB_] must be at least 1'
from {0,0}, pnum=0, Contxt=0, in routine 'PDGEMV'.
...



Any suggestions now?

• It sounds like you are running out of memory due to fill in. I can not now as the backtrace is obfuscated due to optimization and lack of debug symbols. How many processes are you using and what is the amount of memory per process? – Abdullah Ali Sivas Aug 1 '20 at 4:21
• For the love of all that's numerically stable, please tell me you're not storing a sparse matrix in dense format? Secondly, you're using Scalapack which is a distributed memory solver. Do you actually have a cluster? Otherwise use a single-processor solver. And if you're running this on a single machine, you may very well run out of memory since 100k squared is about a Terabyte. Did you buy \$50k worth of memory? – Victor Eijkhout Aug 1 '20 at 4:25
• @VictorEijkhout Actually 8*(10^5)^2 bytes is 75 GiB. It sounds plausible that OP has 19 GiB available for the 50kx50k problem but not 75 GiB. – Federico Poloni Aug 1 '20 at 6:45
• The backtrace isn't helpful because the libraries you're using weren't compiled with debugging symbols. The first thing to do would be to rebuild the libraries with debugging symbols so that you can get a readable backtrace. – Brian Borchers Aug 1 '20 at 15:42
• An out of memory error shouldn't show up as a segmentation fault. A segmentation fault is something that you should bring up with the authors of the SCALAPACK library. – Brian Borchers Aug 1 '20 at 15:48