1
$\begingroup$

I am sorry if this question seems like off topic or opinion based, but I was not sure how to go about it.

I am currently working on a 100k x 100k positive definite linear system and trying to solve it using Cholesky factorization from Scalapack 2.1.0 (pdpotrf). My problem is that when I run the same system for 50kx50k matrix (i.e. A(1:50000,1:50000)) my program runs fine. However it is giving segmentation fault in 100k run:

Backtrace for this error:
#0  0x7f8126942b9a in ???
#1  0x7f8126941dc3 in ???
#2  0x7f8125e4824f in ???
#3  0x45409d in ???
#4  0x44382c in pdpotf2_
        at /home/user/sclpk_solver/scalapack-2.1.0/SRC/pdpotf2.f:250
#5  0x414502 in pdpotrf_
        at /home/user/sclpk_solver/scalapack-2.1.0/SRC/pdpotrf.f:262

Normally I would like to use record-replay or such debugging tools to go about it, but given the size of the problem and the fact that it takes it about 10 hours before I get the failure, I am not sure how to even start debugging it. The exact same code and exact same data works fine on intel MKL(ver 2018 service pack 3). But GNU compilers are giving issue (version 4.8 and 7.1).

Any comments are welcome.

UPDATE: I just figured out an important detail: my program works for cores greater then 4. its <=4 when it gives the segmentation fault. I think its because for 100k, 4 or less cores mean each core will have > 50k*50k elements. so there is somewhere integer overflow error. It can be strengthened by the fact that when compiled with intel MKL, it needs -i8 flag for numroc function to work properly.

Also I tried compiling scalapack with "-g -Og" flag, and this time program failed with following output:

{*****,    0}:  On entry to PDPOTRF parameter number    1 had an illegal value
<< Warning >> pdpotrf FAILED      -1
{*****,    0}:  On entry to PDPOTRF parameter number    1 had an illegal value
<< Warning >> pdpotrf FAILED      -1
{    0,    0}:  On entry to PDPOTRS parameter number    1 had an illegal value
<< Warning >> pdpotrs FAILED
��������
 Solved! predicting test values
{    1,    0}:  On entry to PDPOTRS parameter number    1 had an illegal value
<< Warning >> pdpotrs FAILED
��������
 TIme for solving (sec)   9.0224840000000008E-003
{    0,    0}:  On entry to PDPOTRS parameter number    1 had an illegal value
<< Warning >> pdpotrs FAILED
��������
 Solved! delta test values
 TIme for solving (sec)   9.1293109999999993E-003
{    1,    0}:  On entry to PDPOTRS parameter number    1 had an illegal value
<< Warning >> pdpotrs FAILED
��������
~

PBLAS ERROR 'Illegal DESCA[INB_] = 0, DESCA[INB_] must be at least 1'
from {0,0}, pnum=0, Contxt=0, in routine 'PDGEMV'.

PBLAS ERROR 'Illegal DESCA[NB_] = 0, DESCA[NB_] must be at least 1'
from {0,0}, pnum=0, Contxt=0, in routine 'PDGEMV'.

PBLAS ERROR 'Illegal DESCA[RSRC_] = 100000, DESCA[RSRC_] must be either -1, or >= 0 and < 2'
from {0,0}, pnum=0, Contxt=0, in routine 'PDGEMV'.

PBLAS ERROR 'Illegal DESCA[M_] = 0, it must be at least 1'
from {0,0}, pnum=0, Contxt=0, in routine 'PDGEMV'.

PBLAS ERROR 'Illegal DESCX[INB_] = 0, DESCX[INB_] must be at least 1'
from {0,0}, pnum=0, Contxt=0, in routine 'PDGEMV'.
...

Any suggestions now?

Improve this question
$\endgroup$
10
  • 1
    $\begingroup$ It sounds like you are running out of memory due to fill in. I can not now as the backtrace is obfuscated due to optimization and lack of debug symbols. How many processes are you using and what is the amount of memory per process? $\endgroup$
    – Abdullah Ali Sivas
    Commented Aug 1, 2020 at 4:21
  • 2
    $\begingroup$ For the love of all that's numerically stable, please tell me you're not storing a sparse matrix in dense format? Secondly, you're using Scalapack which is a distributed memory solver. Do you actually have a cluster? Otherwise use a single-processor solver. And if you're running this on a single machine, you may very well run out of memory since 100k squared is about a Terabyte. Did you buy $50k worth of memory? $\endgroup$
    – Victor Eijkhout
    Commented Aug 1, 2020 at 4:25
  • 2
    $\begingroup$ @VictorEijkhout Actually 8*(10^5)^2 bytes is 75 GiB. It sounds plausible that OP has 19 GiB available for the 50kx50k problem but not 75 GiB. $\endgroup$
    – Federico Poloni
    Commented Aug 1, 2020 at 6:45
  • 1
    $\begingroup$ The backtrace isn't helpful because the libraries you're using weren't compiled with debugging symbols. The first thing to do would be to rebuild the libraries with debugging symbols so that you can get a readable backtrace. $\endgroup$
    – Brian Borchers
    Commented Aug 1, 2020 at 15:42
  • 2
    $\begingroup$ An out of memory error shouldn't show up as a segmentation fault. A segmentation fault is something that you should bring up with the authors of the SCALAPACK library. $\endgroup$
    – Brian Borchers
    Commented Aug 1, 2020 at 15:48

1 Answer 1

Reset to default
1
$\begingroup$

There were some hints above already that this may be an integer range problem. A 32 bit signed integer goes up to values of about 2e9. A 50k x 50k symmetric matrix lives in an array of size about 5e4 * 5e4 / 2, which just about fits into an int32. The 100k matrix will no longer be accessed correctly with 32 bit integers.

So you may need to go through your program and its interfaces to the solver libraries carefully and see which integers may need to be changed to 64 bit types, possibly using typedefs of unmistakable names to avoid mixups.

The other paradigm would be to make the choice that process-local indices should be 32 bit always and to access bigger data sets only through parallelization.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.