Sorry for the long post but I wanted to include everything that I thought was relevant in the first go.
What I want
I am implementing a parallel version of Krylov Subspace Methods for Dense Matrices. Mainly GMRES, QMR and CG. I realized (after profiling) that my DGEMV routine was pathetic. So I decided to concentrate on that by isolating it. I have tried running it on a 12 core machine but the results below are for a 4 core Intel i3 Laptop. There is not much difference in the trend.
KMP_AFFINITY=VERBOSE output is available here.
I wrote out a small code :
size_N = 15000 A = randomly_generated_dense_matrix(size_N,size_N); %Condition Number is not bad b = randomly_generated_dense_vector(size_N); for it=1:n_times %n_times I kept at 50 x = Matrix_Vector_Multi(A,b); end
I believe this simulates the behaviour of CG for 50 iterations.
What I have tried:
I had originally written the code in Fortran. I translated it to C, MATLAB and Python (Numpy). Needless to say, MATLAB and Python were horrible. Surprisingly, C was better than FORTRAN by a second or two for the above values. Consistently.
I profiled my code to run and it ran for
46.075 seconds. This was when MKL_DYNAMIC was set to
FALSE and all cores were used. If I used MKL_DYNAMIC as true, only (approx.) half the number of cores were in use at any given point of time. Here are a few details:
Address Line Assembly CPU Time 0x5cb51c mulpd %xmm9, %xmm14 36.591s
The most time consuming process seems to be :
Call Stack LAX16_N4_Loop_M16gas_1 CPU Time by Utilization 157.926s CPU Time:Total by Utilization 94.1% Overhead Time 0us Overhead Time:Total 0.0% Module libmkl_mc3.so
Here are a few pictures:
I'm a real beginner at profiling but I realize that the speed up is still not good. The sequential (1 Core) code finishes in 53 seconds. That is a speed up of less than 1.1 !
Real Question : What should I do to improve my speedup?
Stuff that I think might help but I can't be sure:
- Pthreads implementation
- MPI (ScaLapack) implementation
- Manual Tuning (I don't know how. Please recommend a resource if you suggest this)
If anyone needs more (especially regarding memory) details, please let me know what I should run and how. I have never memory profiled before.