0
$\begingroup$

I am new to GNU Octave 6.2.0 on Linux, and I am trying to use the package 'parallel' to make use of 24 threads on my machine.

I have an array (array1) that has about 500,000 rows or values (1 column) and for each member of the array I want to apply a log10 function, like this:

    test = 10*log10(array1.^2); % this works

I tried using the parallel package like so:

test = pararrayfun(24, @(array1) 10*log10(array1.^2), array1, "Vectorized", 1 "ChunksPerProc", 1); % this fails

But I am getting the error (repeating lots of times):

Could not save variable
Could not load variable

I also tried to apply the "smooth" function (from data-smoothing) to the array using a span of 100 using parallel but that too fails:

test2 = smooth(array1,100);  % this works fine

test2 = pararrayfun(24, @(array1) smooth(array1,100), array1); % this fails

Can anyone help me with the proper syntax for using 'parallel' with the above two examples ?

Thanks so much !!

$\endgroup$
5
  • $\begingroup$ What happens if you try running test = pararrayfun(24, @(x) 10*log10(x.^2), array1, "Vectorized", 1 "ChunksPerProc", 1); ? $\endgroup$ Jun 5, 2021 at 21:21
  • $\begingroup$ I get: error: reshape: can't reshape 11184120x1 array to 466005x1 array error: called from chunk_parcellfun>@<anonymous> at line 55 column 33 chunk_parcellfun at line 55 column 15 pararrayfun at line 73 column 28 $\endgroup$ Jun 5, 2021 at 22:32
  • $\begingroup$ Do you happen to be on a two processors machine (each with 12 threads)? Or in some other sort of distributed structure (like two different machines running on a high speed network, MPI type of arrangement) ? $\endgroup$ Jun 6, 2021 at 1:22
  • $\begingroup$ It's a server workstation with dual Intel Xenon E5-2620: 2 numa nodes, each with 6 cores/12 threads for a total of 12 cores/24 threads. $\endgroup$ Jun 6, 2021 at 13:34
  • $\begingroup$ I don't think that Octave can handle that arrangement automatically yet. You should start a parallel server first (octave.sourceforge.io/parallel/function/pserver.html), otherwise Octave won't be able to send the array to the secondary cpu (the primary is the CPU running the Octave session) afaik. $\endgroup$ Jun 6, 2021 at 14:24

1 Answer 1

1
$\begingroup$

I can not reproduce the error. Here is the snippet I am using:

pkg load parallel

vector_x = rand(500000,1);
tic
vector_y = pararrayfun(nproc, @(x) 10.*log10(x.^2), vector_x, "Vectorized", true, "ChunksPerProc", 1);
toc
tic
vector_y = pararrayfun(1, @(x) 10.*log10(x.^2), vector_x, "Vectorized", true, "ChunksPerProc", 1);
toc
tic
for i=1:length(vector_x)
    vector_y(i) = 10.*log10(vector_x(i).^2);
end
toc
tic
vector_y = 10.*log10(vector_x.^2);
toc

I was fooling around and testing some other options to do the same calculation and comparing the time spent for each one. I have a philosophical issue with arrayfun / pararrayfun (What is the use of arrayfun if a for loop is faster?), I believe that they are completely redundant, and they fool MATLAB/Octave users into false sense of efficiency. In my experience, MATLAB's JIT compiler is smart enough to vectorize simple for loops like we have here. However, apparently the Octave interpreter is not smart enough.

I am on a machine with 64 threads and here are the timing results

% pararrayfun using all available threads with ChunksPerProc = 1. Increasing this value 
% (in theory) decreases time on busy machines, but in my case it didn't help.
parcellfun: 64/64 jobs done
Elapsed time is 0.0932262 seconds.

% pararrayfun using 1 thread with ChunksPerProc = 1. Compare against using all the cores. 
% Uses half the amount of time and 1/64th of the cores.
parcellfun: 1/1 jobs done
Elapsed time is 0.0447989 seconds.

% Simple for loop.
Elapsed time is 3.23663 seconds.

% Simple vectorization to let Octave figure out the threading itself. Fastest.
Elapsed time is 0.0195971 seconds.

While I cannot answer your question (as I cannot reproduce the error), I would suggest you not to use pararrayfun for this case. Also, I am yet to encounter a single case where arrayfun/pararrayfun beats simple vectorization (or simple vectorization combined with parfor).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.