My PDE simulation program written in Fortran has to make about 2 million variable time steps. But with each time step it slows down more and more, so that if it initially makes 1000 time steps per second, then when it reaches time step 600k and above, it makes 1000 time steps in as many as several minutes. And the slow down continues the further the simulation runs.

As far as I have established and understood, the problem is related to bufferization. There is enough memory and it does not get filled completely as the simulation progresses, but it does slow down quite a bit as the time steps progress. It looks like some caches get filled so that more and more latency occurs.

The solution would be to free up memory after every certain number of time steps, thus running the simulation in chunks. However, I don't yet know how to do that in Fortran. Does one have to use something like call Malloc() to dynamically allocate each arrays in Fortran, and then use call Free() to deallocate memory after a certain number of time steps has been done? Or is there a better approach toward this problem? Would appreciate some advice.

  • $\begingroup$ just to confirm: are you compiling your code with -O3 optimization level? $\endgroup$ – Anton Menshov Dec 30 '19 at 23:12
  • 1
    $\begingroup$ You've got a bug in your code. But it's hard to tell since we don't have the code. It's also a question that is not appropriate for this particular forum. $\endgroup$ – Wolfgang Bangerth Dec 30 '19 at 23:59
  • 1
    $\begingroup$ Agreed with @WolfgangBangerth, there’s a leak somewhere. Try to hone it down to a minimum not-working example and post on stackoverflow / computer science if you can’t solve the issue yourself. $\endgroup$ – Spencer Bryngelson Dec 31 '19 at 4:13
  • 1
    $\begingroup$ What do you do with the results? Do you compute a statistic for every step and store some accumulated result, or do you write the full state at each time step to the disk? $\endgroup$ – Lutz Lehmann Dec 31 '19 at 14:54
  • 1
    $\begingroup$ @sequence I've been programming for 32 years now, and I've had every bug you can think of not once, but two or three times, in my own programs. Over time, you become pretty good at spotting patterns, and a program that does time stepping and eventually runs out of memory has a memory leak somewhere, and one that slows down over time has a similar kind of issue where for whatever reason, the amount of work you do in each time step increase. I can't tell you where the issue is, but the pattern seems pretty clear to me: It's a bug in your code. $\endgroup$ – Wolfgang Bangerth Jan 9 '20 at 4:57