Effecient CFD programming techniques

Question

I'm trying to make highly efficient CFD programming complex for solving combustion problems. I've finished writing core which realises mathematical model, and now I'm concerned about code performance. Combustion problems are very computationally expensive, with no doubt I'll have to make parallelization of my code, but for now I want to make highly efficient serial code, to use it as a base for parallel program.

For instance there is simplified version of what is going on in program:

I have the main program where time loop is located.

PROGRAM Flame
USE Geometry
CALL ALLOCATION  <- Procedure there memory is allocated for X array
DO while ...
Time=Time + TH
CALL CALCULATIONS
END DO 
END PROGRAM

Modules with all global variables declarations

MODULE Geometry
double precision , allocatable :: X(:) <-- one dimensional array for computational grid.
integer, dimension(NTR) :: KDIF <-- has a mean of pointers on some variable in cell (pressure for example)
END MODULE

Subroutines for calculations and data processing.

SUBROUTINE CALCULATIONS
USE Geometry
IMPLICIT DOUBLE PRECISION (A-H,O-Z)
IJK = 0
DO I=1,NX <- number of cells in one dimension (such cycles for all dimensions) 
  DO I1=1,NTR
    X(IJK+KDIF(I1)) = ... some floating-point operations <-- Possible hot spot
    I1 = I1+1
  END DO
  X(IJK+8)=X(IJK+3)-A/X(IJ+5)-(X(IJ+10)-B)*C/(D*X(IJ+5)) (for example)
IJK = IJK + NKX  <-- place between two cells in 
END DO
END SUBROUTINE

There is much more subroutines and modules in program, but they all have structure similar to that I gave as an example. I've tried to profile my program with VTune Amplifier, and located some hot spots in places like I pointed in code. In Assembly of hot spots like this most time consuming operations are generally movesd qword; move ebx, dword; etc. I'm not familiar with assembly codes but I think it is related to memory operations. Actually I have no idea how to fix that hot spots.

So there are some questions that I need to clarify. Is that style of coding modern? I mean maybe I should use another techniques for such programs, like pointers, FORALL and WHERE operations, which can be faster and more memory efficient, more appropriate for CFD problems? Is such program structure efficient (modules for data, subroutines, allocatable arrays, one-dimensional array even for multidimansional problems)?

Thanks in advance for your attention, answers and advices.

Answer 1

Optimizing code is a very broad problem. Before you get too far into the details of an implementation, you should make sure your models and algorithms are doing what you want them to as efficiently as possible. A good algorithm is better than a good implementation and premature optimization is the root of all evil.

That much being said, here are some comments on the code you've given:

• Don't use implicit typing, ever; it turns typos into bugs.
• There are mixed opinions on forall, where, etc. This discussion is particularly interesting. Summary: forall, where, etc don't provide much, if any, speed, but they can be nice for representing algorithms.
• Don't pass array slices as arguments; they can cause unintended temporary allocations and copies.
• allocatables are fine; only switch to pointers if you need more functionality
• Modules are fine. I personally like to reserve modules for parameters or variables that are nearly parameters (and routines), and move data around explicitly. Fortran is pass by reference, so large array arguments are no more expensive than scalers. I've seen successful codes do it either way.