4
$\begingroup$

I have a CFD code that I want to run on an HPC cluster. Of course, before it can be run, it first needs to be compiled.

What is the best option between compiling on the head node and compiling on one of the compute nodes ? The hardware specifications of head and compute nodes are not necessarily identical, and sometimes all compute nodes aren't the same either : can that change the optimisation behaviour/efficiency of the compiler significantly ? What is the common best practice ?

Of course, in case of compiling on computation nodes, I would be waiting for the workload manager (e.g. slurm) to grant "RUN" state to the job. The submission script would then compile the software and then run it. Doing otherwise would not be nice to the other users…

Improve this question
$\endgroup$
5
  • 8
    $\begingroup$ Best to ask your cluster administrators for advice on this one. there are pros and cons to each approach, and the best advice should come from the people most familiar with your particular HPC setup. $\endgroup$
    – GoHokies
    Commented Feb 9, 2016 at 12:46
  • $\begingroup$ If your CFD code could be of interest also for other users you can ask your admins to install it system wide. A good sys-op should know about how to optimise and tune the code for the peculiarities of the cluster at hand. $\endgroup$
    – Stefano M
    Commented Feb 9, 2016 at 21:58
  • $\begingroup$ Thanks for your answers, already knowing that both options can be considered counts as an answer for me, so thanks for pointing this out ! To answer @StefanoM 's question, no I don't think my code will be relevant to other users (it only covers a few user cases, therefore another user will need to modify it first in order for it to become relevant to them). $\endgroup$
    – Gael Lorieul
    Commented Feb 10, 2016 at 16:45
  • $\begingroup$ StefanoM GoHokies you should turn your comments into an answer… First to do so gets the points ! $\endgroup$
    – Gael Lorieul
    Commented Feb 15, 2016 at 8:31
  • $\begingroup$ A lot depends on the build procedure for the software. Most compilers have options to generate code optimized for a specific version of a processor architecture (e.g. Haswell, Skylake, etc. for Intel processors.) In some cases you can specify this so that code optimized for the actual target processor is produced even if compiled on a different processor. In other cases, the build process will compile for a generic version of the architecture. A third possibility is that the build process will compile code optimized for the architecture of the machine its running on. $\endgroup$
    – Brian Borchers
    Commented Oct 18, 2018 at 18:35

1 Answer 1

Reset to default
6
$\begingroup$

Let me supply a belated answer from the POV of a cluster user and a cluster administrator.

A well-designed cluster will, in general, be as homogeneous as possible, with login-nodes being generally a bit beefier due to being the single point of contact of users to the cluster. A well maintained cluster will also have a good documentation of the system, for users who want to get their hands really dirty.

Over time I have acquired a few rules of thumb for developing and using parallel software which I will share here:

  1. If the goal is simply a running program try to compile it on the login node. If possible (and if you are using Make) use parallel compilation.
  2. Same as 1, but the compilation is progressing slowly. The impact on other users may not be negligible, especially if you use parallel make (make -j <n>). In this case you should try to estimate (lavishly) the time required to compile the code and submit a job to the queue.
  3. If the goal is to have an optimized code but not rely on architectural optimisations then you should submit the compilation job in exclusive mode, i.e. requesting an entire node for the compilation. In this way other users will not interfere with the optimisation process. An example of a code that i would compile this way would be NumPy, with its profiler guided optimisations and link-time optimisations. Simple -O3 optimisations can be used with the first two versions, too.

  4. If the goal is to have an optimised code and use architectural optimisations (AVX, SSE etc.) then it is safer to compile on the nodes, since that is where the job will run. Caveat: If the cluster you are running on consists of multiple CPU architectures (e.g. Intel Broadwell, Skylake), compare for instance http://www.bwhpc-c5.de/wiki/index.php/BwUniCluster_Hardware_and_Architecture then you have two options:

    1. Use only options common to both architectures. This generally entails optimising for the older one.

    2. Compile on nodes with a specific architecture and optimise for that architecture. Generally job schedulers such as SLURM allow selection of resources via constraints. Ask your cluster-admin though. Afterwards, when you run the code make sure to limit the set of nodes it can be run on only to those with the fitting architecture, else the program will likely crash.

These are rough guidelines. Myself I prefer not to submit a compilation script to the queue if I am not 100% sure that the compilation will succeed. In such cases I submit a request for an interactive session, which is essentially like ssh-ing to a node, only the batch system will kick you out once the requested time is up.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.