Let me supply a belated answer from the POV of a cluster user and a cluster administrator.
A well-designed cluster will, in general, be as homogeneous as possible, with login-nodes being generally a bit beefier due to being the single point of contact of users to the cluster. A well maintained cluster will also have a good documentation of the system, for users who want to get their hands really dirty.
Over time I have acquired a few rules of thumb for developing and using parallel software which I will share here:
- If the goal is simply a running program try to compile it on the login node. If possible (and if you are using Make) use parallel compilation.
- Same as 1, but the compilation is progressing slowly. The impact on other users may not be negligible, especially if you use parallel make (
make -j <n>). In this case you should try to estimate (lavishly) the time required to compile the code and submit a job to the queue.
- If the goal is to have an optimized code but not rely on architectural optimisations then you should submit the compilation job in exclusive mode, i.e. requesting an entire node for the compilation. In this way other users will not interfere with the optimisation process. An example of a code that i would compile this way would be NumPy, with its profiler guided optimisations and link-time optimisations. Simple
-O3 optimisations can be used with the first two versions, too.
If the goal is to have an optimised code and use architectural optimisations (AVX, SSE etc.) then it is safer to compile on the nodes, since that is where the job will run. Caveat: If the cluster you are running on consists of multiple CPU architectures (e.g. Intel Broadwell, Skylake), compare for instance http://www.bwhpc-c5.de/wiki/index.php/BwUniCluster_Hardware_and_Architecture then you have two options:
Use only options common to both architectures. This generally entails optimising for the older one.
Compile on nodes with a specific architecture and optimise for that architecture. Generally job schedulers such as SLURM allow selection of resources via constraints. Ask your cluster-admin though. Afterwards, when you run the code make sure to limit the set of nodes it can be run on only to those with the fitting architecture, else the program will likely crash.
These are rough guidelines. Myself I prefer not to submit a compilation script to the queue if I am not 100% sure that the compilation will succeed. In such cases I submit a request for an interactive session, which is essentially like
ssh-ing to a node, only the batch system will kick you out once the requested time is up.