2
$\begingroup$

I am purchasing a new workstation to run FDS (Fire Dynamics Simulator) simulations, a CFD code for thermally driven fluid flow.

Currently, I am using an Ubuntu Linux build with a Xeon E5-2630 v3 @ 2.40 GHz with 8 cores and 16GB of RAM. The total mesh is about 20,000 cells (4.5 by 4.5 by 10cm), divided into 8 meshes, 3.75mm inner cell size, and it takes about 3 days 10 hours to run the current iteration.

I would like to improve my computation times significantly and decrease my mesh size to 1 or 2mm, ideally (so almost a factor of 16*16 in 3 dimensions). I would like to run a Windows 10/11 PC instead of Linux.

The software runs MPI parallel computing, and it is ideal to run 1 mesh per 1 thread with no hyperthreading, but computation improvements are seen for running in serial (8 meshes in 16 cores, etc.). There are decreased benefits at higher number of meshes, limit to performance around 16^3 cells per mesh. RAM recommended at 2GB per core.

Any suggestions on speeds, architecture, number of cores?

Based on PassMark Benchmarks, for instance, the i9-12900K (16 cores, 3.2Ghz base) has a score of 40,258 vs. Xeon Gold 6230R (26 cores, 2.10GHz base) with a score of 30,391. The Xeon is $1,100 more expensive and usually sold with a Workstation setup, whereas the i9 is more for high performance desktops. The scores seem consistent with other benchmarks (Geekbench, PCMark).

TLDR: Are professional benchmark comparisons accurate for say an i9-12 versus a current Xeon Gold or Platinum? If the overall speed is the same, will I gain anything with a slower CPU with more cores, or one designed for scientific/workstation use? Or does this vary case by case for software? What is best CPU for MPI parallel computing for CFD and/or serial computing for CFD on a $3,000-5,000 budget?

Thanks for any help!


Note: I have also been exploring running FDS on AWS following a tutorial from documentation (https://aws.amazon.com/blogs/compute/fire-dynamics-simulation-cfd-workflow-using-aws-parallelcluster-elastic-fabric-adapter-amazon-fsx-for-lustre-and-nice-dcv/) but it is tough to do all the coding. If this is a faster option (72cores c5n.18xlarge instance) then I might just do that.

$\endgroup$
5
  • 1
    $\begingroup$ Nobody here will know the details of FDS sufficiently well to know how it uses the hardware. You should ask this question on the FDS-specific forums. $\endgroup$ Feb 1 at 2:59
  • $\begingroup$ Thank you, I will ask them my further questions. I was hoping for maybe just some insight as to why two processors had the same benchmarks when one was 26 cores and 1,800 USD and the other 16 cores and 650 USD. I think maybe the tests they are using are not the same as intended use for a high performance Xeon workstation. $\endgroup$
    – John S.
    Feb 1 at 3:05
  • $\begingroup$ It seems I need to do some more research on CFD setups and Xeon workstation vs i processors. $\endgroup$
    – John S.
    Feb 1 at 3:53
  • 3
    $\begingroup$ I'm assuming it's using an implicit time stepping scheme with finite volumes and a subgrid turbulence model. If you have chemistry degrees of freedom (in addition to thermo-mechanics), the problem size can quickly become quite large and you will be memory constrained. I'd suggest trying Azure cloud machines instead of buying a machine if that's an issue (I would recommend at least 64 GB - 128 GB memory). The Xeon/AMD EPYC chips are good for that sort of thing. $\endgroup$ Feb 1 at 3:53
  • $\begingroup$ Thank you. I currently am learning how to run the program in AWS as there is documentation. If it is blazing my fast, I might not buy an expensive machine. $\endgroup$
    – John S.
    Feb 1 at 13:34

1 Answer 1

3
$\begingroup$

Here are some generic recommendations on HPC workstation and server node procurement.

First, figure out if you need ECC memory or not, since that determines whether you buy Intel Core and AMD Ryzen desktops/workstations versus Intel Xeon, AMD ThreadRipper/EPYC or ARM workstations/servers. Most likely it won't matter but if you are doing things where a single erroneous result would create problems, you ought be using ECC.

Second, essentially all CFD workloads are bandwidth-limited, which means DRAM channels are more important than cores, so long as you have enough cores to saturate your DRAM channels.

As of September 2022, Intel Xeon 14 nm servers (e.g. Cascade Lake) have 6 channel DDR4, whereas Intel Xeon 10 nm servers (e.g. Ice Lake) as well as all AMD and ARM servers I know of have 8 DDR4 channels. Intel Core and AMD ThreadRipper workstations have fewer DDR channels (at most 4, from what I've seen, but I haven't looked exhaustively).

Now, if your CFD problems are small and potentially fit into L3 cache, you should consider that in more detail. AMD is offering more L3 than Intel, although Intel L3 caches are monolithic, which means they are fully shared by all cores, whereas AMD L3 caches are per CCD/CCX (see e.g. this for details), which may be relevant depending on how the code is parallelized. OpenMP parallelism usually needs a monolithic cache, whereas MPI mostly doesn't care.

The other issue is whether the code scales ideally. Not all codes strong- and weak-scale ideally, and I recall that FDS has at least some MPI scaling problems, at least in multi-node scenarios, although I can't determine if they are are relevant to your use cases.

Anyways, here some examples of systems that I'd considering building, assuming that I was running open-source codes that don't depend on some vendor compiling binaries for me, and thus there is no problem using ARM AArch64 instead of Intel/AMD x86.

The Puget Systems mini-ATX desktops for Intel and AMD 12-core CPUs with 4x32 GB of memory both come in around \$4K. The closet equivalent in the ARM world would be the Apple M1 Studio with 10-20 cores, which is \$2-5K, depending on how you equip it.

For most CFD codes, I'd expect the microarchitectural differences between Intel, AMD, ARM Neoverse and Apple Silicon to be far less important than memory configuration and core count. AMD appears to be better than Intel right now, depending on which generations one compares, but I consider the non-uniform cache and memory issues on AMD to be a major inconvenience. On the other hand, if your code requires the Intel Fortran compiler, then you have to buy an x86 system.

Finally, cost is primarily a function of core count and frequency, although there are some cases where lower power CPUs cost more because of that. You'll find that 1/2 to 2/3 of the max core count with modest frequencies are a good deal, and in most cases, these will achieve around the same memory bandwidth of the top bin (most expensive) part, so they'll be a good deal for CFD.

Detailed conflict-of-interest statement

I currently have significant stock holdings in NVIDIA (NVDA), Intel (INTC) and Apple (AAPL).

I don't own AMD stock because that would have created a conflict of interest with my jobs at Intel (past) and NVIDIA (current).

I have no business relationship with Avantek or Puget Systems. I just picked them as examples because I remembered their names first and their websites are reasonable to navigate. There are lots of vendors out there, both small and large.

I have not and will not receive any commission from anybody for this post or because of your purchase of any of the mentioned technology.

$\endgroup$
5
  • 1
    $\begingroup$ From report High Performance Computing as a Resource for Fire Engineering Design, FireNZE April 2016: "... diminished returns for parallel processing of an FDS model typically occurs with allocation of between 8 and 16 processes. Run-time reductions through the application of more than 16 processes to a single simulation can be expected to be minimal. ... Processor speed remains a significant factor for reduced run-time .." Wouldn't that suggest a platform based on something like a relatively highly-clocked 16-core CPU, eg. Xeon W-3335? $\endgroup$
    – njuffa
    Sep 25 at 21:42
  • 1
    $\begingroup$ In my own benchmarks, I have found significant improvements going from 1-2 processes to 8 and to 16. Less when going from 16-18. The code splits up the "environment" into meshes that it then assigns to processors, so I have found my sweet spot is 18 meshes because I want the fire plume in the center of the environment to not be split by the meshes. Makes a more accurate calculation, in my opinion. $\endgroup$
    – John S.
    Sep 26 at 12:41
  • 2
    $\begingroup$ To Jeff Hammond, thanks a lot for your response. I have actually purchased the PC, an Intel Xeon(R) Gold 6242R 20 core single processor with 64GB RAM. It is a quite lower spec than your suggested, but as the above commenter mentioned, there are diminished returns in FDS above 16 processes/meshes. I brought my issue to the FDS Discussion Group as suggested above and they expressed that I should have 2GB memory for each core of the processor. In my own benchmarks, I noticed that a higher clock speed was almost as important as number of cores, so long as you had at least 8-16. $\endgroup$
    – John S.
    Sep 26 at 12:46
  • 1
    $\begingroup$ +1 for proper conflict of interest statement. $\endgroup$
    – MPIchael
    Sep 26 at 12:47
  • $\begingroup$ Good to know about these FDS issues. I have no experience with the code. $\endgroup$ Sep 26 at 12:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.