# RAMdisks and finite element calculations

I heard about Ramdisks some years ago and even set up one for testing on mymachine. I didn't test it thoroughly so I couldn't really judge the performance improvements (in comparison to plain a SSD).

Ramdisks are in short parts of your working memory, which are allocated to a logical(software) disk. Therefore you can access them as an additional drive, put files and programs, as well as run simulations on them.

Programs cache in RAM anyway, but a lot of programs write to files, where they store their mesh data, intermediate results etc. Depending on the internals of the program, improvements could be possible. In my case the use would be for Finite Element calculations. The tool I use was created (and is mostly maintained) as scientific project, so many optimizations could be missing. I'm aware of possible data loss, it is acceptable, though it has to be accounted for.

Is there any advantage to be gained by using ramdisks? Has anyone tried using ramdisks for their calculations, was there any worthwhile (work/reward) improvement? Are there cases where it makes sense to use them?

• A typical OS kernel would cache filesystem IO by itself already: when a program writes something to a file, the typical behaviour is that it is also cached in main memory by the OS kernel. I say this because your question seems to assume that file IO is done directly, without caching. With caching, though, increasing disk bandwidth would have less of an effect. – Kirill Sep 20 '16 at 17:17

Truly efficient programs on modern high performance computing architectures require a careful design of algorithm and data structures to take advantage of the full memory hierarchy. (Not to speak about parallel programming issues, in which you may have multiple threads, multiple processes, SMP, NUMA, multiple interconnected nodes...)

If your question is rephrased as

Will a legacy program, written when out-of-core storage was a choice driven by main memory constraints, benefit from moving its scratch files to a ramdisk?

I would answer that the only way to know is by experiment. But as already pointed out by Kirill, I would not expect dramatic improvements, since almost every modern OS caches disk access.

In the *nix world you should check the concept of tmpfs which can be implemented with just a couple of lines in /etc/fstab and is much more flexible than an ordinary ramdisk. See for example the discussion in tmpfs.txt from the linux kernel.

• Thanks for the answer. I'll look into it, but I guess only out of curiosity and not for proffesional purposes. I actually made a tmpfs a few years ago, but I didn't test it with anything at the time though, as I'm windows bound for my work. – WalyKu Sep 20 '16 at 21:09

Most modern operating systems already cache reads from and writes to files in main memory, and so you don't gain anything by using a RAM disk -- it's just another layer of data caching, and it is not going to be any faster than the caches the operating system keeps for you. On the other hand, you are giving up a sizable chunk of memory that you can no longer use for computations.

The only place where it makes sense is if the file system you are working on is an network file system (e.g., NFS-mounted). In that case, operations such as linking executables that have a lot of random accesses into files become very slow because they tend to require frequent synchronization between the machine you're on and the file server (which may update the file on disk, or simply keep a copy of it in a cache). This synchronization over the network is expensive and one should try to avoid it. You can either avoid this by working on the local disk (e.g., in a /tmp file system) or in a RAM disk.

• I don't know the details of Linux I/O caching but if one has e.g. 8 GB of randomly accessed files, I suspect that Linux I/O caching will not be very good, whereas dumping the whole mess in /dev/shm or tmpfs will do wonders. At the very least, one would have to change the defaults in Linux - this requires admin privileges - to caching that much. – Jeff Hammond Sep 21 '16 at 3:14
• @Jeff 8GB is not very much these days on typical machines, so without specific evidence (measurements) to the contrary one should probably assume it will behave well with default settings. IIRC, it will just use all available idle memory, which may well be enough for 8GB of files with enough RAM. – Kirill Sep 21 '16 at 5:09
• I think that "you don't gain anything by using a RAM disk" is too drastic. Legacy programs have scratch files (out-of-core temporary memory) which are completely volatile (deleted at program termination.) Dirty pages are written back to disk after some time (vm.dirty_expire_centisecs) say 30s, so there is the possibility that volatile data gets on disk, just to be deleted afterwards. With tmps you guarantee that volatile data will never go on disk, at least if you have enough RAM. Delayed write back will not hit performance very bad, but it is better to avoid unnecessary disk writes. – Stefano M Sep 21 '16 at 7:59
• @StefanoM -- I'm not buying your arguments without concrete evidence. The only cost you point out is that files in the file cache are occasionally written back to disk, but this costs essentially no CPU time. The OS directs the DMA chip to move the data to the drive which will then write things from its own cache to the spinning disk. All of this happens in the background. Deleting a file later on also costs essentially nothing. The millisecond you spend on each every 30 seconds is not measurable. – Wolfgang Bangerth Sep 21 '16 at 13:22
• @Kirill I would not have made that comment if I had not measured an application running significantly faster with a ramdisk than otherwise. In that case, I had a 2 TB node; the same job did not run on a 64 GB node. – Jeff Hammond Sep 21 '16 at 16:07