I am working on some medium to large scale finite element codes. By using established and available tools I am able to have an algorithm that scales well up to about 10,000 cores. Investigating scaling beyond this requires investigating larger meshes, which leads to my problem.
Once meshes start getting large (in range of 100s of gigabytes to terabyte ranges), simply getting them to a cluster environment can completely overwhelm any cost of solving the resulting system. 100GB-1TB mesh sizes aren't especially large sizes by today's standard of nodes as well, which can have upwards of 64GB of memory each (and in many cases more than that)
SO how is this commonly handled? Are there common ways to improve the bandwidth in transferring data to a cluster? Do you just need to be on an incredibly high bandwidth connection, or physically ship a drive containing all the data you want?
As a followup question: if I could re-engineer this, would it solve the problem to rely more on automatic mesh refinement so that we start with a smaller starting mesh and refine at need in-memory?