Overview and Prior Research
I am looking for a way store a (in principle arbitrarily) large "3D-table" for interpolation/ lookup in combination with python.
I have considered
- CSV files, but, as far as I know, I will run into file size issues starting at 2GB
- Pytables: as I understood it, it is close to a proper database, which sounds like overkill to me
- PostgreSQL: when run locally, I might get into trouble with memory + overkill
There are a number of questions on SE about similar problems, but I have found none specific to python and the interpolation aspect.
The data I am interested in is ~ (10.000 x 10.000 x 100) and a rough estimation got me to a CSV size of ~200GB. In principle I could split off the last dimension and create 100 different (10.000 x 10.000) tables, which would then take ~2GB, each.
What are reasonable data structures and python packages for such problems? I know in meteorology there exists a "3D-pandas" with a corresponding file type, but I cannot remember/ find it.
Evaluation in chunks without too much effort might be important, too. I am trying to use this lookup table with JAX, so if by any chance, you happen to know a good solution there, it would be perfect.
Speed might be an issue, depending on how often I have to load the table. Overall execution time is on the scale of 10h.