4
$\begingroup$

Overview and Prior Research

I am looking for a way store a (in principle arbitrarily) large "3D-table" for interpolation/ lookup in combination with python.

I have considered

  • CSV files, but, as far as I know, I will run into file size issues starting at 2GB
  • Pytables: as I understood it, it is close to a proper database, which sounds like overkill to me
  • PostgreSQL: when run locally, I might get into trouble with memory + overkill

There are a number of questions on SE about similar problems, but I have found none specific to python and the interpolation aspect.

Setup

The data I am interested in is ~ (10.000 x 10.000 x 100) and a rough estimation got me to a CSV size of ~200GB. In principle I could split off the last dimension and create 100 different (10.000 x 10.000) tables, which would then take ~2GB, each.

Questions

What are reasonable data structures and python packages for such problems? I know in meteorology there exists a "3D-pandas" with a corresponding file type, but I cannot remember/ find it.

Evaluation in chunks without too much effort might be important, too. I am trying to use this lookup table with JAX, so if by any chance, you happen to know a good solution there, it would be perfect.

Speed might be an issue, depending on how often I have to load the table. Overall execution time is on the scale of 10h.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Check out the HDF5 binary format. In python you can make use of it via h5py package or as you already mentioned via pytables. I already used it to write and access simulation data with >100GB and its pretty good performance wise. $\endgroup$
    – Pepe
    Sep 22, 2021 at 18:13
  • 1
    $\begingroup$ Do have to use the exact data or is an approximation sufficient? How smooth is your data? You may consider locally refined octree or quadtree data structures, where each quad or oct consists of a high order polynomial. To reduce storage you may use a projection instead of a naive interpolation. There are many options if you can live with an approximation. $\endgroup$
    – ConvexHull
    Sep 22, 2021 at 22:13
  • 1
    $\begingroup$ Or you could just use a lossy compression scheme like in NetCDF (or using fpzip or similar projects). $\endgroup$
    – Wolfgang Bangerth
    Sep 23, 2021 at 3:07

2 Answers 2

Reset to default
4
$\begingroup$

Since you are on a uniform $x-y-z$ grid, you are in luck that others have had similar issues before. Specifically, I would suggest that you take a look at HDF5 for a low-level way of storing this kind of data as arrays.

But in the end, you want to not just store an array, but actually do something with it: interpolate. For this, a higher-level approach would be to use NetCDF format, which can under the hood use HDF5 to store data. The NetCDF libraries have ways to do this, plus a lot more. One of the features you might find interesting giving the size of the data you have is that it (i) can interpret large arrays as 3d data, (ii) compress this data by using spatial information about which data is located where in $x-y-z$ space.

You can find the documentation for NetCDF here: https://www.unidata.ucar.edu/software/netcdf/docs/index.html Both HDF5 and NetCDF also have Python interfaces.

Improve this answer
$\endgroup$
-2
$\begingroup$

Use a deconvolutional neural network.

There are ways to build a deconvolutional neural network that takes a source image, and then uses the neural network to build a higher-resolution version of that image. This is basically the reverse of using a convolutional neural network to simplify images into their low-resolution features. Depending on what your data looks like, it might be possible to do the same for your data - in which case, you no longer need to store the interpolated data, since you can just run a portion of it through the neural net to generate it on command.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Would the people who have downvoted be willing to explain why, so that I can improve my answer? $\endgroup$
    – nick012000
    Sep 24, 2021 at 7:03
  • 1
    $\begingroup$ Your answer sounds something like this: "Hey, here's an arbitrary dataset $\{x, y, z\}$. I'll just throw a learning algorithm at it." You neither want to learn from the data, nor you want to generalize something. You already have the exact data available, which should be represented as accurate and efficient as possible. $\endgroup$
    – ConvexHull
    Sep 24, 2021 at 11:27
  • $\begingroup$ @ConvexHull You do want to generalise something: the interpolation of the data. Why store the interpolated data when it can be generated dynamically whenever you need it? $\endgroup$
    – nick012000
    Sep 25, 2021 at 4:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.