reading and processing large GRIB files

Question

I'm trying to convert and process GrADS (grib) files into netcdf files. The files have values every 3 hours (2890 per year), 12 z levels, 120 variables, 241 columns and 236 rows. The data is almost 80 gb per year and I have to process a total of 50 years.

Before converting into netcdf I'm trying to calculate daily averages and sums for a subset of the variables. So far I've tried with GrADS but I still haven't been able to make the script to work. Going through the GrADS users forum to find some examples of a similar task I got the feeling that GrADS is not as commonly used as it used to be years ago so I'm curious about what people currently use when they have to deal with data as large as these (I come from a GIS background, so I'm not used to data as big).

First hand I'll say that I won't be able to use R or Python due to RAM limitation, the relational database option is also an issue due to not being able to find a tool to import from grib to SQL tables. What other options I have for such a "simple" task?

Answer 1

I would suggest cdo. CDO can read GRIB files (though I think it only supports GRIB2 files) and has a host of options for computing means, subsetting, regridding etc. If you add the -f nc option as well it will output to NetCDF automatically.

ECMWF's ecCodes library will probably also come in handy.

I typically use a combination of CDO and ecCodes. As an example, if I want to process a GRIB file containing global meteorological fields from a climate model containing only one timestep, I would use the following:

# This ecCodes command splits the GRIB file "model_output+000000" into different
# vertical levels. In this case, the file only contains one kind of vertical level
# - pressure (hPa), but it could conceivably contain output on sigma levels, hybrid
# sigma coordinates, altitude etc. The "+000000" refers to the timestep contained
# in this file (the zeroth timestep).
grib_copy model_output+000000 model_output+000000_[typeOfLevel]

# The "*_isobaricInhPa" file was created by the previous command.
cdo -f nc setgridtype,regular model_output+000000_isobaricInhPa model_output+000000.nc

Answer 2

Looks like wgrib2 could turn the data into csv format.

http://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/csv.html

Once it’s csv, you have a variety of options.

Any etl tool could pull in a csv to do the calculations. Given I work for Talend, I would use Talend Studio.

The Australian Ocean Data Network has a set of Talend routines for netcdf. I’ve not tried them, but that seems worth checking out. https://github.com/aodn