List of data munging libraries

Question

I am trying to put together a complete list of all of the tools that computational scientists have found useful when trying to munge data, i.e. take data in one format, extract the useful bits, and put it into a different format.

So what are the libraries that you've used in the past to obtain data from fixed-width, CSV, XML, etc. files, to sort the data, and to output it into useful formats?

Answer 1

My favorite language for these tasks is Python; Perl also works for those who are so inclined.

Python modules:

• The csv module from the Python standard library handles parses CSV files.
• The Python standard library also has modules for standard markup languages like XML, HTML, SGML, etc.
• For fixed-width strings, see this post on Stack Overflow.
• I haven't used this module, but supposedly p3d is a Python module that will help with parsing protein data in pdb format.
• The regular expressions module in Python can be quite useful.
• You can also Google or check the Python Package Index for other Python packages that suit your needs.

Perl modules:

• The Text::CSV module works for CSV processing. I've used it, and it's relatively intuitive.
• The XML::Simple and XML::Parser modules seem like they would work for XML processing, though there are a lot of XML modules available on the Comprehensive Perl Archive Network (CPAN), so feel free to pick one that suits your preference.
• The HTML::Tree::Scanning modules seem to serve the same sort of purpose for HTML.
• This post from Stack Overflow talks about extracting fixed-width data from files using Perl.
• Regular expressions are baked into Perl.
• CPAN is also a good source to look for other libraries.

C++ libraries:

• There are a bunch of XML parser libraries (see this post on Stack Overflow). Particularly reputable are Expat, Xerces, and the C++ bindings to Gdome2 (my personal favorite).
• This post from Stack Overflow talks about extracting data from CSV files.
• For other standard formats, there's a good chance there's a parser out there (HTML, etc.). A lot of times, the Perl and Python versions of parser libraries are language bindings for C or C++ parser libraries.
• For fixed-width files, you can use fprintf.
• This post from Stack Overflow talks about C++ regular expression libraries; also see Boost::Regex and PCRE (from C).

C libraries:

• Again, there are many XML parser libraries. Worth checking out are Expat, Gdome2, and libxml2.
• A lot of people seem to roll their own C libraries for CSV parsing. I found documentation for a library, but it seems a little sketchy.
• For fixed-width files, you can again use fprintf.
• The best C regular expression library I know of is PCRE.

Hope this helps.

Answer 2

When the main task is changing the shape or structure of data rather than reading a specific file format (like .mat or hfd5 for a matrix of the same shape), then I find Mathematica excellent for the task. Its native data structure is essentially a tree (list of lists of lists ...), and it can do flexible transformations on them using pattern matching or other means.

It has two weaknesses in my opinion:

• If it doesn't already support the input/output file format, it's not the best tool to implement your own importer in (unless the format is trivial). Here's the listing of supported formats. (PDB is supported.)

• It works well if all your data fits comfortably into memory. I find that it is not a very good tool for sequential data processing (i.e. reading a small bit of the input and outputting it immediately).

For kneading complex or structured data into the desired shape (think high dimensional ragged arrays or XML-like structured data), it is a very good tool though.

Answer 3

I aggree that Python is really good on this task. I would like to add some tools to the list:

• If you want to extract data from the web, or maybe fill in a form easilly, then mechanize is the best option.
• I heard that BeautifulSoup is really really good on parsing XML/HTML files.