I am trying to put together a complete list of all of the tools that computational scientists have found useful when trying to munge data, i.e. take data in one format, extract the useful bits, and put it into a different format.

So what are the libraries that you've used in the past to obtain data from fixed-width, CSV, XML, etc. files, to sort the data, and to output it into useful formats?


My favorite language for these tasks is Python; Perl also works for those who are so inclined.

Python modules:

  • The csv module from the Python standard library handles parses CSV files.
  • The Python standard library also has modules for standard markup languages like XML, HTML, SGML, etc.
  • For fixed-width strings, see this post on Stack Overflow.
  • I haven't used this module, but supposedly p3d is a Python module that will help with parsing protein data in pdb format.
  • The regular expressions module in Python can be quite useful.
  • You can also Google or check the Python Package Index for other Python packages that suit your needs.

Perl modules:

  • The Text::CSV module works for CSV processing. I've used it, and it's relatively intuitive.
  • The XML::Simple and XML::Parser modules seem like they would work for XML processing, though there are a lot of XML modules available on the Comprehensive Perl Archive Network (CPAN), so feel free to pick one that suits your preference.
  • The HTML::Tree::Scanning modules seem to serve the same sort of purpose for HTML.
  • This post from Stack Overflow talks about extracting fixed-width data from files using Perl.
  • Regular expressions are baked into Perl.
  • CPAN is also a good source to look for other libraries.

C++ libraries:

  • There are a bunch of XML parser libraries (see this post on Stack Overflow). Particularly reputable are Expat, Xerces, and the C++ bindings to Gdome2 (my personal favorite).
  • This post from Stack Overflow talks about extracting data from CSV files.
  • For other standard formats, there's a good chance there's a parser out there (HTML, etc.). A lot of times, the Perl and Python versions of parser libraries are language bindings for C or C++ parser libraries.
  • For fixed-width files, you can use fprintf.
  • This post from Stack Overflow talks about C++ regular expression libraries; also see Boost::Regex and PCRE (from C).

C libraries:

  • Again, there are many XML parser libraries. Worth checking out are Expat, Gdome2, and libxml2.
  • A lot of people seem to roll their own C libraries for CSV parsing. I found documentation for a library, but it seems a little sketchy.
  • For fixed-width files, you can again use fprintf.
  • The best C regular expression library I know of is PCRE.

Hope this helps.

  • 2
    $\begingroup$ +1 Good summary, I as well prefer Python. I would add to the list the NetCDF4 module for Python. Netcdf is data format commonly used in geosciences, and besides Python, there are interfaces for this format for Fortran, C/C++ and Java. $\endgroup$ – milancurcic Jan 6 '12 at 23:29

When the main task is changing the shape or structure of data rather than reading a specific file format (like .mat or hfd5 for a matrix of the same shape), then I find Mathematica excellent for the task. Its native data structure is essentially a tree (list of lists of lists ...), and it can do flexible transformations on them using pattern matching or other means.

It has two weaknesses in my opinion:

  • If it doesn't already support the input/output file format, it's not the best tool to implement your own importer in (unless the format is trivial). Here's the listing of supported formats. (PDB is supported.)

  • It works well if all your data fits comfortably into memory. I find that it is not a very good tool for sequential data processing (i.e. reading a small bit of the input and outputting it immediately).

For kneading complex or structured data into the desired shape (think high dimensional ragged arrays or XML-like structured data), it is a very good tool though.

  • $\begingroup$ I think Mathematica is a fine tool for this task if you're post-processing data (which seems to be what you're suggesting in your answer), or your entire workflow is in Mathematica. If you need to process data as an intermediate step in a language other than Mathematica, Mathematica won't be so helpful. Furthermore, Mathematica is not free unless you belong to an institution that has a site license. $\endgroup$ – Geoff Oxberry Feb 15 '12 at 20:21
  • $\begingroup$ @Geoff The OP did not mention that he is looking for free tools only. I expect most readers here work in academia, so many will have access to Mathematica. It is the tool I use most often, and I tried to describe what it is useful for and what it isn't. It is one of my main tools for reshaping data. To be honest, I find it quite a turnoff that there is always some kind of negative feedback when one mentions a non-free tool. $\endgroup$ – Szabolcs Feb 15 '12 at 21:01
  • $\begingroup$ I agree that the OP didn't mention he was looking for free tools only. I also agree that it's frustrating to see people hash out the free versus non-free. I think commercial tools tend to be more polished and usable than their free counterparts, and I think that outright hostility to commercial products is misplaced. Nevertheless, paying for a license is a barrier to entry for some people, and one that I think is worth acknowledging. $\endgroup$ – Geoff Oxberry Feb 15 '12 at 21:58
  • 1
    $\begingroup$ There's not just "free" and "non-free". The "non-free" category splits into "affordable for most scientists" and "other". A Mathematica license is of the same order of magnitude of my annual working budget, so I need a very good reason to buy one. Most commercial programs are much cheaper. $\endgroup$ – khinsen Feb 16 '12 at 14:01

I aggree that Python is really good on this task. I would like to add some tools to the list:

  • If you want to extract data from the web, or maybe fill in a form easilly, then mechanize is the best option.
  • I heard that BeautifulSoup is really really good on parsing XML/HTML files.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.