Pandas limitations and its alternatives in Python

Question

I read somewhere that Pandas was first developed for the financial world, at least not especially for natural sciences (physics, biology and so on), so is there any similar data analysis Python package more “natural sciences oriented“?

I have just started using Pandas and I have already encountered two problems I couldn’t solve without using other packages or homemade solution:

• How to manage uncertainties?
• How to define easily the units of my data?

Maybe there are other issues but I lack experience to be more precise. For the moment, I think about the uncertainties package to solve the first point but I am not sure it will work fine with Pandas and will not decrease the computation speed. Actually, I am not looking for a way of computing with uncertainties, just a simple way of storing uncertainties along with my imported data. For the second point, I didn't find a better solution than creating a dictionary apart from my DataFrame to manage the units associated to each data.

If you have ever experienced the same issues, how did you solve it or which other package(s) do you use for data manipulation/storing/analysis in Python in natural sciences?

Answer 1

I agree with Davidmh that calculating uncertainties should not be handled by an automatic library. You will very quickly run into a case where the automatics fail (try doing a Fourier transform for instance).

You say however that you just want to keep the uncertainties with your data. Why not just add them as an extra column in your dataframe? This is how I typically manage uncertainties with pandas.

Pandas has no support of units, however anything can go into a dataframe so you could use the quantities package directly. Not all functionality will work in pandas however (though a surprising amount still will) and there will be a performance penalty.

There has been some discussion to allow for the attachment of metadata in pandas, but so far nothing seems to have come of it.

With the possible exception of C++11, I know of no language or library that would give you really good, first class unit support. There is always a loss of performance and a lack of compatibility

.

Answer 2

Managing uncertainties is actually a quite delicate statistics problem. The known expression for error propagation using squared partial derivatives is good when the errors are normally distributed, independent, and small. This is usually the case; and in fact, even if the normality or independence are not fully satisfied, for most practical cases the result could be reasonably close to the real one, provided you are only interested in an reasonable estimation of the interval.

Another posibility is to repeat the operation with the upper and the lower bound, so that $\sin(2\pm0.1) = \sin(2)^{\sin(2.1)}_{\sin(1.9)}$, but this is only correct if the intervals are hard thresholds (like the output of a non noisy instrument with accuracy 0.1).

For the cases where more accuracy is needed, if you have a nice analytical model for your uncertainties, there are methods, like maximum likelihood, that can be used to derive the correct estimators.

But if you want a correct result valid for any possible input (imagine having to analytically model a noisy detector where the noise is of the same order as the precision), taking into account all correlations, and being able to explore all the relevant parameter space, you need Monte Carlo methods. Add random noise to several of instances your inputs and run the full analysis. Of course, this may mean that you have to multiply your computational time by hundreds or thousands, but it is always parallelisable.

So, in the end, it all depends on what you want to do, to what degree of accuracy, and how many resources you have.