31

If you have some experience with Python (or even not), I would recommend using the Python scientific software that is available (SciPy,Pandas),...) together with Matplotlib. Being a programming environment, you have full control over your data flows, data manipulations and plotting. You can also use the "full applications" Mayavi2 or Veusz.


15

Based on what you're saying, a general purpose scripting language is probably your best bet, as long as it has some sort of graphing capability that you can access (whether built-in or imported). In that vein, MATLAB will work, although you'll have to toy with line widths, symbols, and axes for presentation-quality plots. Given your criteria, I'd say the ...


12

Consider using the HDF5 file format. HDF5 is a hierarchical data storage format with several nice features: platform independent storage: the library takes care of little/big endianness for you hierarchical layout of datasets: like a filesystem within a file large, growable n-dimensional array storage mixed dataset types can exist within one file (ie, ...


10

I think you might have to split your output to match your targets: for the movies of properties, you probably don't need the full spatial resolution and all variables. Carefully choose what you want to show and think about the final resolution of the movie you will be showing, it probably won't have 8 billion pixels. For the Fourier analyses (or things like ...


10

I think this is an excellent question, and one that is at the forefront of my own mind right now. When reading most math-ish journals, Matlab seems to be acceptable, but it just doesn't measure up IMHO to some of the beautiful plots and diagrams that appear in journals like PNAS, Nature, PLoS ONE, etc. After discussing exactly this issue in my research ...


9

You're going to get a much better answer if you provide a few more technical details about what kind of data you're trying to put under version control, how you want to store different versions of the data, what components are likely to change and what components aren't, and whether you're truly going to have tree-like history (branches, merges). HDF5 files ...


8

Let me write a short review of Mathematica's graphics functionality, since that is what I'm familiar with. What can it do? To see some example, check out the Mathematica code gallery on the Wolfram site. It's not specifically about visualization, but almost all examples include some plotting. You can also browse the plotting (basic) and graphics (more ...


8

I think the current masters of this art are the large particle physics experiments (I am most familiar with CDF and D0 because I am old and work at the University of Chicago). They have hardware triggers that discard petabytes (or more) a year. However, this is the whole subject of quantization/discretization, or "throwing away only what you don't need". I ...


8

I generally use gnuplot. It can be hard to learn, but it produces very clean plots. It can also be set to produce plots in encapsulated postscript format (.eps) which allows for easy embedding into documents written in $\LaTeX$.


8

Another software package to consider is Asymptote. Asymptote is actually a programming language (with a C++ like syntax) that produces vector based output. One nice feature is that text is rendered with LaTeX (including math equations), so the text in your figures is consistent with your document. I wrote a Python wrapper to Asymptote called PyAsy so that ...


8

I would perhaps argue that the main virtue of XML is the ease with which one can write a parser, rather than the ease with which you can write an actual XML document. JSON seems a slightly better alternative to my mind. Both have the advantage of being standards to some degree, meaning people won't have to learn whichever arcane syntax your program of choice ...


6

If you want to remove the nonmonotonic structure without changing anything else, you can do an isotonic least squares fit: http://en.wikipedia.org/wiki/Isotonic_regression


6

Histograms are not useful for high dimensional data. The curse of dimensionality affects one quite fast. As in your case if the grid is of size 7**6, you have on average one point in one bin. Kernel density estimator are better suited as long as you keep the kernel bandwidth large enough. In my experience the top hat kernel as k-nearest neighbor yields ...


5

In molecular dynamics simulations we have the same problem: given a cutoff radius $r_c$, find all particles within at most $r_c$ of each other. The simplest $\mathcal O(n)$ approach is to divide the space into cells of edge length of at least $r_c$ and to compare every particle in every cell to all particles in the 26 neighboring cells (in three dimensions, ...


5

If you already know TikZ, you might be interested in R and the tikzDevice; here is an example: tikzDevice - TikZ output from R, here is another example: TikZ diagrams with R: tikzDevice.


5

Dig deeper into PGF plots: 1) can do 3d diagrams: python-matplotlib can't do them, gnuplot is ugly in comparison 2) has consistent fonts in size and family with the base document ) 3) can be templated for specific colorschemes There is nothing else like it on the market. :)


5

TLDR Use Python to manage/modify your input and coral your output, and use HDF5 to organize/store your data. As complex as it might seem at first it'll still be simpler than SQL-anything. Longer answer + Example I personally use a combination of Python scripting and the HDF5 file format to deal with these kinds of situations. Python scripting can handle the ...


5

I agree with Davidmh that calculating uncertainties should not be handled by an automatic library. You will very quickly run into a case where the automatics fail (try doing a Fourier transform for instance). You say however that you just want to keep the uncertainties with your data. Why not just add them as an extra column in your dataframe? This is how I ...


4

I find Tecplot to be quite powerful when it comes to data visualization and data manipulation.


4

My favorite language for these tasks is Python; Perl also works for those who are so inclined. Python modules: The csv module from the Python standard library handles parses CSV files. The Python standard library also has modules for standard markup languages like XML, HTML, SGML, etc. For fixed-width strings, see this post on Stack Overflow. I haven't ...


4

If you are interested in a "standardized" format, you should have a look at the matrix market exchange formats. The "coordinate format" (which is good for sparse matrices) simply adds metadata to the format suggested by Aron in his answer and specifies how the data has to be formatted.


4

I'll try to give you my view on my limited experience, which only covers a few type of plots: 2D data plots, scatterplots, and graph-based diagrams (trees and graphs, flowcharts); but first allow me to deviate from the question for a bit: First, I would like to say that the importance of producing high quality plots is often overlooked. Plots are not only ...


4

The most common case is that you cannot allocate a memory block larger than what is physically available for your GPU card. However, depending on your hardware and driver, the maximum allocatable size might be a fraction of the total physically installed memory (e.g. old AMD opencl drivers only allowed a maximum of 500 MB), so it is better if you read into ...


4

Numpy has a file format that is pretty simple, which makes it perfectly compatible with basically every other high level language. (https://www.numpy.org/devdocs/reference/generated/numpy.lib.format.html) It looks like the format is much lighter than boost or hdf5. The docs say it should be easy enough to write a parser yourself, if necessary, which I would ...


3

Interoperable Technologies for advanced Petascale Scientific Computing, or ITAPS, has defined several interfaces in an effort to standardize data communication between large-scale software components. In particular, take a look at Tim Tautges' MOAB software package, which implements the ITAPS iMesh interface.


3

I'm surprised nobody has mentioned Nico Schlömer's excellent tools matlab2tikz and matplotlib2tikz yet. If you are using LaTeX for document preparation and either Matlab or Python for data processing, you can easily get high quality vector plots which you can post process to your heart's content: Prepare your plots in Matlab or Python, including axes, ...


3

My quick vote is for QtiPlot. Though it isn't perfect, it provides the best combination of GUI easiness, along with python scripting so what you're really doing then is "Qtiplot/Python/(Illustrator or Inkscape)". Qtiplot is extensible, so you can create your own scripts/macros which can run python commands to open and process data and then dump them into ...


3

The question is a bit broad, so I will provide a correspondingly vague answer that suggests possible techniques in such cases. 1) On-the-fly processing, which you are already working on. One way to do on-the-fly processing and yet decouple it from the data-generating step is to generate a cyclic output file that always contains the last N steps, and have ...


3

Peter LePage is pretty famous in lattice-QCD circles for suggesting a method whereby unfeasibly-large lattice grids could be reduced by finding and apply good short ranges analytic solutions. This is roughly equivalent to noticing that a set of well chosen splines can allow accurate integration with fewer knots than the trapezoid method (except that as in ...


3

Actually, I found the answer here . Thanks for the help though...


Only top voted, non community-wiki answers of a minimum length are eligible