17
$\begingroup$

I need to write some data from a computation, that will be read later by Paraview (.vtu or vtk file).

When it comes to file size , should I go for the ASCII format or the Binary format ?

$\endgroup$
20
$\begingroup$

If your only worry is file size, then you want binary files. For an illustrative example, lets assume you are writing 1 double precision floating point number to a file. Let's assume that the file system can handle this perfectly and holding the file, headers, and padding are all 0.

For a binary file, that number would take the exact size of the number in RAM, or 8 bytes.

In ASCII format, it would hold:

  • 16 digits of the base
  • 1 period for the decimal
  • 1 char to delimit the exponent
  • 1 char for the sign of the exponent
  • 2-3 char for the exponent

Assuming it uses only 1 byte for a character, That is 22 bytes to hold the same number. This doesn't count the characters required to dilimit between numbers (usually atleast 1). Therefore file size for ASCII format will be about 3 times larger.

You can trade in file size for the precision in the stored files (only keep 5-6 digits in the base), but that depends on what you are using them for. The main advantage of ASCII is for debugging or producing human readable data.

$\endgroup$
  • 3
    $\begingroup$ Also important in the scientific arena is long-term archiving and reliable sharing, which is why, despite it's inefficiencies, ASCII CSV is so prevalent and recommended (PDF). $\endgroup$ – horchler Sep 3 '13 at 21:46
  • 2
    $\begingroup$ Another useful point is that although ASCII CSV encoding isn't very efficient, using a file compression utility (like zip, gzip, etc.) on your ascii file will typically bring the file size down to something similar to the size of a binary file. $\endgroup$ – Brian Borchers Sep 4 '13 at 1:30
  • 3
    $\begingroup$ Be careful because some input/output libraries aren't careful enough to get bit for bit reproducibility as you output IEEE Double Precision numbers in ASCII and then read them back in. In my experience, using 17 or 18 decimal digits is sometimes necessary for safety. $\endgroup$ – Brian Borchers Sep 4 '13 at 2:01
  • 5
    $\begingroup$ Concerning horchler's comment: I'm sure well-used, standardised open binary formats such as HDF5 will be around for a long time. That's what I'd personally recommend. $\endgroup$ – AlexE Sep 5 '13 at 9:13
  • 1
    $\begingroup$ + I stick to binary whenever possible, for accuracy, compactness, peace of mind, and (especially) speed. Then if I need further compactness, I can zip it. If I need to be able to visually read the contents, I can write a little program for that. On the other hand, if it's more important to be visual, and easily passed around to random programs like Excel, R, etc. then CSV is the way to go. $\endgroup$ – Mike Dunlavey Sep 9 '13 at 0:34
15
$\begingroup$

In practice, you rarely need data in visualization files that's more accurate than, say, 3 valid digits. In that case, ASCII is -- maybe surprisingly -- often more compact than binary form. If you're thinking about archiving, then bzip-ing these ASCII files is likely going to yield the smallest files you can get.

That said, Paraview reads VTU format which has a compressed binary form (XML-based, but the data is first libz-compressed and then uuencoded again to yield ASCII text). On typical files, this saves a factor of 4-10. For large files, this is definitely the way to go.

$\endgroup$
  • 2
    $\begingroup$ I voted this up for the contrast to the other answer. I don't have a strong opinion either way, but there's a good point to be had here. $\endgroup$ – Bill Barth Sep 4 '13 at 2:50
  • $\begingroup$ Alternatively, explicitly zero the low bits and compress the binary. $\endgroup$ – Jed Brown Sep 5 '13 at 3:51
  • $\begingroup$ Wow, that would require quite some bit-fiddling. Or are there functions that do that? (Other than casting to float and back to double.) $\endgroup$ – Wolfgang Bangerth Sep 5 '13 at 11:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.