I need to write some data from a computation, that will be read later by Paraview (.vtu or vtk file).
When it comes to file size , should I go for the ASCII format or the Binary format ?
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
7
If your only worry is file size, then you want binary files. For an illustrative example, lets assume you are writing 1 double precision floating point number to a file. Let's assume that the file system can handle this perfectly and holding the file, headers, and padding are all 0.
For a binary file, that number would take the exact size of the number in RAM, or 8 bytes.
In ASCII format, it would hold:
- 16 digits of the base
- 1 period for the decimal
- 1 char to delimit the exponent
- 1 char for the sign of the exponent
- 2-3 char for the exponent
Assuming it uses only 1 byte for a character, That is 22 bytes to hold the same number. This doesn't count the characters required to dilimit between numbers (usually atleast 1). Therefore file size for ASCII format will be about 3 times larger.
You can trade in file size for the precision in the stored files (only keep 5-6 digits in the base), but that depends on what you are using them for. The main advantage of ASCII is for debugging or producing human readable data.
answered Sep 3, 2013 at 15:22
-
3$\begingroup$ Also important in the scientific arena is long-term archiving and reliable sharing, which is why, despite it's inefficiencies, ASCII CSV is so prevalent and recommended (PDF). $\endgroup$– horchlerCommented Sep 3, 2013 at 21:46
-
4$\begingroup$ Another useful point is that although ASCII CSV encoding isn't very efficient, using a file compression utility (like zip, gzip, etc.) on your ascii file will typically bring the file size down to something similar to the size of a binary file. $\endgroup$ Commented Sep 4, 2013 at 1:30
-
3$\begingroup$ Be careful because some input/output libraries aren't careful enough to get bit for bit reproducibility as you output IEEE Double Precision numbers in ASCII and then read them back in. In my experience, using 17 or 18 decimal digits is sometimes necessary for safety. $\endgroup$ Commented Sep 4, 2013 at 2:01
-
5$\begingroup$ Concerning horchler's comment: I'm sure well-used, standardised open binary formats such as HDF5 will be around for a long time. That's what I'd personally recommend. $\endgroup$– AlexECommented Sep 5, 2013 at 9:13
-
1$\begingroup$ + I stick to binary whenever possible, for accuracy, compactness, peace of mind, and (especially) speed. Then if I need further compactness, I can zip it. If I need to be able to visually read the contents, I can write a little program for that. On the other hand, if it's more important to be visual, and easily passed around to random programs like Excel, R, etc. then CSV is the way to go. $\endgroup$ Commented Sep 9, 2013 at 0:34
$\begingroup$
$\endgroup$
3
In practice, you rarely need data in visualization files that's more accurate than, say, 3 valid digits. In that case, ASCII is -- maybe surprisingly -- often more compact than binary form. If you're thinking about archiving, then bzip-ing these ASCII files is likely going to yield the smallest files you can get.
That said, Paraview reads VTU format which has a compressed binary form (XML-based, but the data is first libz-compressed and then uuencoded again to yield ASCII text). On typical files, this saves a factor of 4-10. For large files, this is definitely the way to go.
answered Sep 4, 2013 at 2:28
-
2$\begingroup$ I voted this up for the contrast to the other answer. I don't have a strong opinion either way, but there's a good point to be had here. $\endgroup$ Commented Sep 4, 2013 at 2:50
-
$\begingroup$ Alternatively, explicitly zero the low bits and compress the binary. $\endgroup$ Commented Sep 5, 2013 at 3:51
-
$\begingroup$ Wow, that would require quite some bit-fiddling. Or are there functions that do that? (Other than casting to float and back to double.) $\endgroup$ Commented Sep 5, 2013 at 11:45
$\begingroup$
$\endgroup$
tl;dr -- store files in utf8. If it's tabular, use TAB-separated values.
It seems to me that the correct options are:
- utf8 text (not ASCII. We're not all American English speakers)
- binary
ASAICT the only real advantage of binary files is performance. It's much faster to load a memory dump into memory than generate text on the way out or parse it on the way in.
Or here: https://auth0.com/blog/beating-json-performance-with-protobuf/ (This is not scientific, and is discussing overall performance, with the big differences being the amount of transmitted data and the parsing time in a case biased towards textual data.)
The chances that a binary file format will properly support unicode text is poor, so if you care about data integrity, don't use binary. Also have you heard about endian issues? Different binary representations of signed integers and floats?
Text representations of -100000 and -1e+6 do not change value depending on your CPU (in utf-8 and ASCII, at any rate).
The chances that a program that understands a binary file will still understand it or still run 50 years from now is unknown, probably not good. If you care about longevity, don't use binary.
It's often hard to read binary data from another program, so if you care about interoperability, don't use binary.
Aside: CSV is a terrible file format. It's simple, yet ill-defined and requires a stateful parser. Don't use CSV. If you must, use TSV. It's simpler but better defined, and trivial to parse.
https://chriswarrick.com/blog/2017/04/07/csv-is-not-a-standard/
https://www.cloudbakers.com/blog/everything-you-didnt-want-to-have-to-know-about-csv
If you're worried about size, compress.
(I came here looking for studies of the relative size efficiency of compressed binary representation vs. compressed text. I still haven't found good information aside from this study on VRML, but I'm not even sure if it's a comparison of base64 encoded binary vs. binary. https://www.cs.unc.edu/~isenburg/papers/is-bcraf-03.pdf.)
