I need to write some data from a computation, that will be read later by Paraview (.vtu or vtk file).
When it comes to file size , should I go for the ASCII format or the Binary format ?
Computational Science Stack Exchange is a question and answer site for scientists using computers to solve scientific problems. It only takes a minute to sign up.Sign up to join this community
If your only worry is file size, then you want binary files. For an illustrative example, lets assume you are writing 1 double precision floating point number to a file. Let's assume that the file system can handle this perfectly and holding the file, headers, and padding are all 0.
For a binary file, that number would take the exact size of the number in RAM, or 8 bytes.
In ASCII format, it would hold:
Assuming it uses only 1 byte for a character, That is 22 bytes to hold the same number. This doesn't count the characters required to dilimit between numbers (usually atleast 1). Therefore file size for ASCII format will be about 3 times larger.
You can trade in file size for the precision in the stored files (only keep 5-6 digits in the base), but that depends on what you are using them for. The main advantage of ASCII is for debugging or producing human readable data.
In practice, you rarely need data in visualization files that's more accurate than, say, 3 valid digits. In that case, ASCII is -- maybe surprisingly -- often more compact than binary form. If you're thinking about archiving, then bzip-ing these ASCII files is likely going to yield the smallest files you can get.
That said, Paraview reads VTU format which has a compressed binary form (XML-based, but the data is first libz-compressed and then uuencoded again to yield ASCII text). On typical files, this saves a factor of 4-10. For large files, this is definitely the way to go.
tl;dr -- store files in utf8. If it's tabular, use TAB-separated values.
It seems to me that the correct options are:
ASAICT the only real advantage of binary files is performance. It's much faster to load a memory dump into memory than generate text on the way out or parse it on the way in.
Or here: https://auth0.com/blog/beating-json-performance-with-protobuf/ (This is not scientific, and is discussing overall performance, with the big differences being the amount of transmitted data and the parsing time in a case biased towards textual data.)
The chances that a binary file format will properly support unicode text is poor, so if you care about data integrity, don't use binary. Also have you heard about endian issues? Different binary representations of signed integers and floats?
Text representations of -100000 and -1e+6 do not change value depending on your CPU (in utf-8 and ASCII, at any rate).
The chances that a program that understands a binary file will still understand it or still run 50 years from now is unknown, probably not good. If you care about longevity, don't use binary.
It's often hard to read binary data from another program, so if you care about interoperability, don't use binary.
Aside: CSV is a terrible file format. It's simple, yet ill-defined and requires a stateful parser. Don't use CSV. If you must, use TSV. It's simpler but better defined, and trivial to parse.
If you're worried about size, compress.
(I came here looking for studies of the relative size efficiency of compressed binary representation vs. compressed text. I still haven't found good information aside from this study on VRML, but I'm not even sure if it's a comparison of base64 encoded binary vs. binary. https://www.cs.unc.edu/~isenburg/papers/is-bcraf-03.pdf.)