0
$\begingroup$

I would like to store text data (matrix with numbers and letters) in a light way (light volume for easy exchange, and light to open it).

Question: are binary files lighter than txt? is it always true? And why?

$\endgroup$
3
  • 4
    $\begingroup$ What does "light" actually mean? Do you mean "amount of bytes" or "easy to read to humans" or "easy to read by computers" or "easy to exchange between platforms"? So much ambiguity :-) $\endgroup$ – Wolfgang Bangerth Aug 24 '20 at 15:58
  • $\begingroup$ Are you willing/able to compress the text files? $\endgroup$ – Brian Borchers Aug 25 '20 at 5:21
  • $\begingroup$ THX for your highlights. My target consists in having a light document (in storage size), without too much transformations like compression. $\endgroup$ – lelorrain7 Aug 27 '20 at 14:06
7
$\begingroup$

Purely size-wise, it is better to use binary. In ASCII representation, full expansion of a double is about 16+4 characters -depending on how you want to represent them size changes slightly- which is (at minimum) 20 bytes while in binary format it would take 8 bytes. That is more than half storage gain. Depending on how your binary formatting, I/O may also be few magnitudes faster. In my experience, for example, PETSc binary files are about 15 times faster to read when the file sizes are bigger than 200-300 MBs.

The problem is the documentation. Every one can look at an ASCII file and probably can figure out the structure and code a program to read it. Unfortunately, same is not true for binary files. You have to know the format ahead of time and even then it may not be easy to write a program to read it. Hence portability is big issue. A little bit less known problem is the treatment of numbers. It is 2020 and we like to think that everything is standardized but sometimes it is not. I had an issue like that when I tried to read a custom binary file and thought "long double is long double". It ended up being not so much, but compiler dependent.

As asked in the comments, you should clarify your goals and question so people can provide a better answer.

$\endgroup$
1
  • $\begingroup$ Thank you, your answer is the information I was looking for. interesting to know that binary (the elemental code) is not standard :D $\endgroup$ – lelorrain7 Aug 27 '20 at 14:09
2
$\begingroup$

If file size is an issue, you can use the zlib library functions gzopen, gzwrite, gzread, and gzclose to directly write and read the text to a compressed file.

Apart from possibly saving even more space than writing numbers in binary format, it has the advantage that the file can be uncompressed from the command line with gzip to have a look at its content, which avoids the documentation and compiler dependency problems mentioned in the answer by Abdullah Ali Sivas.

$\endgroup$
2
$\begingroup$

A relevant anecdote: Years ago I realized that the fellow who sat next to me in my church choir had played a role in defining the SGML standard. Sometime after that he started a conversation by asking, in a half-serious, sales-pitchy way, "In your experiment, do you store your data in a self-documenting text-based representation?"

I thought about it for a minute. I thought about what a hassle it had been to write tools to read our binary data files, and how many hours I had spent debugging our data acquisition hardware and software by frowning at a hexdump like a movie hacker. I thought about how nice it would be to have data files that I could just read. But what I said to him was, "Two hundred megabytes a minute, and we'll run for a year."

"Oh," he said, and then in a quieter voice, "oh, my."

What I recall about the data analysis for that experiment is that the bottleneck was disk I/O: mostly the CPU sat idle, waiting for the data to come in from the drive. We experimented with storing gzipped data files, but we had designed the hardware so that there really wasn't much compressible low-entropy information in the data stream. Also there was some part of the analysis where it was convenient to seek through the data in a logical order that was different from the hardware order; seeking was trivial in a binary file with a bunch of fixed-width records, but seeking on the output of a gzip pipe was expensive or impossible.

My experience has generally been that the most expensive part of a computation is figuring out how to do it, and therefore that the clarity of having a human-readable data file is nearly always worth the computational cost of having to wait for some parser turn those text characters into numbers. But that advantage goes away as the fraction of the data which will ever actually be looked at by a human shrinks.

A method that has served me well has been to build the first version of my datafile/analysis pipeline in the way that's the easiest to write, and only worry about fixing inefficiencies if the delays become annoying before the problem is solved.

$\endgroup$
1
  • 1
    $\begingroup$ en.wikipedia.org/wiki/OpenFOAM, an open source CFD library, actually lets the user decide whether to store the big data files in ASCII or binary. Thus, when you are fiddling with your code, switch to ASCII to help yourself in the troubleshooting. If everything is ready, simply switch to binary. Well thought out by the developers. $\endgroup$ – Dohn Joe Aug 31 '20 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.