5
$\begingroup$

What's the best practice for naming output files from scripts that you're experimenting with? I'm trying different parameters in my data generation script and I'd like to keep track of which files correspond to which parameters. If I use the standard way of naming with timestamps I soon lose track of how the files were created.

$\endgroup$
7
$\begingroup$

The problem of recording how a particular piece of scientific data was generated has come to be known as "provenance tracking". Here is an article on it which also includes the description of a software tool, Sumatra: http://rrcns.readthedocs.io/en/latest/provenance_tracking.html.

Another, simpler Python tool to do something similar is recipy.

$\endgroup$
2
$\begingroup$

This question might get closed for being opinion-based, but I think it's an important question ... so here's my opinion.

As far as I know, there's no real industry best practice for this kind of thing -- a lot of "don't"s, but not a lot of "do"s. For example, Stanford has a web page dedicated to file naming.

What's worked well for me is to create a series of folders whose names are numerical keys (e.g. "7001", "7002", "7003", ...), as well as a text file which gives a description of each folder (e.g. date, numerical parameters, motivation, brief summary of results, etc.).

I like this scheme because the folder names are short which makes it easy to find the one I want. Having a description of each folder in one text file makes it nice for finding results months later, and remembering why I did what I did.

$\endgroup$
  • $\begingroup$ I do something similar to you. One thing I do on top of that is make sure the description files are easy to parse so I can use separate algorithms to quickly search for and grab the output file I want so the process is less manual. $\endgroup$ – spektr May 11 '17 at 18:47
1
$\begingroup$

The best approach may depend on the problem at hand. Just make sure that: 1) it is well automated and 2) every detail is recorded. I have been naming files with date and most important parameters (or the parameters that are being changed).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.