8
$\begingroup$

Sometimes it is useful in my work to modify someone else open-source code or find out how to develop specific things for your own application. However, not all software have good documentation.

What is a good way to understand the overall structure of a code base?

For example, which routines calls which routines, etc. I might use a documentation tool such as Doxygen for this purpose myself, however, I was wondering if there is a better strategy?

$\endgroup$
  • 5
    $\begingroup$ I think this is a reasonable question to ask, but I also think that it should probably be asked on programmers.SE instead; it's not strictly computational science, but more conceptual programming. $\endgroup$ – Geoff Oxberry Dec 17 '11 at 23:05
  • 2
    $\begingroup$ Are there techniques for doing this that are specific to scientific codes? $\endgroup$ – David Ketcheson Dec 18 '11 at 5:33
  • $\begingroup$ I use Doxygen because it is easy and can give nice pictures with graphs which shows how routines are connected. Note that it may be necessary to make a main routine which wraps about everything so get such a full picture. $\endgroup$ – Allan P. Engsig-Karup Dec 18 '11 at 8:25
  • $\begingroup$ @DavidKetcheson: I don't think there are methods for doing this specific to scientific codes. Given the strong tradition of undocumented research codes, perhaps there should be. $\endgroup$ – Geoff Oxberry Dec 18 '11 at 22:23
9
$\begingroup$

The following threads are tangentially related:

For the first part of my thesis, I spent 18 months modifying undocumented Fortran code, and one of the first tasks was to try and understand the overall structure of a code base. The most important thing I suggest you do is take notes in a text file any time you figure something out. You don't want to have to relearn or rediscover things during this time-consuming and frustrating process.

In my case, there was no "API" to speak of, in that the arguments of functions weren't self-documenting, because the previous programmer used Fortran 77-like style, and thus, short identifiers with little to no hint of what they meant. There were no tests, and since it's Fortran, no headers. Adding even more fun to the mix, there were a couple functions here and there written in C or C++.

Things that worked for me (assuming you work in Linux):

  • grep. Learn to love grep; you'll use it in the shell to find where functions are declared and called, and the output will tell you what files to look in.
  • The "find" command in your favorite code editor or IDE. Once you know what file to look in, the "find" command will save you time in hunting around for function calls.
  • Aggressive refactoring, if you can change the code base. I made the variable names self-documenting so that I didn't have to spend mental effort relearning things I'd already figured out. I also streamlined the design of the code as I figured it out so it was less confusing. No more 1000 line long main program!
  • Aggressive commenting. Again, if you can change the code base, comment things so that you know what you figured out. I didn't use Doxygen, but Doxygen is good for this.
  • nm. It can be useful on libraries if you don't have source code for them, but want to know if a function you encountered is in that library. However, it only works if the library's symbols haven't been stripped.
  • Step through the code with a debugger. It's way more efficient than using print statements. ddd and gdb are great, and on virtually every Linux system out there. Feel free to use your favorite debugger.
  • Bug the developers. This option is really best for very targeted questions. If you go to them (as I did), and say, "I don't understand what's going on here," they might take pity on you and try to explain things in general detail, but that's going to be of limited use in the long run. You're going to have to do the leg work, because the developers didn't do it for you in writing documentation and explaining the structure of their code base in writing. The developers are really good for when you're really stuck on small things (if they remember what they did).

Things I'd wish I'd thought of earlier, or just weren't options for me:

  • Doxygen. If you tweak the Doxyfile options, Doxygen will automatically generate a lot of documentation for you, even without the special Doxygen commenting syntax, which could be a good place to start; I've used this on later projects I encountered and it's been incredibly helpful.
  • Unit testing. If you can alter the code base, and you have a some idea of what it's supposed to do, write unit tests for various functions. (It's a useful skill to pick up regardless.)
  • If you're working with C/C++, look at the headers.
  • Write example programs. Not an option for me on that Fortran project, but it's been useful for me in picking up third-party APIs. Also, look at their example programs, if they have any.
  • Use gcov and lcov to do a coverage analysis on typical runs of the code, if you have examples or executables to work with. If there are examples that are supposed to execute large parts of the code base, these two tools combined will tell you how many times each line of code is visited. It's most useful with debugging flags enabled. If a part of the code isn't visited at all, it's probably less important to understand it right away. If a part of the code is visited a lot, it's probably worth understanding what it is. Maybe the code is executed a lot because it's an unimportant loop, or it could be a key function that a lot of other functions rely on. You can't tell just on the coverage analysis, but the coverage analysis gives you some idea of where to focus your time.
  • Static code analysis tools like splint can tell you if something fishy is going on in the code, like some variables are never used.
  • Profiling. Again, gives you data that doesn't immediately tell you what is or is not important, but suggests what might be important. If a lot of CPU time is spent calling one function, you may want to look at that and see what it does. You can also use the profiling output with dot and graphviz to generate call graphs and see how many times functions are called, like the coverage analysis. For complex codes, a graphical analysis could be a lot more helpful.
  • If you're working in C, Frama-C is supposed to be helpful in analyzing code, but I've never used it because it seemed too complicated to be worth the effort. I do some work in pure C, but it's mostly code that I write; I've never worked with undocumented C code.
$\endgroup$
3
$\begingroup$

I always tell my students to read a code from the bottom up: you start in main() and see what it calls. Typically this is only a small number of functions. Then you look into the functions called from main() which typically define the overall flow of the algorithm (time stepping loop, assembly, solver, output, etc). Go two or so levels deep to get an overview of the algorithm from 30,000 ft. The rest can often be gleaned from doxygen documentation etc.

But as I said, the message is: Read code from the bottom up.

$\endgroup$
  • 3
    $\begingroup$ Your advice is well taken, but doesn't sound like reading code from the bottom up at all. Rather, it sounds like reading from the top down. Main() (or Program, in Fortran) is the top-most program unit in an executable code. Furthermore, your advice requires a little tweaking when the code in question is a library that defines an API. There is no main() function, in which case you have to combine your strategy of seeing what functions call with deciding which functions matter for you, usually by looking at (or coding) example programs. $\endgroup$ – Geoff Oxberry Dec 22 '11 at 7:47
  • $\begingroup$ There may be some confusion about "from the bottom" because programers who learned Pascal early on often put the entry point at the end (or bottom) of the file---because Pascal made only one pass and everything had to be introduced before it was called. Like Geoff, however, I prefer the logic in which Main() is at the notional "top" of a tree that grows downward. $\endgroup$ – dmckee Dec 30 '11 at 3:12
  • $\begingroup$ Yes, I meant dmckee's way of having main() at the end of the file because one would otherwise have to forward-declare everything. $\endgroup$ – Wolfgang Bangerth Dec 31 '11 at 14:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.