Let's say that I have a numerical code and I'm doing convergence testing on the code. The code fails the convergence test, that is, the order of convergence is not as expected.

Checking the order of convergence of a numerical code is an integration test, so the test failing does not tell a developer much about where the cause of the test failure might be. That's because an integration test looks at an entire system and not a particular component. What can I do to locate the bug?

Unit tests are my first thought, but for the particular case motivating the question, I have good code coverage and many unit tests. I've identified all the functions called during the integration test, but I can't seem to find anything clearly wrong with the functions. I'm suspecting there is a bug or gap in my tests.

I might be able to add more assertions to the code to see if that finds something off.

Another possibility is that the zeroth order error I'm seeing is false. About a decade ago, when I was doing convergence testing of a different software, the convergence test showed a zeroth order error. A coworker suggested increasing the precision from single to double for the convergence test. That made the test pass. In this case, I've confirmed that the convergence plots are independent of the precision.

I've also found that the code converges correctly for some particular cases, which does help narrow down where the bug is. So trying different integration tests has helped, though not enough to isolate the bug.

  • 5
    $\begingroup$ Have you looked at the method of manufactured solutions? Depending on the numerical method you use, it might be easily applicable in its original form/require minor modifications or completely irrelevant. $\endgroup$
    – Anton Menshov
    Commented Apr 3 at 16:27
  • 1
    $\begingroup$ In addition to integral tests, it may be beneficial to look at pointwise comparisons of a computed an exact solution (obtained via MMS as Anton suggests). This may help to identify problematic regions corresponding to, e.g., incorrectly implemented boundary conditions. $\endgroup$
    – whpowell96
    Commented Apr 3 at 18:19
  • $\begingroup$ I have a bunch of exact solutions that I'm using. (Or at least I think they are exact!) No MMS needed in this case. Pointwise checks are a good idea in general, but not helpful in my case. I've produced a greatly simplified code which converges to an exact solution properly, so I think from here I'm just going to add features until something breaks. I should have started using convergence testing sooner. $\endgroup$ Commented Apr 3 at 22:16
  • 1
    $\begingroup$ In general it's a good idea to have some flags that allow you to turn off a certain portion of your code to disable certain operators or source terms of your PDE. You might even find out that the unexpected results come from the combined effect of two or more pieces of your solver, even if they would work properly when used alone. $\endgroup$
    – Rigel
    Commented Apr 4 at 17:13
  • 1
    $\begingroup$ Perhaps there is something to be learned from W. Rider's chapter on "The Foundations of Verification in Modeling and Simulation". The rest of the book is also interesting. $\endgroup$
    – IPribec
    Commented Apr 16 at 13:44

1 Answer 1


In this answer, I'll say what I did to fix the bug, though I'm not going to mark this as the answer as I want to hear other advice.

What I did to fix the bug was rewrite the solver gradually, focusing only on the subset of the solver that was having issues. (Other convergence tests which did not involve a certain feature passed.) I started with a simplified problem, doing a convergence test for that. Then I added a new complexity to the problem and did a convergence test. I repeatedly added complexities and tested each stage with its own convergence test. Eventually I built up a working solver focused on the subset of the problem that had issues. This converged at the expected rate. This gradual development pattern was what I should have done in the first place, and I'll have to keep this in mind for the future. This will allow one to identify where bugs are. If in one stage a convergence test fails, but the previous stage's convergence test passed, you know the code added in the new stage is responsible.

I used exact solutions for each stage in this case, but you can use the method of manufactured solutions if exact solutions are not possible.

With the new solver focused on a subset of the problem, I was able to copy and paste a function I suspected was the source of the bug into the old solver's code. The old solver then passed all convergence tests. I don't know precisely what was wrong with the old solver, as the formulation stayed the same in both cases, and I had rewritten the function that I suspected was the source of the problem multiple times before the gradual rewrite! It appears that the bug was subtle and gradually developing that function was necessary to make it work.

This all may seem obvious to some, but it wasn't obvious to me, so I figure it's worth documenting.

  • 2
    $\begingroup$ Thank you for following up! $\endgroup$
    – MPIchael
    Commented Apr 15 at 5:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.