I am charged with maintaining a buildserver on Teamcity which is meant to test our in house FE software. Currently our test suite consists of a list of benchmarks which run every time a commit is made using svn as version control software. The benchmarks are fully meshed boundary value problems intended to test various features of the software. The features are tested by running the head revision of the kernel on each benchmark and then comparing an output file with a reference file. The reference file is a result for a previous run of the test which was deemed "correct" by an expert/academic. The tests are thus considered to be "broken" when the result of this output file is different from that generated by the initial run.
I would like to improve upon this system by having our test suite run comparisons based on physical/analytical solutions for each benchmark rather than an arbitrary output file. However, in this case, how does one judge when a test is "broken"? The analytical solution can and will never be exactly the same as the numerical one, so comparing two files is out of the question. Does there exist a general method/quality control process perhaps based on statistics which can ensure that our code remains scientifically viable? How do major software companies like ANSYS or Bentley ensure that their scientific software does not deviate from exact solutions?
Please note I am in a commercial software development office setting and am looking for things that can be implemented in a straightforward manner within the Teamcity/tortoisesvn/Visual Studio environment.
Has anyone had such experience before?