Measures of parsimony for numerical models?

Question

There are hundreds of different types of performance measures for numerical models, many of which are applicable to many different types of models. But a good model doesn't just perform well, it performs well while being as simple as possible, and no simpler. Are there any general-ish measures of a model's parsimony?

In some fields you can build in some kind of ensured parsimony, like in evolutionary neural-network model generation, where you can punish complexity. I'm thinking more along the lines of physics-based models, where the model is to some extent defined by theory.

Answer 1

Not entirely sure this is applicable to your situation, but I think it may be. Statisticians use something called an information criterion for purposes very similar to this. It captures the trade off between good fit of the observations (in particular, the likelihood of the observations given the model and a setting of the parameters) and the complexity of the model (in particular, the number of fitted parameters).

There are several possible choices for this trade off, and therefore several different criteria. I think Akaike's information criterion (with or without sample size correction) and Bayes' information criterion are used most. Wikipedia has formulas and more background info.