Approximate a distribution function from a finite sample

Question

Say I have a simulation that produces a single floating point number as a result, and a different number is produced each time the simulation runs. These numbers are randomly distributed according to some unknown distribution. I would like to approximate the underlying density distribution function based on a finite sample of numbers.

I know that I could use a histogram, but the size of the bins, which is essentially arbitrary, will significantly affect the results. I would like a way to do this which has no "arbitrariness".

Any ideas?

Answer 1

Since a distribution function contains more information than the finite set of numbers you start with, you clearly have to add information in the process. This information comes in the form of a model that you assume, and whose parameters you adjust to make the model fit your finite sample. Unless the nature of your problem suggests some model, the choice will always be arbitrary to some extent.

With a histgram, your model is that of a piecewise constant function with the pieces (bins) having a fixed width. With Kernel Density Estimation, your model is the kernel. There is no way around choosing a model, the best you can do is to make an informed choice based on what you know (or can reasonably assume) about your data.

Answer 2

Kernel density estimation is a good suggestion. Another option is to construct the empirical CDF and then seek the distribution that best fits. Depending on the form of your output this might be more appropriate than a KDE, which is typically used for multimodal distributions. More detail on the empirical CDF here:

http://en.wikipedia.org/wiki/Empirical_distribution_function

Once you form the empirical CDF you can use a goodness of fit metric (http://en.wikipedia.org/wiki/Goodness_of_fit) or maximum likelihood estimation (http://en.wikipedia.org/wiki/Maximum_likelihood) to see which distribution type and parameter values best fit your data. The Pearson distribution is sometimes used in these situations if you have no good reason to assume the data follows one of the usual suspects due to its flexibility (http://en.wikipedia.org/wiki/Pearson_distribution).

Lastly, as you noted, you only have finite data. It can be important to understand the uncertainty in the best fit given the limited data, which can be done via Bayesian inference (http://en.wikipedia.org/wiki/Bayesian_inference). Properly implemented, this can account for both your model selection and the best-fit parameters given that model.

Answer 3

What you are looking for is Kernel Density Estimation.

Algorithms to carry this out are built into many scientific software packages and libraries. There is still a degree of arbitrariness in the choice of kernel and bandwidth, but there are heuristics for these.

Answer 4

I think you also need to decide what you are going to do with this distribution when you've got it. If you think you can sample from it instead of running your simulation to get random variates from your process then you then have to ask yourself some more questions about how important accuracy is, and how variations in your approximation to the true density will affect the outcome of sampling from an approximated density versus getting samples via the process...