I have a decision tree model -- that I've trained with some data.

I just got a new data set in and I want to run it through the model. Is there some way I can measure a "confidence interval" for the classes that new observations are being sorted into?


If you're using random forests -- and you should be; they're just better than decision trees, for most purposes -- then yes, you can measure a "confidence interval". A random forest is a collection of trees. Each tree gives you a prediction. The random forest classifier takes the majority vote (or average) of those predictions.

To get a confidence score, you can now look at the distribution of the predictions from all of the trees. If there are 1000 trees, each one has made one vote, so you can look at the distribution of the 1000 votes for that input.

  • For instance, if you're doing boolean classification, the random forests classifier will take the majority vote of those 1000 votes; but you can also calculate what fraction of those 1000 votes agree with the output of the classifier, and use that as a confidence score.

  • For instance, if you're doing regression, the random forests predictor may take the average or median of the 1000 values from the 1000 trees; but you can also sort those 1000 values, find the 5th and 95th quartiles, and use those as a sort of 90% "confidence interval".

Good libraries will do all this for you, and may also implement some more sophisticated methods.

There's lots more written about this in the literature on random forests. See, e.g., https://stats.stackexchange.com/q/56895/2921, https://stats.stackexchange.com/q/12425/2921, http://blog.datadive.net/prediction-intervals-for-random-forests/, http://blog.revolutionanalytics.com/2016/03/confidence-intervals-for-random-forest.html

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.