How to Score Your Analytical Predictions Accurately

By Anasse Bari, Mohamed Chaouchi, Tommy Jung

When analyzing the quality of a predictive model, you’ll want to measure its accuracy. The more accurate a forecast the model makes, the more useful it is to the business, which is an indication of its quality. This is all good — except for when the predicted event is rare. In such case, the high accuracy of the predictive model may be meaningless.

For example if the probability of a rare event to occur is 5 percent, a model that simply answers “no” all the time when asked whether the rare event has occurred would be right 95 percent of the time. But how useful would such a model be?

Thus, if your business must deal routinely with rare events (if such a thing is possible), don’t rely on accuracy alone as a measure of your model’s reliability.

In such a case, you can evaluate the efficacy and the quality of a predictive model in the light of the how likely the rare event is to take place. A useful metric to follow is to specify which types of errors you can accept from the model and which you cannot.

Here’s a quick list of other ways to evaluate your model:

  • Check to see whether the model’s output meets your evaluation criteria.

  • Devise a testing strategy so you can test your model repeatedly and consistently.

  • Measure how well the model meets the business goals for which it was built.

  • Assess the risks of deploying the model live.

Help stamp out overfitting. When building a predictive model, keep in mind that your dataset is only a sample of the whole population. There will always be unknown factors that your data cannot account for, no matter what.

  • Approach the analysis of your predictive model with care, starting with this quick checklist:

  • Prepare your data with the utmost diligence before using it to train your model.

  • Carefully consider outliers before including or excluding them.

  • Remain vigilant in repeated testing and evaluation.

  • Cross-check sample data and test data to steer away from overfitting.

  • Consult your domain knowledge experts often and appropriately.