When to trust an AI model: New approach can improve uncertainty estimates
What does the IF-COMP technique approximate?
The IF-COMP technique approximates stochastic data complexity using a special function called an influence function. It combines influence functions and temperature-scaling to provide efficient, well-calibrated uncertainty quantifications that reflect a model's true confidence. This helps improve uncertainty estimates in machine-learning models and can be applied to large deep-learning models used in various real-world settings.
How does MDL quantify a model's uncertainty?
MDL (minimum description length) quantifies a model's uncertainty by considering all possible labels a model could assign to a test point2. If there are many alternative labels that fit well, the model's confidence in its chosen label decreases accordingly. MDL uses a short code to describe confidently labeled datapoints and a longer code to capture possibilities for uncertain ones. This is based on stochastic data complexity, which can be efficiently estimated using IF-COMP, an approximation technique developed by MIT researchers.
What principle underlies the new MIT uncertainty approach?
The new MIT uncertainty approach is based on the minimum description length principle (MDL), which enables better quantification and calibration of uncertainty for test points the model has been asked to label. This principle does not require the assumptions that can hamper the accuracy of other methods, and the developed technique, IF-COMP, makes MDL fast enough to use with large deep-learning models.