Why Optimizing Proper Loss Functions is Important for AI Predictors?
When it comes to developing AI predictors, optimizing proper loss functions is crucial. This optimization process is believed to result in predictors with excellent calibration properties. The idea behind this belief is that by using proper loss functions, the predictors are designed to predict ground-truth probabilities accurately, leading to calibration.
Challenges in Optimizing Proper Loss Functions
However, the reality is that most machine learning models are trained to minimize loss over specific families of predictors. These restricted families are unlikely to contain the ground truth. This raises questions about the circumstances in which optimizing proper loss over a restricted family can yield calibrated models. We also need to understand the specific calibration guarantees it provides.
A Solution to the Problem
In this work, we provide a comprehensive answer to these questions. Instead of aiming for global optimality, we introduce a local optimality condition. This condition states that the predictor’s (proper) loss cannot be significantly reduced by post-processing its predictions with a specific family of Lipschitz functions. Any predictor that satisfies this local optimality condition is shown to exhibit smooth calibration, as defined in Kakade-Foster (2008) and Błasiok et al. (2023).
It is worth noting that well-trained DNNs are likely to satisfy the local optimality condition. This provides an explanation as to why they are calibrated without relying on any other loss minimization technique. Furthermore, our research reveals that the connection between local optimality and calibration error works both ways. Predictors that are nearly calibrated also tend to be nearly locally optimal.
In conclusion, optimizing proper loss functions is essential in developing AI predictors with good calibration properties. Our work sheds light on the relationship between local optimality and calibration, offering a better understanding of how to achieve accurate predictions in the field of artificial intelligence.