Loss functions are fundamental to supervised classification, significantly impacting model learning efficiency and performance. While cross-entropy (CE) is widely adopted, the mean absolute error (MAE) loss is recognized for its robustness, despite presenting optimization challenges.
Generalized Cross-Entropy (GCE) was recently introduced to strike a balance between optimization difficulty and robustness, interpolating between CE and MAE. However, existing GCE formulations often lead to a non-convex optimization over classification margins, increasing the risk of underfitting and resulting in suboptimal performance on complex datasets.
Addressing these limitations, a novel Minimax Generalized Cross-Entropy (MGCE) formulation has been proposed. This new approach transforms the optimization over classification margins into a convex problem, mitigating the issues associated with non-convexity such as local optima and underfitting. Furthermore, the research demonstrates that MGCE can provide a crucial upper bound on the classification error.
The proposed bilevel convex optimization can be efficiently implemented using stochastic gradient computation, facilitated by implicit differentiation. Experimental evaluations on benchmark datasets confirm MGCE's superior performance, showcasing strong accuracy, faster convergence rates, and enhanced model calibration. Its robustness is particularly evident in the presence of label noise, marking a significant advancement for developing more resilient AI models.