Accurate Layerwise Interpretable Competence Estimation
Estimating machine learning performance “in the wild” is both an important and unsolved problem. In this paper, we seek to examine, understand, and predict the pointwise competence of classification models. Our contributions are twofold: First, we establish a statistically rigorous definition of competence that generalizes the common notion of classifier confidence; second, we present the ALICE (Accurate Layerwise Interpretable Competence Estimation) Score, a pointwise competence estimator for any classifier. By considering distributional, data, and model uncertainty, ALICE empirically shows accurate competence estimation in common failure situations such as class-imbalanced datasets, out-of-distribution datasets, and poorly trained models. Our contributions allow us to accurately predict the competence of any classification model given any input and error function. We compare our score with state-of-the-art confidence estimators such as model confidence and Trust Score, and show significant improvements in competence prediction over these methods on datasets such as DIGITS, CIFAR10, and CIFAR100.
@incollectionNIPS2019_9548 title: Accurate Layerwise Interpretable Competence Estimation author: Rajendran Vickram and LeVine William booktitle: Advances in Neural Information Processing Systems 32 editor: H. Wallach and H. Larochelle and A. Beygelzimer and F. d extquotesingle Alch'e-Buc and E. Fox and R. Garnett pages: 13981--13991 year: 2019 publisher: Curran Associates Inc. url: http://papers.nips.cc/paper/9548-accurate-layerwise-interpretable-competence-estimation.pdf