skip to main content
Language:
Search Limited to: Search Limited to: Resource type Show Results with: Show Results with: Search type Index

Stable reliability diagrams for probabilistic classifiers

Proceedings of the National Academy of Sciences - PNAS, 2021-02, Vol.118 (8) [Peer Reviewed Journal]

Copyright © 2021 the Author(s). Published by PNAS. ;Copyright © 2021 the Author(s). Published by PNAS. 2021 ;ISSN: 0027-8424 ;EISSN: 1091-6490 ;DOI: 10.1073/pnas.2016191118 ;PMID: 33597296

Full text available

Citations Cited by
  • Title:
    Stable reliability diagrams for probabilistic classifiers
  • Author: Dimitriadis, Timo ; Gneiting, Tilmann ; Jordan, Alexander I
  • Subjects: Physical Sciences
  • Is Part Of: Proceedings of the National Academy of Sciences - PNAS, 2021-02, Vol.118 (8)
  • Description: A probability forecast or probabilistic classifier is reliable or calibrated if the predicted probabilities are matched by ex post observed frequencies, as examined visually in reliability diagrams. The classical binning and counting approach to plotting reliability diagrams has been hampered by a lack of stability under unavoidable, ad hoc implementation decisions. Here, we introduce the CORP approach, which generates provably statistically consistent, optimally binned, and reproducible reliability diagrams in an automated way. CORP is based on nonparametric isotonic regression and implemented via the pool-adjacent-violators (PAV) algorithm-essentially, the CORP reliability diagram shows the graph of the PAV-(re)calibrated forecast probabilities. The CORP approach allows for uncertainty quantification via either resampling techniques or asymptotic theory, furnishes a numerical measure of miscalibration, and provides a CORP-based Brier-score decomposition that generalizes to any proper scoring rule. We anticipate that judicious uses of the PAV algorithm yield improved tools for diagnostics and inference for a very wide range of statistical and machine learning methods.
  • Publisher: United States: National Academy of Sciences
  • Language: English
  • Identifier: ISSN: 0027-8424
    EISSN: 1091-6490
    DOI: 10.1073/pnas.2016191118
    PMID: 33597296
  • Source: Geneva Foundation Free Medical Journals at publisher websites
    PubMed Central

Searching Remote Databases, Please Wait