@inproceedings{kranzlein-etal-2021-making-heads,
    title = "Making Heads and Tails of Models with Marginal Calibration for Sparse Tagsets",
    author = "Kranzlein, Michael  and
      Liu, Nelson F.  and
      Schneider, Nathan",
    editor = "Moens, Marie-Francine  and
      Huang, Xuanjing  and
      Specia, Lucia  and
      Yih, Scott Wen-tau",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2021",
    month = nov,
    year = "2021",
    address = "Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2021.findings-emnlp.423/",
    doi = "10.18653/v1/2021.findings-emnlp.423",
    pages = "4919--4928",
    abstract = "For interpreting the behavior of a probabilistic model, it is useful to measure a model{'}s calibration{---}the extent to which it produces reliable confidence scores. We address the open problem of calibration for tagging models with sparse tagsets, and recommend strategies to measure and reduce calibration error (CE) in such models. We show that several post-hoc recalibration techniques all reduce calibration error across the marginal distribution for two existing sequence taggers. Moreover, we propose tag frequency grouping (TFG) as a way to measure calibration error in different frequency bands. Further, recalibrating each group separately promotes a more equitable reduction of calibration error across the tag frequency spectrum."
}