@inproceedings{johnson-marasovic-2023-much,
    title = "How Much Consistency Is Your Accuracy Worth?",
    author = "Johnson, Jacob K.  and
      Marasovi{\'c}, Ana",
    editor = "Belinkov, Yonatan  and
      Hao, Sophie  and
      Jumelet, Jaap  and
      Kim, Najoung  and
      McCarthy, Arya  and
      Mohebbi, Hosein",
    booktitle = "Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2023.blackboxnlp-1.19/",
    doi = "10.18653/v1/2023.blackboxnlp-1.19",
    pages = "250--260",
    abstract = "Contrast set consistency is a robustness measurement that evaluates the rate at which a model correctly responds to all instances in a bundle of minimally different examples relying on the same knowledge. To draw additional insights, we propose to complement consistency with relative consistency{---}the probability that an equally accurate model would surpass the consistency of the proposed model, given a distribution over possible consistencies. Models with 100{\%} relative consistency have reached a consistency peak for their accuracy. We reflect on prior work that reports consistency in contrast sets and observe that relative consistency can alter the assessment of a model`s consistency compared to another. We anticipate that our proposed measurement and insights will influence future studies aiming to promote consistent behavior in models."
}