@inproceedings{dev-etal-2023-building,
    title = "Building Stereotype Repositories with Complementary Approaches for Scale and Depth",
    author = "Dev, Sunipa  and
      Jha, Akshita  and
      Goyal, Jaya  and
      Tewari, Dinesh  and
      Dave, Shachi  and
      Prabhakaran, Vinodkumar",
    editor = "Dev, Sunipa  and
      Prabhakaran, Vinodkumar  and
      Adelani, David Ifeoluwa  and
      Hovy, Dirk  and
      Benotti, Luciana",
    booktitle = "Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP)",
    month = may,
    year = "2023",
    address = "Dubrovnik, Croatia",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2023.c3nlp-1.9/",
    doi = "10.18653/v1/2023.c3nlp-1.9",
    pages = "84--90",
    abstract = "Measurements of fairness in NLP have been critiqued for lacking concrete definitions of biases or harms measured, and for perpetuating a singular, Western narrative of fairness globally. To combat some of these pivotal issues, methods for curating datasets and benchmarks that target specific harms are rapidly emerging. However, these methods still face the significant challenge of achieving coverage over global cultures and perspectives at scale. To address this, in this paper, we highlight the utility and importance of complementary approaches that leverage both community engagement as well as large generative models, in these curation strategies. We specifically target the harm of stereotyping and demonstrate a pathway to build a benchmark that covers stereotypes about diverse, and intersectional identities. We discuss the two approaches, their advantages and constraints, the characteristics of the data they produce, and finally, their potential to be used complementarily for better evaluation of stereotyping harms."
}