@inproceedings{merx-etal-2024-generating,
    title = "Generating bilingual example sentences with large language models as lexicography assistants",
    author = "Merx, Raphael  and
      Vylomova, Ekaterina  and
      Kurniawan, Kemal",
    editor = "Baldwin, Tim  and
      Rodr{\'i}guez M{\'e}ndez, Sergio Jos{\'e}  and
      Kuo, Nicholas",
    booktitle = "Proceedings of the 22nd Annual Workshop of the Australasian Language Technology Association",
    month = dec,
    year = "2024",
    address = "Canberra, Australia",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest_wac_2008/2024.alta-1.5/",
    pages = "64--74",
    abstract = "We present a study of LLMs' performance in generating and rating example sentences for bilingual dictionaries across languages with varying resource levels: French (high-resource), Indonesian (mid-resource), and Tetun (low-resource), with English as the target language. We evaluate the quality of LLMgenerated examples against the GDEX (Good Dictionary EXample) criteria: typicality, informativeness, and intelligibility (Kilgarriff et al., 2008). Our findings reveal that while LLMs can generate reasonably good dictionary examples, their performance degrades significantly for lower-resourced languages. We also observe high variability in human preferences for example quality, reflected in low interannotator agreement rates. To address this, we demonstrate that in-context learning can successfully align LLMs with individual annotator preferences. Additionally, we explore the use of pre-trained language models for automated rating of examples, finding that sentence perplexity serves as a good proxy for {\textquotedblleft}typicality{\textquotedblright} and {\textquotedblleft}intelligibility{\textquotedblright} in higher-resourced languages. Our study also contributes a novel dataset of 600 ratings for LLM-generated sentence pairs, and provides insights into the potential of LLMs in reducing the cost of lexicographic work, particularly for low-resource languages."
}