@inproceedings{petrov-etal-2025-q,
    title = "{Q}{\&}{A}-{LF} : A {F}rench Question-Answering Benchmark for Measuring Fine-Grained Lexical Knowledge",
    author = "Petrov, Alexander  and
      Mancas, Alessandra Thais  and
      Binet, Viviane  and
      Venant, Antoine  and
      Lareau, Francois  and
      Lepage, Yves  and
      Langlais, Phillippe",
    editor = "Angelova, Galia  and
      Kunilovskaya, Maria  and
      Escribe, Marie  and
      Mitkov, Ruslan",
    booktitle = "Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing - Natural Language Processing in the Generative AI Era",
    month = sep,
    year = "2025",
    address = "Varna, Bulgaria",
    publisher = "INCOMA Ltd., Shoumen, Bulgaria",
    url = "https://preview.aclanthology.org/corrections-2026-01/2025.ranlp-1.110/",
    pages = "962--969",
    abstract = "We introduce Q{\&}A-LF, a French, question-answering benchmark designed to assess the extent to which large language models capture fine-grained lexical knowledge. We investigate the ability of ChatGPT-4o mini, Qwen2.5-14B, Llama3.0-8B, and Llama3.1-8B to answer questions based on lexical functions from Meaning-Text Theory. Using various prompting setups with different levels of examples and context, we find that Qwen and ChatGPT generally outperform Llama models, achieving up to 70{\%} accuracy, while Llama models reach just above 60{\%}. We identify LFs that are particularly easy or especially challenging for the models. We further investigate whether providing sentence-level context and one-shot prompting improve performance, especially on semantically complex functions."
}