@inproceedings{meng-etal-2024-traffic,
    title = "Traffic Light or Light Traffic? Investigating Phrasal Semantics in Large Language Models",
    author = "Meng, Rui  and
      Liu, Ye  and
      Tu, Lifu  and
      He, Daqing  and
      Zhou, Yingbo  and
      Yavuz, Semih",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.findings-emnlp.503/",
    doi = "10.18653/v1/2024.findings-emnlp.503",
    pages = "8615--8622",
    abstract = "Phrases are fundamental linguistic units through which humans convey semantics. This study critically examines the capacity of API-based large language models (LLMs) to comprehend phrase semantics, utilizing three human-annotated datasets. We assess the performance of LLMs in executing phrase semantic reasoning tasks guided by natural language instructions and explore the impact of common prompting techniques, including few-shot demonstrations and Chain-of-Thought reasoning. Our findings reveal that LLMs greatly outperform traditional embedding methods across the datasets; however, they do not show a significant advantage over fine-tuned methods. The effectiveness of advanced prompting strategies shows variability. We conduct detailed error analyses to interpret the limitations faced by LLMs in comprehending phrase semantics. Code and data can be found at https://github.com/memray/llm{\_}phrase{\_}semantics/."
}