Adjudicating LLMs as PropBank Adjudicators
Julia Bonn, Harish Tayyar Madabushi, Jena D. Hwang, Claire Bonial
Abstract
We evaluate the ability of large language models (LLMs) to provide PropBank semantic role label annotations across different realizations of the same verbs in transitive, intransitive, and middle voice constructions. In order to assess the meta-linguistic capabilities of LLMs as well as their ability to glean such capabilities through in-context learning, we evaluate the models in a zero-shot setting, in a setting where it is given three examples of another verb used in transitive, intransitive, and middle voice constructions, and finally in a setting where it is given the examples as well as the correct sense and roleset information. We find that zero-shot knowledge of PropBank annotation is almost nonexistent. The largest model evaluated, GPT-4, achieves the best performance in the setting where it is given both examples and the correct roleset in the prompt, demonstrating that larger models can ascertain some meta-linguistic capabilities through in-context learning. However, even in this setting, which is simpler than the task of a human in PropBank annotation, the model achieves only 48% accuracy in marking numbered arguments correctly. To ensure transparency and reproducibility, we publicly release our dataset and model responses.- Anthology ID:
- 2024.dmr-1.12
- Volume:
- Proceedings of the Fifth International Workshop on Designing Meaning Representations @ LREC-COLING 2024
- Month:
- May
- Year:
- 2024
- Address:
- Torino, Italia
- Editors:
- Claire Bonial, Julia Bonn, Jena D. Hwang
- Venues:
- DMR | WS
- SIG:
- Publisher:
- ELRA and ICCL
- Note:
- Pages:
- 112–123
- Language:
- URL:
- https://aclanthology.org/2024.dmr-1.12
- DOI:
- Cite (ACL):
- Julia Bonn, Harish Tayyar Madabushi, Jena D. Hwang, and Claire Bonial. 2024. Adjudicating LLMs as PropBank Adjudicators. In Proceedings of the Fifth International Workshop on Designing Meaning Representations @ LREC-COLING 2024, pages 112–123, Torino, Italia. ELRA and ICCL.
- Cite (Informal):
- Adjudicating LLMs as PropBank Adjudicators (Bonn et al., DMR-WS 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/2024.dmr-1.12.pdf