Thematic Fit Bits: Annotation Quality and Quantity Interplay for Event Participant Representation

Yuval Marton, Asad Sayeed

Abstract
Modeling thematic fit (a verb-argument compositional semantics task) currently requires a very large burden of labeled data. We take a linguistically machine-annotated large corpus and replace corpus layers with output from higher-quality, more modern taggers. We compare the old and new corpus versions’ impact on a verb-argument fit modeling task, using a high-performing neural approach. We discover that higher annotation quality dramatically reduces our data requirement while demonstrating better supervised predicate-argument classification. But in applying the model to psycholinguistic tasks outside the training objective, we see clear gains at scale, but only in one of two thematic fit estimation tasks, and no clear gains on the other. We also see that quality improves with training size, but perhaps plateauing or even declining in one task. Last, we tested the effect of role set size. All this suggests that the quality/quantity interplay is not all you need. We replicate previous studies while modifying certain role representation details and set a new state-of-the-art in event modeling, using a fraction of the data. We make the new corpus version public.
Anthology ID:
2022.lrec-1.556
Volume:
Proceedings of the Thirteenth Language Resources and Evaluation Conference
Month:
June
Year:
2022
Address:
Marseille, France
Editors:
Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
5188–5197
Language:
URL:
https://aclanthology.org/2022.lrec-1.556
DOI:
Bibkey:
Cite (ACL):
Yuval Marton and Asad Sayeed. 2022. Thematic Fit Bits: Annotation Quality and Quantity Interplay for Event Participant Representation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5188–5197, Marseille, France. European Language Resources Association.
Cite (Informal):
Thematic Fit Bits: Annotation Quality and Quantity Interplay for Event Participant Representation (Marton & Sayeed, LREC 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/naacl-24-ws-corrections/2022.lrec-1.556.pdf
Data
FrameNet, Penn Treebank
PDF Search