Abstract
The generation of referring expressions (REs) is a non-deterministic task. However, the algorithms for the generation of REs are standardly evaluated against corpora of written texts which include only one RE per each reference. Our goal in this work is firstly to reproduce one of the few studies taking the distributional nature of the RE generation into account. We add to this work, by introducing a method for exploring variation in human RE choice on the basis of longitudinal corpora - substantial corpora with a single human judgement (in the process of composition) per RE. We focus on the prediction of RE types, proper name, description and pronoun. We compare evaluations made against distributions over these types with evaluations made against parallel human judgements. Our results show agreement in the evaluation of learning algorithms against distributions constructed from parallel human evaluations and from longitudinal data.- Anthology ID:
- 2022.lrec-1.320
- Volume:
- Proceedings of the Thirteenth Language Resources and Evaluation Conference
- Month:
- June
- Year:
- 2022
- Address:
- Marseille, France
- Editors:
- Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, Stelios Piperidis
- Venue:
- LREC
- SIG:
- Publisher:
- European Language Resources Association
- Note:
- Pages:
- 2989–2997
- Language:
- URL:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2022.lrec-1.320/
- DOI:
- Cite (ACL):
- T. Mark Ellison and Fahime Same. 2022. Constructing Distributions of Variation in Referring Expression Type from Corpora for Model Evaluation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2989–2997, Marseille, France. European Language Resources Association.
- Cite (Informal):
- Constructing Distributions of Variation in Referring Expression Type from Corpora for Model Evaluation (Ellison & Same, LREC 2022)
- PDF:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2022.lrec-1.320.pdf