Abstract
We report error analysis of outputs from seven Table-to-Text generation models fine-tuned on ToTTo, an open-domain English language dataset. A manual error annotation of a subset of outputs (a total of 5,278 sentences) belonging to the topic of Politics generated by these seven models has been carried out. Our error annotation focused on eight categories of errors. The error analysis shows that more than 45% of sentences from each of the seven models have been error-free. It uncovered some of the specific classes of errors such as WORD errors that are the dominant errors in all the seven models, NAME and NUMBER errors are more committed by two of the GeM benchmark models, whereas DATE-DIMENSION and OTHER category of errors are more common in our Table-to-Text models.- Anthology ID:
- 2022.gem-1.43
- Volume:
- Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates (Hybrid)
- Venue:
- GEM
- SIG:
- SIGGEN
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 456–470
- Language:
- URL:
- https://aclanthology.org/2022.gem-1.43
- DOI:
- Cite (ACL):
- Barkavi Sundararajan, Somayajulu Sripada, and Ehud Reiter. 2022. Error Analysis of ToTTo Table-to-Text Neural NLG Models. In Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 456–470, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
- Cite (Informal):
- Error Analysis of ToTTo Table-to-Text Neural NLG Models (Sundararajan et al., GEM 2022)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/2022.gem-1.43.pdf