Abstract
We present LQVSumm, a corpus of about 2000 automatically created extractive multi-document summaries from the TAC 2011 shared task on Guided Summarization, which we annotated with several types of linguistic quality violations. Examples for such violations include pronouns that lack antecedents or ungrammatical clauses. We give details on the annotation scheme and show that inter-annotator agreement is good given the open-ended nature of the task. The annotated summaries have previously been scored for Readability on a numeric scale by human annotators in the context of the TAC challenge; we show that the number of instances of violations of linguistic quality of a summary correlates with these intuitively assigned numeric scores. On a system-level, the average number of violations marked in a system’s summaries achieves higher correlation with the Readability scores than current supervised state-of-the-art methods for assigning a single readability score to a summary. It is our hope that our corpus facilitates the development of methods that not only judge the linguistic quality of automatically generated summaries as a whole, but which also allow for detecting, labeling, and fixing particular violations in a text.- Anthology ID:
- L14-1467
- Volume:
- Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)
- Month:
- May
- Year:
- 2014
- Address:
- Reykjavik, Iceland
- Editors:
- Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, Stelios Piperidis
- Venue:
- LREC
- SIG:
- Publisher:
- European Language Resources Association (ELRA)
- Note:
- Pages:
- 1591–1599
- Language:
- URL:
- http://www.lrec-conf.org/proceedings/lrec2014/pdf/578_Paper.pdf
- DOI:
- Cite (ACL):
- Annemarie Friedrich, Marina Valeeva, and Alexis Palmer. 2014. LQVSumm: A Corpus of Linguistic Quality Violations in Multi-Document Summarization. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 1591–1599, Reykjavik, Iceland. European Language Resources Association (ELRA).
- Cite (Informal):
- LQVSumm: A Corpus of Linguistic Quality Violations in Multi-Document Summarization (Friedrich et al., LREC 2014)
- PDF:
- http://www.lrec-conf.org/proceedings/lrec2014/pdf/578_Paper.pdf