Summarization Evaluation in the Absence of Human Model Summaries Using the Compositionality of Word Embeddings
Elaheh ShafieiBavani, Mohammad Ebrahimi, Raymond Wong, Fang Chen
Abstract
We present a new summary evaluation approach that does not require human model summaries. Our approach exploits the compositional capabilities of corpus-based and lexical resource-based word embeddings to develop the features reflecting coverage, diversity, informativeness, and coherence of summaries. The features are then used to train a learning model for predicting the summary content quality in the absence of gold models. We evaluate the proposed metric in replicating the human assigned scores for summarization systems and summaries on data from query-focused and update summarization tasks in TAC 2008 and 2009. The results show that our feature combination provides reliable estimates of summary content quality when model summaries are not available.- Anthology ID:
- C18-1077
- Volume:
- Proceedings of the 27th International Conference on Computational Linguistics
- Month:
- August
- Year:
- 2018
- Address:
- Santa Fe, New Mexico, USA
- Editors:
- Emily M. Bender, Leon Derczynski, Pierre Isabelle
- Venue:
- COLING
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 905–914
- Language:
- URL:
- https://aclanthology.org/C18-1077
- DOI:
- Cite (ACL):
- Elaheh ShafieiBavani, Mohammad Ebrahimi, Raymond Wong, and Fang Chen. 2018. Summarization Evaluation in the Absence of Human Model Summaries Using the Compositionality of Word Embeddings. In Proceedings of the 27th International Conference on Computational Linguistics, pages 905–914, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
- Cite (Informal):
- Summarization Evaluation in the Absence of Human Model Summaries Using the Compositionality of Word Embeddings (ShafieiBavani et al., COLING 2018)
- PDF:
- https://preview.aclanthology.org/naacl24-info/C18-1077.pdf