@inproceedings{hirao-etal-2018-automatic,
    title = "Automatic Pyramid Evaluation Exploiting {EDU}-based Extractive Reference Summaries",
    author = "Hirao, Tsutomu  and
      Kamigaito, Hidetaka  and
      Nagata, Masaaki",
    editor = "Riloff, Ellen  and
      Chiang, David  and
      Hockenmaier, Julia  and
      Tsujii, Jun{'}ichi",
    booktitle = "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing",
    month = oct # "-" # nov,
    year = "2018",
    address = "Brussels, Belgium",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/D18-1450/",
    doi = "10.18653/v1/D18-1450",
    pages = "4177--4186",
    abstract = "This paper tackles automation of the pyramid method, a reliable manual evaluation framework. To construct a pyramid, we transform human-made reference summaries into extractive reference summaries that consist of Elementary Discourse Units (EDUs) obtained from source documents and then weight every EDU by counting the number of extractive reference summaries that contain the EDU. A summary is scored by the correspondences between EDUs in the summary and those in the pyramid. Experiments on DUC and TAC data sets show that our methods strongly correlate with various manual evaluations."
}