@inproceedings{huidrom-etal-2025-assessing,
    title = "Assessing Semantic Consistency in {D}ata{-}to{-}{T}ext Generation: A Meta-Evaluation of Textual, Semantic and Model-Based Metrics",
    author = "Huidrom, Rudali  and
      Lorandi, Michela  and
      Mille, Simon  and
      Thomson, Craig  and
      Belz, Anya",
    editor = "Flek, Lucie  and
      Narayan, Shashi  and
      Phương, L{\^e} Hồng  and
      Pei, Jiahuan",
    booktitle = "Proceedings of the 18th International Natural Language Generation Conference",
    month = oct,
    year = "2025",
    address = "Hanoi, Vietnam",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-you-zhang-rochester/2025.inlg-main.6/",
    pages = "98--107",
    abstract = "Ensuring semantic consistency between semantic-triple inputs and generated text is crucial in data{-}to{-}text generation, but continues to pose challenges both during generation and in evaluation. In order to assess how accurately semantic consistency can currently be assessed, we meta-evaluate 29 different evaluation methods in terms of their ability to predict human semantic-consistency ratings. The evaluation methods include embeddings{-}based, overlap{-}based, and edit{-}distance metrics, as well as learned regressors and a prompted `LLM{-}as{-}judge' protocol. We meta-evaluate on two datasets: the WebNLG 2017 human evaluation dataset, and a newly created WebNLG-style dataset that none of the methods can have seen during training. We find that none of the traditional textual similarity metrics or the pre-Transformer model-based metrics are suitable for the task of semantic consistency assessment. LLM-based methods perform well on the whole, but best correlations with human judgments still lag behind those seen in other text generation tasks."
}