@inproceedings{de-vergnette-etal-2025-evaluation,
    title = "Evaluation Framework for Layered Meaning Representation",
    author = "de Vergnette, R{\'e}mi  and
      Amblard, Maxime  and
      Guillaume, Bruno",
    editor = "Lai, Kenneth  and
      Wein, Shira",
    booktitle = "Proceedings of the Sixth International Workshop on Designing Meaning Representations",
    month = aug,
    year = "2025",
    address = "Prague, Czechia",
    publisher = "Association for Computational Lingustics",
    url = "https://preview.aclanthology.org/gwc-25-ingestion/2025.dmr-1.5/",
    pages = "38--48",
    abstract = "We propose different modular evaluation metrics for Layered Meaning Representation, defined as YARN, a semantic formalism encoded using rich structures that generalize AMR graphs. While existing metrics like SMATCH evaluate graph-based semantic representations such as AMR, they cannot directly handle YARN{'}s more complex structures. We make full use of the modular nature of YARN to propose two families of metrics, depending on the linguistic features and type of semantic phenomenon targeted. The first one, SMATCHY, extends the AMR SMATCH metric. We also propose YARNBLEU, based on the SEMBLEU metric for AMR. We evaluate both families on a small dataset of human annotated YARN structures, adding random modifications simulating annotation mistakes and show that SMATCHY provides a more consistent and reliable approach with respect to the type of modifications considered."
}