@inproceedings{franco-scalabrin-2026-diagnosing,
    title = "Diagnosing Compositional Generalization in Transformers on {R}e{COGS} with Compositional Graph Similarity",
    author = "Franco, Bruno  and
      Scalabrin, Edson",
    editor = "Bernard, Timoth{\'e}e  and
      Chersoni, Emmanuele  and
      Rambelli, Giulia",
    booktitle = "Proceedings of the Third Workshop on the Bridges and Gaps between Formal and Computational Linguistics ({B}ri{G}ap-3)",
    month = jul,
    year = "2026",
    address = "Paris, France",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-brigap/2026.brigap-1.4/",
    pages = "31--39",
    ISBN = "XXX-X-XXXXXX-XX-X",
    abstract = "This paper investigates the evaluation of compositional generalization in Transformer models on the ReCOGS benchmark. The problem addressed is that ReCOGS relies on Semantic Exact Match, a binary metric that assigns the same penalty to minor local mismatches and severe structural errors, limiting diagnostic interpretation. To address this, the study introduces Compositional Graph Similarity (CGS), a graph-based metric that compares predicted and reference semantic structures through explicit edit operations, providing graded and interpretable structural evaluation. The work also uses controlled synthetic datasets to test whether low-scoring ReCOGS categories reflect true model limitations or weaknesses in dataset coverage. Empirical results show that CGS satisfies all seven quality criteria adopted for graph similarity and identifies the lowest-scoring ReCOGS categories as cp recursion (45.0{\%}), obj pp to subj pp (65.4{\%}), and prim to inf arg (66.7{\%}). Follow-up experiments showed 0{\%} Semantic Exact Match under depth extrapolation and constituent-role relocation, but 99.9{\%} Semantic Exact Match for prim to inf arg in isolation. These findings support the conclusion that CGS is more informative than Semantic Exact Match and that Transformer limitations in ReCOGS are partly structural and partly induced by dataset distribution."
}