@inproceedings{perez-etal-2026-position,
    title = "Position: A Semiotic-Hermeneutic Approach to Evaluating Meaning in {LLM} Summaries via the Inductive Conceptual Rating Metric",
    author = "Perez, Natalie  and
      Bhaduri, Sreyoshi  and
      Chadha, Aman",
    editor = "Mille, Simon  and
      Gehrmann, Sebastian  and
      Schmidtov{\'a}, Patr{\'i}cia  and
      Du{\v{s}}ek, Ond{\v{r}}ej  and
      Fadaee, Marzieh  and
      Lo, Kyle  and
      Santus, Enrico  and
      Stanovsky, Gabriel",
    booktitle = "Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics ({GEM})",
    month = jul,
    year = "2026",
    address = "San Diego, California, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.gem-main.81/",
    pages = "1036--1047",
    ISBN = "979-8-89176-423-1",
    abstract = "Meaning in human language is relational and context-dependent, and it emerges, according to Saussure (1916), through a dynamic system of signs rather than fixed relationships between words and concepts. Insights from the study of semiotics and hermeneutics emphasize that meaning arises through interpretive processes shaped by context, which has historically posed challenges for computational processing and evaluation. Building on these perspectives, this article advances an interdisciplinary framework for evaluating meaning in machine-generated language and introduces the Inductive Conceptual Rating (ICR) metric, a qualitative approach grounded in inductive content analysis and reflective thematic analysis that assesses semantic accuracy and meaning alignment in generative artificial intelligence (GenAI) outputs beyond surface-level lexical and similarity metrics. The ICR metric is applied in an empirical study that compares thematic summaries generated by the large language model (LLM) with the human-generated output in five datasets (N = 50-800). Results show that although models achieve high linguistic similarity scores, they consistently unperformed relative to human outputs in capturing recurring, contextually grounded meanings. This work concludes by discussing implications for meaning evaluation and future research."
}