@inproceedings{weissweiler-etal-2025-linguistic,
    title = "Linguistic Generalizations are not Rules: Impacts on Evaluation of {LM}s",
    author = "Weissweiler, Leonie  and
      Mahowald, Kyle  and
      Goldberg, Adele E.",
    editor = "Bonial, Claire  and
      Torgbi, Melissa  and
      Weissweiler, Leonie  and
      Blodgett, Austin  and
      Beuls, Katrien  and
      Van Eecke, Paul  and
      Tayyar Madabushi, Harish",
    booktitle = "Proceedings of the Second International Workshop on Construction Grammars and NLP",
    month = sep,
    year = "2025",
    address = {D{\"u}sseldorf, Germany},
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/iwcs-25-ingestion/2025.cxgsnlp-1.7/",
    pages = "61--74",
    ISBN = "979-8-89176-318-0",
    abstract = "Linguistic evaluations of how well LMs generalize to produce or understand novel text often implicitly take for granted that natural languages are generated by symbolic rules. Grammaticality is thought to be determined by whether sentences obey such rules. Interpretation is believed to be compositionally generated by syntactic rules operating on meaningful words. Semantic parsing is intended to map sentences into formal logic. Failures of LMs to obey strict rules have been taken to reveal that LMs do not produce or understand language like humans. Here we suggest that LMs' failures to obey symbolic rules may be a feature rather than a bug, because natural languages are not based on rules. New utterances are produced and understood by a combination of flexible, interrelated, and context-dependent constructions. We encourage researchers to reimagine appropriate benchmarks and analyses that acknowledge the rich, flexible generalizations that comprise natural languages."
}