@inproceedings{pandey-2023-syntax,
    title = "Syntax-guided Neural Module Distillation to Probe Compositionality in Sentence Embeddings",
    author = "Pandey, Rohan",
    editor = "Vlachos, Andreas  and
      Augenstein, Isabelle",
    booktitle = "Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics",
    month = may,
    year = "2023",
    address = "Dubrovnik, Croatia",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2023.eacl-main.229/",
    doi = "10.18653/v1/2023.eacl-main.229",
    pages = "3143--3149",
    abstract = "Past work probing compositionality in sentence embedding models faces issues determining the causal impact of implicit syntax representations. Given a sentence, we construct a neural module net based on its syntax parse and train it end-to-end to approximate the sentence{'}s embedding generated by a transformer model. The distillability of a transformer to a Syntactic NeurAl Module Net (SynNaMoN) then captures whether syntax is a strong causal model of its compositional ability. Furthermore, we address questions about the geometry of semantic composition by specifying individual SynNaMoN modules' internal architecture {\&} linearity. We find differences in the distillability of various sentence embedding models that broadly correlate with their performance, but observe that distillability doesn{'}t considerably vary by model size. We also present preliminary evidence that much syntax-guided composition in sentence embedding models is linear, and that non-linearities may serve primarily to handle non-compositional phrases."
}