@inproceedings{herrera-etal-2026-exploration,
    title = "Exploration of Sentence Representations in {S}panish {BERT}-like Models",
    author = "Herrera, Gonzalo  and
      Ros{\'a}, Aiala  and
      Chiruzzo, Luis",
    editor = "Claramunt, German Rigau  and
      Gamallo, Pablo  and
      Mu{\~n}oz Guillena, Rafael  and
      Chiruzzo, Luis  and
      Mart{\'i}nez C{\'a}mara, Eugenio",
    booktitle = "Proceedings of {LANLP}: Bridging {I}bero and {L}atin {A}merican {NLP} Communities",
    month = may,
    year = "2026",
    address = "Palma, Mallorca (Spain)",
    publisher = "ELRA Language Resources Association (ELRA)",
    url = "https://preview.aclanthology.org/cawl-year/2026.lanlp-1.8/",
    doi = "10.63317/5moein87oxiw",
    pages = "55--65",
    abstract = "Transformer-based language models, ubiquitous in NLP nowadays, generate internal representations (embeddings) of words and sentences. Yet, systematic comparisons of embedding strategies from various models remain limited. In this work, we evaluate Spanish embeddings from several BERT-like models (BETO, multilingual BERT, XLM-RoBERTa, ROUBERTa) to understand their syntactic and semantic capabilities across layers. We propose novel sentence-level analogy tests to probe generalization. Results show tasks like verb negation or word reordering perform best with embeddings from earlier layers, while nuanced semantic distinctions{---}such as agent or patient gender{---}are better captured by deeper layers. Our findings provide guidelines for embedding strategies and offer a foundation for further NLP research."
}