@inproceedings{suppa-etal-2026-skmteb,
    title = "{S}k{MTEB}: {S}lovak Massive Text Embedding Benchmark and Model Adaptation",
    author = "Suppa, Marek  and
      Ridzik, Andrej  and
      Hl{\textbackslash}'adek, Daniel  and
      K{\textbackslash}v{\{}n{\}}a{\textbackslash}v{\{}z{\}}ekov{\textbackslash}'a, Nat{\textbackslash}'alia  and
      Ondrejov{\textbackslash}'a, Vikt{\textbackslash}'oria",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.2114/",
    pages = "45597--45628",
    ISBN = "979-8-89176-390-6",
    abstract = "We introduce SkMTEB, the first comprehensive MTEB-style text embedding benchmark for Slovak, a low-resource West Slavic language, comprising 31 datasets across 7 task types{---}nearly 4$\times$ the depth of existing multilingual benchmark coverage for Slovak. Our evaluation of 31 embedding models reveals that large instruction-tuned multilingual models achieve the strongest performance, while existing Slovak-specific models trained for NLU tasks transfer poorly to embedding tasks. To address the need for efficient, locally-deployable Slovak embeddings, we develop (45M parameters) and (365M) by applying vocabulary trimming and fine-tuning to Multilingual E5 models. Despite size reductions of up to 62{\%}, our open-source models achieve competitive performance with proprietary APIs while remaining locally deployable for semantic search and retrieval-augmented generation (RAG). We release the benchmark, models, datasets, and code openly, hoping our approach offers a replicable path for other under-resourced languages."
}