@inproceedings{knez-zitnik-2026-improving,
    title = "Improving {S}lovene Language Models for Lexicographic Question Answering through Continued Pretraining and Instruction Fine-Tuning",
    author = "Knez, Timotej  and
      Zitnik, Slavko",
    editor = "Hinrichs, Erhard  and
      Nivre, Joakim  and
      Osenova, Petya  and
      Pustejovsky, James  and
      Zinn, Claus",
    booktitle = "Proceedings of the Workshop on Structured Linguistic Data and Evaluation ({SL}i{DE})",
    month = may,
    year = "2026",
    address = "Palma de Mallorca, Spain",
    publisher = "ELRA Language Resources Association (ELRA)",
    url = "https://preview.aclanthology.org/test-year-match/2026.slide-1.10/",
    doi = "10.63317/2t3zgpnbv52e",
    pages = "114--123",
    abstract = "This paper presents a two-stage training approach to improve the performance of Slovene large language models on lexicographic question-answering tasks. We developed a comprehensive lexical pretraining corpus containing 356,294 Slovene word entries. We constructed the corpus by converting structured data from multiple lexicographic sources into markdown format. Additionally, we created a question-answering dataset with 10,485 QA pairs from diverse sources, including automatically generated questions, a linguistic advisory portal, and community forums. Using the Slovenian GaMS model (based on Gemma 2 9B) and GaMS 3 model (based on Gemma 3 12B), we performed continued pretraining on the lexical corpus, followed by instruction fine-tuning with our QA dataset combined with translated general-domain questions. We compared results to different model configurations. Our results demonstrate significant improvements (text similarity increasing from 0.226 to 0.542, BERTScore F1 of 0.915) in answering Slovene lexicographic questions, validating the effectiveness of domain-specific continued pretraining for low-resource languages."
}