@inproceedings{salehi-jacobs-2024-effect,
    title = "The Effect of Model Capacity and Script Diversity on Subword Tokenization for {S}orani {K}urdish",
    author = "Salehi, Ali  and
      Jacobs, Cassandra L.",
    editor = {Nicolai, Garrett  and
      Chodroff, Eleanor  and
      Mailhot, Frederic  and
      {\c{C}}{\"o}ltekin, {\c{C}}a{\u{g}}r{\i}},
    booktitle = "Proceedings of the 21st SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology",
    month = jun,
    year = "2024",
    address = "Mexico City, Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.sigmorphon-1.6/",
    doi = "10.18653/v1/2024.sigmorphon-1.6",
    pages = "51--56",
    abstract = "Tokenization and morphological segmentation continue to pose challenges for text processing and studies of human language. Here, we focus on written Soran{\^i} Kurdish, which uses a modified script based on Persian and Arabic, and its transliterations into the Kurdish Latin script. Importantly, Perso-Arabic and Latin-based writing systems demonstrate different statistical and structural properties, which may have significant effects on subword vocabulary learning. This has major consequences for frequency- or probability-based models of morphological induction. We explore the possibility that jointly training subword vocabularies using a source script along with its transliteration would improve morphological segmentation, subword tokenization, and whether gains are observed for one system over others. We find that joint training has a similar effect to increasing vocabulary size, while keeping subwords shorter in length, which produces higher-quality subwords that map onto morphemes."
}