@inproceedings{rosensweig-etal-2025-automatic,
    title = "Automatic Text Segmentation of Ancient and Historic {H}ebrew",
    author = "Rosensweig, Elisha  and
      Resnick, Benjamin  and
      Gershuni, Hillel  and
      Guedalia, Joshua  and
      Dershowitz, Nachum  and
      Shmidman, Avi",
    editor = "Anderson, Adam  and
      Gordin, Shai  and
      Li, Bin  and
      Liu, Yudong  and
      Passarotti, Marco C.  and
      Sprugnoli, Rachele",
    booktitle = "Proceedings of the Second Workshop on Ancient Language Processing",
    month = may,
    year = "2025",
    address = "The Albuquerque Convention Center, Laguna",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/moar-dois/2025.alp-1.1/",
    doi = "10.18653/v1/2025.alp-1.1",
    pages = "1--11",
    ISBN = "979-8-89176-235-0",
    abstract = "Ancient texts often lack punctuation marks, making it challenging to determine sentence boundaries and clause boundaries. Texts may contain sequences of hundreds of words without any period or indication of a full stop. Determining such boundaries is a crucial step in various NLP pipelines, especially regarding language models such as BERT that have context window constraints and regarding machine translation models which may become far less accurate when fed too much text at a time. In this paper, we consider several novel approaches to automatic segmentation of unpunctuated ancient texts into grammatically complete or semi-complete units. Our work here focuses on ancient and historical Hebrew and Aramaic texts, but the tools developed can be applied equally to similar languages. We explore several approaches to addressing this task: masked language models (MLM) to predict the next token; fewshot completions via an open-source foundational LLM; and the ``Segment-Any-Text'' (SaT) tool by Frohmann et al. (Frohmann et al., 2024). These are then compared to instructbased flows using commercial (closed, managed) LLMs, to be used as a benchmark. To evaluate these approaches, we also introduce a new ground truth (GT) dataset of manually segmented texts. We explore the performance of our different approaches on this dataset. We release both our segmentation tools and the dataset to support further research into computational processing and analysis of ancient texts, which can be found here `https://github.com/ERC-Midrash/rabbinic{\_}chunker'."
}