@inproceedings{goren-etal-2025-chai,
    title = "{C}ha{I}-{T}e{A}: A Benchmark for Evaluating Autocompletion of Interactions with {LLM}-based Chatbots",
    author = "Goren, Shani  and
      Kalinsky, Oren  and
      Stav, Tomer  and
      Rapoport, Yuri  and
      Fairstein, Yaron  and
      Yazdi, Ram  and
      Cohen, Nachshon  and
      Libov, Alexander  and
      Kushilevitz, Guy",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Ingest-2025-COMPUTEL/2025.naacl-short.3/",
    pages = "18--32",
    ISBN = "979-8-89176-190-2",
    abstract = "The rise of LLMs has deflected a growing portion of human-computer interactions towards LLM-based chatbots.The remarkable abilities of these models allow users to interact using long, diverse natural language text covering a wide range of topics and styles. Phrasing these messages is a time and effort consuming task, calling for an autocomplete solution to assist users. We present **ChaI-TeA**: **Cha**t **I**n**te**raction **A**utocomplete; An autocomplete evaluation framework for LLM-based chatbot interactions. The framework includes a formal definition of the task, curated datasets and suitable metrics. We use it to evaluate 11 models on this task, finding that while current off-the-shelf models perform fairly, there is still much room for improvement, mainly in ranking of the generated suggestions. We provide insights for practitioners working on this task and open new research directions for researchers in the field. We release our framework to serve as a foundation for future research."
}