LLMs in the Real World: Evaluating “AI” in Emergency Contexts

Sara Court, Lara Downing, Micha Elsner


Abstract
This paper offers a call to action. We urge our colleagues in the research community to play a greater role in the articulation of our findings to the public. To illustrate the stakes we present a case study on the initial stages of an LLM-based machine translation application’s deployment in a real-world context: a text-2-911 system advertising capabilities in 55 languages for use in emergencies in which it may be difficult to call operators directly. We identify a number of common misconceptions about technologies such as these, concluding with a set of concrete recommendations and best practices for stakeholders at every stage of the development and deployment pipeline. While the advancement of scientific research often lies in solving the "hard" problems, we argue it is often the "easy" ones— problems for which the latest technology is often unnecessary— that are most overlooked.
Anthology ID:
2026.findings-acl.1779
Volume:
Findings of the Association for Computational Linguistics: ACL 2026
Month:
July
Year:
2026
Address:
San Diego, California, United States
Editors:
Maria Liakata, Viviane P. Moreira, Jiajun Zhang, David Jurgens
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
35748–35760
Language:
URL:
https://preview.aclanthology.org/ingest-acl-workshops/2026.findings-acl.1779/
DOI:
Bibkey:
Cite (ACL):
Sara Court, Lara Downing, and Micha Elsner. 2026. LLMs in the Real World: Evaluating “AI” in Emergency Contexts. In Findings of the Association for Computational Linguistics: ACL 2026, pages 35748–35760, San Diego, California, United States. Association for Computational Linguistics.
Cite (Informal):
LLMs in the Real World: Evaluating “AI” in Emergency Contexts (Court et al., Findings 2026)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-acl-workshops/2026.findings-acl.1779.pdf
Checklist:
 2026.findings-acl.1779.checklist.pdf