Span Labeling with Large Language Models: Shell vs. Meat

Phoebe Mulcaire, Nitin Madnani


Abstract
We present a method for labeling spans of text with large language models (LLMs) and apply it to the task of identifying shell language, language which plays a structural or connective role without constituting the main content of a text. We compare several recent LLMs by evaluating their “annotations” against a small human-curated test set, and train a smaller supervised model on thousands of LLM-annotated examples. The described method enables workflows that can learn complex or nuanced linguistic phenomena without tedious, large-scale hand-annotations of training data or specialized feature engineering.
Anthology ID:
2025.bea-1.62
Volume:
Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025)
Month:
July
Year:
2025
Address:
Vienna, Austria
Editors:
Ekaterina Kochmar, Bashar Alhafni, Marie Bexte, Jill Burstein, Andrea Horbach, Ronja Laarmann-Quante, Anaïs Tack, Victoria Yaneva, Zheng Yuan
Venues:
BEA | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
850–859
Language:
URL:
https://preview.aclanthology.org/acl25-workshop-ingestion/2025.bea-1.62/
DOI:
Bibkey:
Cite (ACL):
Phoebe Mulcaire and Nitin Madnani. 2025. Span Labeling with Large Language Models: Shell vs. Meat. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025), pages 850–859, Vienna, Austria. Association for Computational Linguistics.
Cite (Informal):
Span Labeling with Large Language Models: Shell vs. Meat (Mulcaire & Madnani, BEA 2025)
Copy Citation:
PDF:
https://preview.aclanthology.org/acl25-workshop-ingestion/2025.bea-1.62.pdf