Abstract
A language model may be viewed as a 𝛴-valued stochastic process for some alphabet 𝛴.However, in some pathological situations, such a stochastic process may “leak” probability mass onto the set of infinite strings and hence is not equivalent to the conventional view of a language model as a distribution over ordinary (finite) strings.Such ill-behaved language processes are referred to as *non-tight* in the literature.In this work, we study conditions of tightness through the lens of stochastic processes.In particular, by regarding the symbol as marking a stopping time and using results from martingale theory, we give characterizations of tightness that generalize our previous work [(Du et al. 2023)](https://arxiv.org/abs/2212.10502).- Anthology ID:
- 2024.findings-acl.659
- Volume:
- Findings of the Association for Computational Linguistics ACL 2024
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand and virtual meeting
- Editors:
- Lun-Wei Ku, Andre Martins, Vivek Srikumar
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 11083–11094
- Language:
- URL:
- https://aclanthology.org/2024.findings-acl.659
- DOI:
- Cite (ACL):
- Li Du, Holden Lee, Jason Eisner, and Ryan Cotterell. 2024. When is a Language Process a Language Model?. In Findings of the Association for Computational Linguistics ACL 2024, pages 11083–11094, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
- Cite (Informal):
- When is a Language Process a Language Model? (Du et al., Findings 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.findings-acl.659.pdf