Rationally Reappraising ATIS-based Dialogue Systems

Jingcheng Niu, Gerald Penn


Abstract
The Air Travel Information Service (ATIS) corpus has been the most common benchmark for evaluating Spoken Language Understanding (SLU) tasks for more than three decades since it was released. Recent state-of-the-art neural models have obtained F1-scores near 98% on the task of slot filling. We developed a rule-based grammar for the ATIS domain that achieves a 95.82% F1-score on our evaluation set. In the process, we furthermore discovered numerous shortcomings in the ATIS corpus annotation, which we have fixed. This paper presents a detailed account of these shortcomings, our proposed repairs, our rule-based grammar and the neural slot-filling architectures associated with ATIS. We also rationally reappraise the motivations for choosing a neural architecture in view of this account. Fixing the annotation errors results in a relative error reduction of between 19.4 and 52% across all architectures. We nevertheless argue that neural models must play a different role in ATIS dialogues because of the latter’s lack of variety.
Anthology ID:
P19-1550
Volume:
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2019
Address:
Florence, Italy
Editors:
Anna Korhonen, David Traum, Lluís Màrquez
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5503–5507
Language:
URL:
https://aclanthology.org/P19-1550
DOI:
10.18653/v1/P19-1550
Bibkey:
Cite (ACL):
Jingcheng Niu and Gerald Penn. 2019. Rationally Reappraising ATIS-based Dialogue Systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5503–5507, Florence, Italy. Association for Computational Linguistics.
Cite (Informal):
Rationally Reappraising ATIS-based Dialogue Systems (Niu & Penn, ACL 2019)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-bitext-workshop/P19-1550.pdf
Data
ATIS