@inproceedings{mehrabani-etal-2018-practical,
    title = "Practical Application of Domain Dependent Confidence Measurement for Spoken Language Understanding Systems",
    author = "Mehrabani, Mahnoosh  and
      Thomson, David  and
      Stern, Benjamin",
    editor = "Bangalore, Srinivas  and
      Chu-Carroll, Jennifer  and
      Li, Yunyao",
    booktitle = "Proceedings of the 2018 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers)",
    month = jun,
    year = "2018",
    address = "New Orleans - Louisiana",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/N18-3023/",
    doi = "10.18653/v1/N18-3023",
    pages = "185--192",
    abstract = "Spoken Language Understanding (SLU), which extracts semantic information from speech, is not flawless, specially in practical applications. The reliability of the output of an SLU system can be evaluated using a semantic confidence measure. Confidence measures are a solution to improve the quality of spoken dialogue systems, by rejecting low-confidence SLU results. In this study we discuss real-world applications of confidence scoring in a customer service scenario. We build confidence models for three major types of dialogue states that are considered as different domains: how may I help you, number capture, and confirmation. Practical challenges to train domain-dependent confidence models, including data limitations, are discussed, and it is shown that feature engineering plays an important role to improve performance. We explore a wide variety of predictor features based on speech recognition, intent classification, and high-level domain knowledge, and find the combined feature set with the best rejection performance for each application."
}