Ladislav Kunc


Benchmarking Language-agnostic Intent Classification for Virtual Assistant Platforms
Gengyu Wang | Cheng Qian | Lin Pan | Haode Qi | Ladislav Kunc | Saloni Potdar
Proceedings of the Workshop on Multilingual Information Access (MIA)

Current virtual assistant (VA) platforms are beholden to the limited number of languages they support. Every component, such as the tokenizer and intent classifier, is engineered for specific languages in these intricate platforms. Thus, supporting a new language in such platforms is a resource-intensive operation requiring expensive re-training and re-designing. In this paper, we propose a benchmark for evaluating language-agnostic intent classification, the most critical component of VA platforms. To ensure the benchmarking is challenging and comprehensive, we include 29 public and internal datasets across 10 low-resource languages and evaluate various training and testing settings with consideration of both accuracy and training time. The benchmarking result shows that Watson Assistant, among 7 commercial VA platforms and pre-trained multilingual language models (LMs), demonstrates close-to-best accuracy with the best accuracy-training time trade-off.

Distinguish Sense from Nonsense: Out-of-Scope Detection for Virtual Assistants
Cheng Qian | Haode Qi | Gengyu Wang | Ladislav Kunc | Saloni Potdar
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track

Out of Scope (OOS) detection in Conversational AI solutions enables a chatbot to handle a conversation gracefully when it is unable to make sense of the end-user query. Accurately tagging a query as out-of-domain is particularly hard in scenarios when the chatbot is not equipped to handle a topic which has semantic overlap with an existing topic it is trained on. We propose a simple yet effective OOS detection method that outperforms standard OOS detection methods in a real-world deployment of virtual assistants. We discuss the various design and deployment considerations for a cloud platform solution to train virtual assistants and deploy them at scale. Additionally, we propose a collection of datasets that replicates real-world scenarios and show comprehensive results in various settings using both offline and online evaluation metrics.


Benchmarking Commercial Intent Detection Services with Practice-Driven Evaluations
Haode Qi | Lin Pan | Atin Sood | Abhishek Shah | Ladislav Kunc | Mo Yu | Saloni Potdar
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers

Intent detection is a key component of modern goal-oriented dialog systems that accomplish a user task by predicting the intent of users’ text input. There are three primary challenges in designing robust and accurate intent detection models. First, typical intent detection models require a large amount of labeled data to achieve high accuracy. Unfortunately, in practical scenarios it is more common to find small, unbalanced, and noisy datasets. Secondly, even with large training data, the intent detection models can see a different distribution of test data when being deployed in the real world, leading to poor accuracy. Finally, a practical intent detection model must be computationally efficient in both training and single query inference so that it can be used continuously and re-trained frequently. We benchmark intent detection methods on a variety of datasets. Our results show that Watson Assistant’s intent detection model outperforms other commercial solutions and is comparable to large pretrained language models while requiring only a fraction of computational resources and training data. Watson Assistant demonstrates a higher degree of robustness when the training and test distributions differ.


Recipes for building voice search UIs for automotive
Martin Labsky | Ladislav Kunc | Tomas Macek | Jan Kleindienst | Jan Vystrcil
Proceedings of the EACL 2014 Workshop on Dialogue in Motion

Mostly Passive Information Delivery – a Prototype
Jan Vystrčil | Tomas Macek | David Luksch | Martin Labský | Ladislav Kunc | Jan Kleindienst | Tereza Kašparová
Proceedings of the EACL 2014 Workshop on Dialogue in Motion