@inproceedings{lee-etal-2018-dialcrowd,
    title = "{D}ial{C}rowd: A toolkit for easy dialog system assessment",
    author = "Lee, Kyusong  and
      Zhao, Tiancheng  and
      Black, Alan W.  and
      Eskenazi, Maxine",
    editor = "Komatani, Kazunori  and
      Litman, Diane  and
      Yu, Kai  and
      Papangelis, Alex  and
      Cavedon, Lawrence  and
      Nakano, Mikio",
    booktitle = "Proceedings of the 19th Annual {SIG}dial Meeting on Discourse and Dialogue",
    month = jul,
    year = "2018",
    address = "Melbourne, Australia",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/iwcs-25-ingestion/W18-5028/",
    doi = "10.18653/v1/W18-5028",
    pages = "245--248",
    abstract = "When creating a dialog system, developers need to test each version to ensure that it is performing correctly. Recently the trend has been to test on large datasets or to ask many users to try out a system. Crowdsourcing has solved the issue of finding users, but it presents new challenges such as how to use a crowdsourcing platform and what type of test is appropriate. DialCrowd has been designed to make system assessment easier and to ensure the quality of the result. This paper describes DialCrowd, what specific needs it fulfills and how it works. It then relates a test of DialCrowd by a group of dialog system developer."
}