@inproceedings{yin-etal-2019-benchmarking,
    title = "Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach",
    author = "Yin, Wenpeng  and
      Hay, Jamaal  and
      Roth, Dan",
    editor = "Inui, Kentaro  and
      Jiang, Jing  and
      Ng, Vincent  and
      Wan, Xiaojun",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)",
    month = nov,
    year = "2019",
    address = "Hong Kong, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Author-page-Marten-During-lu/D19-1404/",
    doi = "10.18653/v1/D19-1404",
    pages = "3914--3923",
    abstract = "Zero-shot text classification (0Shot-TC) is a challenging NLU problem to which little attention has been paid by the research community. 0Shot-TC aims to associate an appropriate label with a piece of text, irrespective of the text domain and the aspect (e.g., topic, emotion, event, etc.) described by the label. And there are only a few articles studying 0Shot-TC, all focusing only on topical categorization which, we argue, is just the tip of the iceberg in 0Shot-TC. In addition, the chaotic experiments in literature make no uniform comparison, which blurs the progress. This work benchmarks the 0Shot-TC problem by providing unified datasets, standardized evaluations, and state-of-the-art baselines. Our contributions include: i) The datasets we provide facilitate studying 0Shot-TC relative to conceptually different and diverse aspects: the {\textquotedblleft}topic{\textquotedblright} aspect includes {\textquotedblleft}sports{\textquotedblright} and {\textquotedblleft}politics{\textquotedblright} as labels; the {\textquotedblleft}emotion{\textquotedblright} aspect includes {\textquotedblleft}joy{\textquotedblright} and {\textquotedblleft}anger{\textquotedblright}; the {\textquotedblleft}situation{\textquotedblright} aspect includes {\textquotedblleft}medical assistance{\textquotedblright} and {\textquotedblleft}water shortage{\textquotedblright}. ii) We extend the existing evaluation setup (label-partially-unseen) {--} given a dataset, train on some labels, test on all labels {--} to include a more challenging yet realistic evaluation label-fully-unseen 0Shot-TC (Chang et al., 2008), aiming at classifying text snippets without seeing task specific training data at all. iii) We unify the 0Shot-TC of diverse aspects within a textual entailment formulation and study it this way."
}