@inproceedings{tafjord-etal-2019-quartz,
    title = "{Q}ua{RT}z: An Open-Domain Dataset of Qualitative Relationship Questions",
    author = "Tafjord, Oyvind  and
      Gardner, Matt  and
      Lin, Kevin  and
      Clark, Peter",
    editor = "Inui, Kentaro  and
      Jiang, Jing  and
      Ng, Vincent  and
      Wan, Xiaojun",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)",
    month = nov,
    year = "2019",
    address = "Hong Kong, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Author-page-Marten-During-lu/D19-1608/",
    doi = "10.18653/v1/D19-1608",
    pages = "5941--5946",
    abstract = "We introduce the first open-domain dataset, called QuaRTz, for reasoning about textual qualitative relationships. QuaRTz contains general qualitative statements, e.g., {\textquotedblleft}A sunscreen with a higher SPF protects the skin longer.{\textquotedblright}, twinned with 3864 crowdsourced situated questions, e.g., {\textquotedblleft}Billy is wearing sunscreen with a lower SPF than Lucy. Who will be best protected from the sun?{\textquotedblright}, plus annotations of the properties being compared. Unlike previous datasets, the general knowledge is textual and not tied to a fixed set of relationships, and tests a system`s ability to comprehend and apply textual qualitative knowledge in a novel setting. We find state-of-the-art results are substantially (20{\%}) below human performance, presenting an open challenge to the NLP community."
}