Abstract
Policy compliance detection is the task of ensuring that a scenario conforms to a policy (e.g. a claim is valid according to government rules or a post in an online platform conforms to community guidelines). This task has been previously instantiated as a form of textual entailment, which results in poor accuracy due to the complexity of the policies. In this paper we propose to address policy compliance detection via decomposing it into question answering, where questions check whether the conditions stated in the policy apply to the scenario, and an expression tree combines the answers to obtain the label. Despite the initial upfront annotation cost, we demonstrate that this approach results in better accuracy, especially in the cross-policy setup where the policies during testing are unseen in training. In addition, it allows us to use existing question answering models pre-trained on existing large datasets. Finally, it explicitly identifies the information missing from a scenario in case policy compliance cannot be determined. We conduct our experiments using a recent dataset consisting of government policies, which we augment with expert annotations and find that the cost of annotating question answering decomposition is largely offset by improved inter-annotator agreement and speed.- Anthology ID:
- 2021.emnlp-main.678
- Volume:
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
- Month:
- November
- Year:
- 2021
- Address:
- Online and Punta Cana, Dominican Republic
- Editors:
- Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 8622–8632
- Language:
- URL:
- https://aclanthology.org/2021.emnlp-main.678
- DOI:
- 10.18653/v1/2021.emnlp-main.678
- Cite (ACL):
- Marzieh Saeidi, Majid Yazdani, and Andreas Vlachos. 2021. Cross-Policy Compliance Detection via Question Answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8622–8632, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Cite (Informal):
- Cross-Policy Compliance Detection via Question Answering (Saeidi et al., EMNLP 2021)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-3/2021.emnlp-main.678.pdf
- Data
- BoolQ, ShARC