This repository includes the data and the code relevant for our submission. Analysis of this data is provided in "Necessity and Sufficiency Analysis.ipynb".

"HateCheck_templates_and_results.pickle" includes the test cases from HateCheck and the predictions of our trained classifiers on these.

"HateCheck_necc_suff_perturbations.pickle" includes the perturbations we generated for the HateCheck test cases, together with the masking patterns.

"HateCheck_necc_suff_preds.pickle" includes the predictions of the classifier on the original instances and the perturbations.

"HateCheck_necc_suff_result_all.pickle" includes the necessity and sufficiency values for each token in each of the explicit HateCheck test cases.

"perturbation_functions.py" includes all the functions we have used to generate the perturbations and to calculate the necessity and sufficiency values. 

