\section{Additional Annotation Details}
\label{sec:appendix_B}

Table~\ref{tab:compact-agreement} contains further details on inter-annotator agreement according to Krippendorff's Alpha \citep{castro-2017-fast-krippendorff}, which was used due to its ability to handle multiple annotators and incomplete annotation coverage. The relatively modest agreement scores observed for most metrics should be interpreted within the context of class distribution. For instance, Grammaticality shows particularly low agreement (0.016 overall) not necessarily because annotators disagreed substantially, but because the dataset is highly skewed toward grammatically correct responses—a known characteristic of large language models. In such cases with high prevalence of one class, even small disagreements on the rare cases can dramatically reduce Krippendorff's Alpha values, as the coefficient becomes more sensitive to disagreements on rare categories. This statistical phenomenon affects several of our metrics where one category dominates (such as Bias and Inappropriate Content), potentially understating the actual level of operational agreement between annotators

Table~\ref{tab:label-distribution} summarizes the details of our annotated datasets in terms of each of the categories. Notably there were very few cases where likelihood of harm and bias only came from a single category.
