@inproceedings{choi-etal-2026-compass,
    title = "{COMPASS}: A Framework for Evaluating Organization-Specific Policy Alignment in {LLM}s",
    author = "Choi, Dasol  and
      Lee, DongGeon  and
      Kartono, Brigitta Jesica  and
      Berndt, Helena  and
      Kwon, Taeyoun  and
      Jang, Joonwon  and
      Park, Haon  and
      Yu, Hwanjo  and
      Kahng, Minsuk",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.2139/",
    pages = "46087--46133",
    ISBN = "979-8-89176-390-6",
    abstract = "As large language models are deployed in high-stakes enterprise applications, from healthcare to finance, ensuring adherence to organization-specific policies has become essential. Yet existing safety evaluations focus exclusively on universal harms. We present COMPASS (Company/Organization Policy Alignment Assessment), the first systematic framework for evaluating whether LLMs comply with organizational allowlist and denylist policies. We apply COMPASS to eight diverse industry scenarios, generating and validating 5,920 queries that test both routine compliance and adversarial robustness through strategically designed edge cases. Evaluating seven state-of-the-art models, we uncover a fundamental asymmetry: models reliably handle legitimate requests ({\ensuremath{>}}95{\%} accuracy) but catastrophically fail at enforcing prohibitions, refusing only 13{--}40{\%} of adversarial denylist violations. These results demonstrate that current LLMs lack the robustness required for policy-critical deployments, establishing COMPASS as an essential evaluation framework for organizational AI safety."
}