@inproceedings{kumar-etal-2026-odasim,
    title = "{ODAS}im: Ordered, Distinctive and Absolute Semantic Similarity for Code Explanation Evaluation",
    author = "Kumar, Prince  and
      Munigala, Vitobha  and
      Sen, Jaydeep  and
      Mittal, Ashish  and
      Kumar, Vishwajeet  and
      Tamilselvam, Srikanth G.",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.findings-acl.1415/",
    pages = "28390--28403",
    ISBN = "979-8-89176-395-1",
    abstract = "Code explanations are increasingly generated by large language models and used in software engineering workflows, making reliable evaluation essential. However, existing model-based and embedding-based methods often fail to distinguish correct explanations from partially or fully incorrect ones, and their similarity scores are poorly calibrated and do not reflect meaningful differences in explanation quality. To address this, we propose ODASim(Orderly, Dstinctive, and Absolute Similarity), a model-agnostic graded fine-tuning framework for embedding models that learns calibrated similarity representations between code and explanations. To support fine-grained supervision and evaluation, we also introduce ODA-X, a novel benchmark for code-to-explanation quality grading, comprising code{--}explanation pairs graded similarity labels derived from strategic perturbations of gold explanations. We apply our ODASim approach to multiple embedding models and evaluate it on two benchmarks: widely popular CodeXGLUE and our proposed benchmark ODA-X, spanning four programming languages - Python, Java, JavaScript, and Go. Results show that our method achieves up to 35{\%} improvement in F1 score and 85{\%} reduction in Expected Calibration Error (ECE), enabling reliable evaluation of code to explanation quality."
}