@inproceedings{fedrushkov-etal-2026-semantic,
    title = "Semantic vs. Structural Signals: Log-Probability and {LLM}-as-a-Judge for Reference-Free Code Evaluation",
    author = "Fedrushkov, Dmitriy  and
      He, Yulong  and
      Smirnov, Ivan  and
      Aliev, Artem  and
      Kovalchuk, Sergey",
    editor = "Mille, Simon  and
      Gehrmann, Sebastian  and
      Schmidtov{\'a}, Patr{\'i}cia  and
      Du{\v{s}}ek, Ond{\v{r}}ej  and
      Fadaee, Marzieh  and
      Lo, Kyle  and
      Santus, Enrico  and
      Stanovsky, Gabriel",
    booktitle = "Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics ({GEM})",
    month = jul,
    year = "2026",
    address = "San Diego, California, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.gem-main.55/",
    pages = "574--581",
    ISBN = "979-8-89176-423-1",
    abstract = "Reference-free evaluation of LLM-generated code is essential when execution-based testing is unavailable or costly. We compare two paradigms: $\textbf{explicit LLM-as-a-Judge}$ scoring, which assigns a quality score to a solution, and $\textbf{log-probability scoring}$, which uses $\log P_\theta(\text{code} \mid \text{task})$ as an instruction-free signal.Across HumanEval-X, we find that the two approaches capture $\textit{qualitatively different aspects}$ of code correctness. Explicit judges {---} particularly larger models {---} perform strongly on generated code, reflecting their ability to reason about task-solution alignment, but fail to distinguish correct solutions from minimally mutated ones. Log-probability exhibits the opposite pattern: weaker performance on generated code, but consistent pairwise separation of canonical from mutated solutions.These results reveal a $\textbf{discrimination-ranking dissociation}$ and show that the two paradigms provide complementary, non-interchangeable signals: explicit judges capture semantic correctness, while log-probability captures local structural consistency."
}