\documentclass[11pt]{article}

\usepackage[final]{acl}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{microtype}
\usepackage{inconsolata}
\usepackage{graphicx}
\usepackage{listings}
\usepackage{xcolor}
\usepackage{csquotes}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{subcaption}

\lstdefinestyle{codebox}{
  basicstyle=\ttfamily\scriptsize,
  frame=single,
  framerule=0.4pt,
  columns=fullflexible,
  keepspaces=true,
  showstringspaces=false,
  breaklines=true,
  breakatwhitespace=false,
  breakindent=0pt,
  tabsize=2
}

\input{commands}

% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\title{WWTC@UniA at SemEval-2026 Task 13:\\ BERT-based Code Authorship Detection and Qualitative Analysis}

% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
%         Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
%         Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \And  ... \And
%         Author n \\ Address line \\ ... \\ Address line}
% To start a separate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \AND
%         Author 2 \\ Address line \\ ... \\ Address line \And
%         Author 3 \\ Address line \\ ... \\ Address line}

% \author{First Author \\
%   Affiliation / Address line 1 \\
%   Affiliation / Address line 2 \\
%   Affiliation / Address line 3 \\
%   \texttt{email@domain} \\\And
%   Second Author \\
%   Affiliation / Address line 1 \\
%   Affiliation / Address line 2 \\
%   Affiliation / Address line 3 \\
%   \texttt{email@domain} \\}

\author{
  Linda Kupfer\thanks{Equal contribution.} \hspace{8mm} Lisa Hader\footnotemark[1] \hspace{8mm} Christian Jaumann\footnotemark[1] \hspace{8mm} Annemarie Friedrich\\
  University of Augsburg, Germany \\
  \texttt{\{firstname.lastname\}@uni-a.de}
}

%\author{
%  \textbf{First Author\textsuperscript{1}},
%  \textbf{Second Author\textsuperscript{1,2}},
%  \textbf{Third T. Author\textsuperscript{1}},
%  \textbf{Fourth Author\textsuperscript{1}},
%\\
%  \textbf{Fifth Author\textsuperscript{1,2}},
%  \textbf{Sixth Author\textsuperscript{1}},
%  \textbf{Seventh Author\textsuperscript{1}},
%  \textbf{Eighth Author \textsuperscript{1,2,3,4}},
%\\
%  \textbf{Ninth Author\textsuperscript{1}},
%  \textbf{Tenth Author\textsuperscript{1}},
%  \textbf{Eleventh E. Author\textsuperscript{1,2,3,4,5}},
%  \textbf{Twelfth Author\textsuperscript{1}},
%\\
%  \textbf{Thirteenth Author\textsuperscript{3}},
%  \textbf{Fourteenth F. Author\textsuperscript{2,4}},
%  \textbf{Fifteenth Author\textsuperscript{1}},
%  \textbf{Sixteenth Author\textsuperscript{1}},
%\\
%  \textbf{Seventeenth S. Author\textsuperscript{4,5}},
%  \textbf{Eighteenth Author\textsuperscript{3,4}},
%  \textbf{Nineteenth N. Author\textsuperscript{2,5}},
%  \textbf{Twentieth Author\textsuperscript{1}}
%\\
%\\
%  \textsuperscript{1}Affiliation 1,
%  \textsuperscript{2}Affiliation 2,
%  \textsuperscript{3}Affiliation 3,
%  \textsuperscript{4}Affiliation 4,
%  \textsuperscript{5}Affiliation 5
%\\
%  \small{
%    \textbf{Correspondence:} \href{mailto:email@domain}{email@domain}
%  }
%}

\begin{document}
\maketitle
\begin{abstract}
This paper describes our system for SemEval-2026 Task 13 on detecting machine-generated code. % across multiple programming languages, generators, and application scenarios.
We fine-tune small encoder-only models for detecting human-written versus machine-generated code and for identifying which large language model (LLM) family was used to obtain code. 
%Unlike most previous studies, which often use encoder-only models that are specifically pre-trained for the code domain, we show that %this is unnecessary and instead use
%a general-purpose model, i.e., ModernBERT, also works well.
%our results show that using
We find that a strong, general-purpose model (ModernBERT) outperforms models specifically pre-trained for the code domain.
In the official evaluation, our system ranks 5th on subtask B and 6th on subtask C.
Our detailed analysis reveals that comments and other natural language text that is part of the code snippets provide valuable information for identifying the LLM family that generated it.
Moreover, we show that the embeddings of our finetuned ModernBERT do not distinguish well between LLM families, but they cluster human-written code by programming language.




\end{abstract}

\section{Introduction}

Large language models (LLMs)
%are now widely used across a broad range of applications, including natural language processing, data analysis, and software development \cite{Kumar-2024-Large}.
%In particular, LLMs
have recently become popular tools for programming code generation and code completion, substantially influencing modern software engineering workflows \cite{Kumar-2024-Large}. %, HAQUE2025100204}.
Although these capabilities increase productivity, they also raise concerns about the authorship, accountability, and trustworthiness of generated code \cite{HAQUE2025100204}. %Having a reliable ability 
Being able to distinguish between human-written and machine-generated code is therefore of growing importance, e.g., for academic integrity, software maintenance, %security audits,
and intellectual property protection.

To address these challenges, SemEval-2026 Task~13\footnote{\href{https://github.com/mbzuai-nlp/SemEval-2026-Task13}{https://github.com/mbzuai-nlp/SemEval-2026-Task13}} \citep{orel-etal-2026-semeval-2026} 
focuses on the detection of machine-generated code under diverse %and realistic 
conditions. 
% The Shared Task builds upon enriched datasets introduced in prior work \citep{orel-etal-2025-droid, orel-etal-2025-codet} and consists of three subtasks. 
The shared task builds on the AICD dataset \citep{orel2026aicdbenchchallengingbenchmark} and is divided into three subtasks. 
Subtask A focuses on binary detection, distinguishing machine-generated and human-written code. Subtask B is a multi-class classification task, i.e., human-written code or code from one of 10 LLM families. Subtask C %lfocuses on hybrid code detection and
extends the setting of subtask A by adding a \textit{hybrid} class (code partially written by an LLM) and an \textit{adversarial} class, where the LLM is instructed to mimic human code.
%, notably the Droid resource suite \citep{orel2025droid} and the CoDet-M4 benchmark \cite{orel-etal-2025-codet}. 
More details on the task can be found in Appendix \ref{ssec:task_appendix}.

\begin{figure}[t]
\includegraphics[width=1\columnwidth]{images/WWTC_Teaser.drawio.pdf}
  \caption {Code snippet of shared task dataset containing not only code and comments, but additional natural language explanations. We analyze the impact of these individual components.
  %to better understand detection performance.
  } 
  \label{fig:teaser_image}
\end{figure}


Most existing approaches for detecting AI-generated code are based on encoder-only models that are specifically pre-trained for the code domain and then further fine-tuned for detection \citep{Nguyen-2024-GPTSniffer, Xu-2025-Distinguishing, orel-etal-2025-codet}. 
In contrast, the best-performing approach of our team, \textit{\underline{W}ho~\underline{W}rote~\underline{T}he~\underline{C}ode?~@~\underline{Uni}versity~of~\underline{A}ugsburg} (WWTC@UniA), uses the general-purpose language model ModernBERT-large \citep{warner-etal-2025-smarter} and fine-tunes it for the detection task.



We found that several code snippets in the dataset, particularly machine-generated instances, contain natural language explanations also outside comments.
To analyze how different types of content contribute to the model's ability to identify its source, 
we heuristically separate code snippets into pure code, comments, and natural language explanations (see \autoref{fig:teaser_image}).
We find that all of these parts play an important role, especially when classifying LLM family.
Our analysis also tests several loss functions and data augmentation to mitigate class imbalance, and provides a visual analysis of the embedding space.


In the official evaluation, our system ranks 5th out of 34 teams in subtask B and 6th out of 32 teams in subtask C, demonstrating the effectiveness of our approach across different detection settings. 


\section{Related Work}

%This section outlines existing work on the detection of machine-generated code.
Early approaches to detecting machine-generated code use statistics on code-level for decision tree learning \citep{Li-2023-Discriminating, Idialu-2024-Whodunit}.
%, demonstrating that these statistics are reliable indicators of authorship \citep{Li-2023-Discriminating, Idialu-2024-Whodunit}.
% \citet{Nguyen-2024-GPTSniffer} use in their approach GPTSniffer CodeBERT \citep {feng-etal-2020-codebert} which should learn the binary classification of whether a code snippet was written by ChatGPT\footnote{\href{https://chatgpt.com/}{https://chatgpt.com/}} or a human.
\citet{Nguyen-2024-GPTSniffer} fine-tune CodeBERT \citep {feng-etal-2020-codebert} for the binary classification of whether a code snippet was written by ChatGPT\footnote{\href{https://chatgpt.com/}{https://chatgpt.com/}} or a human.
% \citet{Xu-2025-Distinguishing} propose a contrastive learning-based approach to fine-tune UniXcoder \citep{guo-etal-2022-unixcoder} for the same detection task.
For the same detection task, \citet{Xu-2025-Distinguishing} propose a contrastive learning-based approach to fine-tuning UniXcoder \citep{guo-etal-2022-unixcoder}.
%to also predict whether the code was written by ChatGPT or a human. 
% \citet{orel-etal-2025-codet} use a broader dataset, covering more programming languages, code generators, domains and hybrid settings. They 
\citet{orel-etal-2025-codet} %use a more diverse dataset and 
evaluate several detection approaches, including traditional methods like decision trees and more advanced methods with fine-tuning CodeBERT, CodeT5 \citep{wang-etal-2021-codet5} and UniXcoder. %The UniXcoder based approach outperforms the other evaluated approaches on this dataset. 

% In DroidCollection, \citet{orel-etal-2025-droid} collected 1 million instances, which were generated by 43 language models in seven languages and different coding domains. 
%To evaluate performance on the dataset, they use models that were already specialized for the task, such as GPTSniffer. 
% \citet{orel-etal-2025-droid} evaluate several different approaches, including the fine-tuning of a %simple 
% Graph Convolutional Network \citep{nipf-2017-Semi}, as well as CatBoost \citep{Ostroumova-2018-CatBoost} and ModernBERT \citep{warner-etal-2025-smarter}.
\citet{orel-etal-2025-droid} train several different approaches, including a Graph Convolutional Network \citep{nipf-2017-Semi}, CatBoost \citep{Ostroumova-2018-CatBoost}, and ModernBERT \citep{warner-etal-2025-smarter}.
As they identify ModernBERT as the best-performing model, we focus on this model. %also employ it for our approach.
%The ModernBERT-based approach outperforms all other models. 



\section{System Description}
\label{sec:description}

To classify code snippets for all tasks, we fine-tune BERT-based models \citep{devlin-etal-2019-bert}.
We add a classification head, which receives the embedding of the [CLS] token as input.
Our model selection includes CodeBERT\footnote{\href{https://huggingface.co/microsoft/codebert-base}{https://huggingface.co/microsoft/codebert-base}} \citep {feng-etal-2020-codebert}, RoBERTa\footnote{\href{https://huggingface.co/FacebookAI/roberta-base}{https://huggingface.co/FacebookAI/roberta-base}} \citep{Lui-2019-RoBERTa}, and ModernBERT-large\footnote{\href{https://huggingface.co/answerdotai/ModernBERT-large}{https://huggingface.co/answerdotai/ModernBERT-large}} \citep{warner-etal-2025-smarter}.
For CodeBERT and RoBERTa, we use an input length of 512 tokens. 
Since ModernBERT supports longer input lengths and 16-44\% of the instances in the dataset are longer than 512 tokens (see Appendix \ref{ssec:dataset_appendix}), we experiment with input lengths of 512, 1024, and 2048 tokens.
We do not evaluate longer inputs as very few instances exceed 2048 tokens.
If a code snippet exceeds the input length, it is truncated.

\textbf{Training settings.}
We perform hyperparameter optimization on ModernBERT with a maximum token length of 2048 and 10 trials using Optuna \citep{Akiba-2019-Optuna}. 
All models are trained for 10 epochs with a batch size of 128, a learning rate of 1.2e-4, a warm-up ratio of 0.073, a weight decay of 0.03, and a linear learning rate scheduler. We use early stopping with a patience of 5 (evaluating on the validation set every 1000 steps). 
Early stopping always occurred when training on the entire train set, usually after about 5 epochs.
Because of the considerable class imbalance in task B (see Appendix~\ref{ssec:dataset_appendix}), instead of cross-entropy (CE) loss, we also evaluate focal loss \citep{lin-2020-focal} and weighted CE loss \citep{Cui-2019-Class} for task B. 


\textbf{Data augmentation.}
To mitigate class imbalance, we experiment with data augmentation.
For each model family used to generate the original dataset, we select a model from the corresponding family, except for OpenAI, as we only use open-weights models. 
% The exact models used can be found in Appendix~\ref{ssec:llm_based_approaches}. 
The respective LLM is prompted to change the variable names and code structure while maintaining functionality.
%The exact models and prompt template can be seen in Appendix~\ref{ssec:llm_based_approaches}. 
We use a temperature value of 0.3 to add variety to the generated code snippets and generate up to two additional examples for each original instance.
%To analyze the effect of these additional examples on performance, we compare three settings: (1) adding one generated example for each original example, (2) adding two examples for each original example, and (3) replacing each original example with a generated one.
The examples for both the human class and OpenAI remain unchanged. 
% The resulting class distribution and additional implementation details can be seen in Appendix~\ref{ssec:llm_based_approaches}.
% For further implementation details, refer to Appendix~\ref{ssec:llm_based_approaches}.
For the implementation details and resulting class distributions, see Appendix~\ref{ssec:llm_based_approaches}.



\textbf{LLM-based classification.}
% We also evaluate the performance of LLMs in a 0-shot setting on subtask A.
We also evaluate the performance of LLMs in a 0-shot Chain of Thought (CoT) setting on subtask A.
% The LLM is instructed to determine whether the code was written by a human or generated by a machine by focusing on indicators such as variable naming, comment style, and traces of generator outputs.
The LLM is instructed to determine whether the code was written by a human or a machine by focusing on indicators such as variable naming, comment style, and traces of generator outputs.
To improve the decision-making process, we instruct it to \enquote{think step by step} \citep{Kojima-2022-Large}.
For the exact prompt template, refer to Appendix~\ref{ssec:llm_based_approaches}. 
To improve reproducibility, we run the LLM with a temperature of 0 and use an open-weights LLM, i.e., Qwen2.5-Coder-32B.\footnote{\href{https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct}{https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct}}



%\subsection{Data Cleaning}
\textbf{Content type assignments.}
When manually inspecting the dataset provided for the shared task, we found several instances that not only contain code lines, but also comments and additional natural language explanations not marked as comments.
%examples of code snippets that include not only code, but also other natural language explanations.
This occurs especially frequently in LLM-generated instances (see Appendix~\ref{ssec:dataset_appendix}). 
%For example, some snippets explained the written code (see \autoref{fig:ex_dataset_not_cleaned} and \autoref{fig:ex_dataset_impossible_for_heuristic} in Appendix \ref{ssec:dataset_appendix}).
We hypothesize that these additional texts may provide a strong cue towards LLM code authorship.
%We analyze the impact on performance of such additional texts, since they might make LLM-generated code snippets easier to detect.

To understand how the different content types within code snippets contribute to detecting LLM-generated code, %. Therefore, we split the code snippets into three categories:
% we assign one of three categories (pure code, comments within the code, and natural language) to each line.
% we first identify for each line whether it is likely code (including comments) or rather natural language. In a second step, we check the code lines for language-specific comment characters to further separate comments from code.
%we split the code snippets into three categories .
%Due to the size of the provided datasets, performing this task manually is not feasible, 
% assigning one of three content types (pure code, comments within the code, and natural language) to each line with a heuristic rule-based approach.
we split the code snippets into three content types, i.e., pure code, comments within the code, and natural language.
First, we identify whether each line is likely code (including comments) or natural language. Next, we check the code lines for language-specific comment characters to further separate comments from code.
%, which iterates through the code snippets line by line, % searching for patterns typical of code or natural language text.
%identifying patterns typical of code or natural language.
% Therefore, we automate it using a heuristic-based approach.
% The heuristic iterates through the code snippets line by line and searches for patterns that are typical of either code or natural language text.
Indicators for code include programming language-specific keywords, syntax terminators (e.g., \enquote{\{}, \enquote{\}}, or \enquote{]:}), if the line starts with a language-specific comment character, or includes variable assignments or type pointers identified via regular expressions.
We assume natural language text %indicators include that
if the line exhibits a typical sentence structure (i.e., it starts with a letter and ends with a punctuation mark), if it contains at least five stopwords, or if it begins with bullet points or enumerations.
To avoid outliers, the predictions are smoothed in comparison to the lines directly before and after. E.g., if the lines on either side are identified as code, then the middle line is also labeled as code.
% We validate the heuristic by applying it to 100 randomly selected instances from the dataset. The annotator agrees with the heuristic for 93 of these instances.

We validate the heuristic on 100 randomly selected instances from the dataset, observing agreement with the human annotator, one of the authors holding an undergraduate degree in computer science, in 93 cases.
The disagreements can be narrowed down to the heuristic incorrectly classifying lines of natural language as code. 
% For a more detailed description of the disagreements refer to Appendix \ref{ssec:validation_heuristic}.
For a detailed description of the disagreements, see Appendix \ref{ssec:validation_heuristic}.


\section{Development Results and Ablations}


This section presents the results of our approaches and several ablations on the validation set. 
% We use the official evaluation metric of the Shared Task as evaluation metric, i.e., the Macro-F1 score. The Macro-F1 score is the unweighted arithmetic mean of the F1 scores calculated independently for each class.
We use the official evaluation metric of the shared task, the Macro-F1 score, which is computed as the unweighted arithmetic mean of the class-wise F1 scores.
To evaluate the difficulty of each subtask, we use a random baseline that predicts class labels based on the empirical class distribution observed in the respective validation set (for details see Appendix~\ref{ssec:random_baseline}). 
%This baseline provides a lower-bound performance estimate for each subtask. 
% For a more detailed description of the implementation details refer to Appendix \ref{ssec:random_baseline}.

\subsection{Main Results and Findings}
\autoref{tab:main_res} shows that the fine-tuning of encoder-only models outperforms the 0-shot CoT LLM approach as well as the random baseline by a large margin. 
The LLM-based approach performs only %marginally 
slightly 
better than the random baseline, suggesting that it is not well-suited to the given task. Consequently, it is not evaluated on subtasks B and C.

\textbf{Encoder-only approaches.} For encoder-only approaches, the version that fine-tunes ModernBERT-large achieves the best performance across all subtasks, which is consistent with prior work \citep{orel-etal-2025-droid}. 
However, its superior performance compared to the other two models is not solely due to the use of longer input lengths, as the version using only 512 tokens also performs considerably better. 
Still, increasing the input length improves performance, consistent with the fact that for most data splits, more than 20\% of instances exceed 512 tokens (see Appendix~\ref{ssec:dataset_appendix}). This difference is most pronounced in subtask B. %, which is potentially the most challenging due to its 11 classes and imbalanced class distribution. % Therefore, it is beneficial to use all available features.

\textbf{Pre-training of models.}
% By analyzing the performance of CodeBERT and RoBERTa, we aim to determine if prior fine-tuning of the model on the coding domain improves performance.  For instance, CodeBERT is based on RoBERTa-base \citep{Lui-2019-RoBERTa} and fine-tuned on code with a dataset \citep{Husain-2019-CodeSearchNet} to learn general-purpose code representations.
We next compare CodeBERT and RoBERTa.
CodeBERT is based on the RoBERTa architecture \citep{Lui-2019-RoBERTa} but pre-trained on code \citep{Husain-2019-CodeSearchNet} to learn general-purpose code representations.
% Since the performance of both models is about the same on our tasks, this indicates that additional fine-tuning in the code domain is not necessary.
Since the performance of both models is about the same on our tasks, code-specific pre-training is not necessary if training sets are as large as those provided in the context of the shared task.
However, in-domain pre-training may be beneficial if the datasets used for detection fine-tuning are smaller.



\begin{table}
\footnotesize
\centering
\setlength\tabcolsep{4pt}
\begin{tabular}{l|c|ccc}
\toprule
Model & max\_len & A & B & C \\
\midrule
ModernBERT & 2048 & \textbf{99.81} & 67.58 & \textbf{89.38} \\
\hspace{2mm} + focal loss & 2048 & -  & 67.17 & - \\
\hspace{2mm} + weighted-ce & 2048 & - & \textbf{68.53} & - \\
%\hspace{2mm} + mean-pooling & 2048 & - & 67.51 & 89.25 \\
ModernBERT & 1024 & 99.80 & 67.32 & 89.26 \\
ModernBERT & 512 & 99.71 & 65.11 & 88.04 \\
CodeBERT & 512 & 99.50 & 57.44 & 84.61 \\
RoBERTa & 512 & 99.47 & 56.36 & 84.26 \\
\midrule
Qwen2.5-Coder-32B & - & 51.35 & - & - \\
\midrule
random & - & 49.97 & 9.1 & 25.07 \\
% \midrule
% Results on test set & - & 53.47 & 42.99 & 64.77 \\
\bottomrule
\end{tabular}
\caption{Macro-F1 scores on validation sets of all subtasks. max\_len = maximum input length for models.}
\label{tab:main_res}
\end{table}


\textbf{Performance by content type.}
\begin{figure}[t]
\includegraphics[width=1\columnwidth]{images/modernbert_clean_ablation.pdf}
  \caption {Macro-F1 per subtask on validation set per code snippet component. original = code snippet as in dataset, cleaned = code + comments, code = code only, comments = comments only. Model: ModernBERT. For comments, we filter the validation set for instances that contain comments. Loss: CE for A and C, weighted CE for B.} 
  \label{fig:cleaning_ablation}
\end{figure}
\autoref{fig:cleaning_ablation} compares performance when using only selected parts of the code snippets filtered by content type (as discussed in \autoref{sec:description}). % extracted by our data cleaning heuristic. 
% A representative example of these extracted categories is provided in \autoref{fig:teaser_image}.
For examples of these extracted parts, see \autoref{fig:teaser_image} or Appendix~\ref{ssec:validation_heuristic}.
Using the original code snippets yields the best performance for all subtasks. Removing the natural language parts impairs performance, particularly for subtasks B and C, suggesting that the natural language in the code snippets helps distinguish the generators. Using code or comments alone is less effective than using both together, though it still performs reasonably well. This shows that both parts contribute to the detection capabilities.


\subsection{Addressing Class Imbalance}


% To analyze the effect of these additional examples on performance, we compare three settings: (1) adding one generated example for each original example, (2) adding two generated examples for each original example, and (3) replacing each original example with a generated one. The examples for both the human class and OpenAI remain unchanged.

To mitigate the class imbalance in subtask B, we experiment with data augmentation and different loss functions.
Using weighted CE slightly improves performance; however, using focal loss is not beneficial (see \autoref{tab:main_res}).
The model we use for fine-tuning is ModernBERT-large with a context length of 2084, and weighted CE. 

Our data augmentation experiments show negative results.
When only one or two examples are added per original example, validation performance decreases slightly (see \autoref{tab:upsampling_model}).
To test the hypothesis that our augmented code snippets differ too much from the original data, we replace all non-human-written code snippets (except for those generated by OpenAI) with their counterparts in the augmented data.
%Using only the newly generated examples in combination with the original human examples and those generated by OpenAI
This considerably worsens the Macro-F1 score, i.e., it is indeed the case that the augmented data follows a different distribution, perhaps due to using a different prompt.
%our augmented code snippets differ too greatly from the original ones, making them unsuitable for fixing the class imbalance.
Our final leaderbord submission does not use any augmented training data.
%e use no augmentation in our public leaderboard submission. % Consequently, we do not employ any augmentation in our approach submitted to the public leaderboard.



\begin{table}
\footnotesize
\centering
\setlength\tabcolsep{4pt}
\begin{tabular}{l|c}
\toprule
Approach & Macro-F1  \\
\midrule
ModernBERT-large & \textbf{68.53} \\
\hspace{2mm} +1 example & 66.26\\
\hspace{2mm} +2 examples & 65.9 \\
\midrule
only augmented &  25.5\\
\bottomrule
\end{tabular}
\caption{Results on validation set of subtask B for different data augmentation settings. '+x example(s)' = x new generated example(s) for each original example, except for OpenAI and human. Only augmented = replace original examples with generated one (original data for human and OpenAI).}
\label{tab:upsampling_results}
\end{table}




\subsection{Analysis of Fine-tuned Models}
\label{sec:analysis}


\begin{figure*}[t]
  \centering

  \begin{subfigure}[t]{0.48\textwidth}
    \centering
    \includegraphics[width=\linewidth]{images/ModernBERT-large_tsne_task_B_markiert.pdf}
    \caption{By LLM.}
    \label{fig:B_LLM_tSNE}
  \end{subfigure}
  \hfill
  \begin{subfigure}[t]{0.48\textwidth}
    \centering
    \includegraphics[width=\linewidth]{images/ModernBERT-large_tsne_lang_B_test.pdf}
    \caption{By programming language.}
    \label{fig:B_language_tSNE}
  \end{subfigure}

  \caption{t-SNE visualizations of the validation set for task B of (fine-tuned) ModernBERT. max\_len=2048, loss=weighted CE. Circles = clusters with snippets generated by LLMs.
  %using ModernBERT-large. The model is trained with weighted cross-entropy loss and an input length of 2048 tokens.
  }
  \label{fig:B_tsne_double}
\end{figure*}

To better understand the fine-tuned models' performance, we analyze their embeddings on the validation set.
First, we use PCA \citep{pearson1901liii} to identify the 50 most important components, then we project the vectors into two-dimensional space using t-SNE \citep{Maaten-2008-Viusalizing}.


\autoref{fig:B_LLM_tSNE} visualizes the embeddings of the instances from the validation set of subtask B by LLM family.
It shows that there is no clear boundary between the eleven code generators. However, groups with similar embeddings can be identified. For example, Mistral, Gemma, IBM-Granite, and DeepSeek-AI form one cluster (red-pink-brown-darkgreen cluster roughly in center), as do Phi, 01-ai, Qwen, OpenAI, BigCode, and Meta-LLaMA (purple-lightgreen-orange cluster at bottom right corner).
Therefore, distinguishing within these groups of models seems to be harder, explaining why we only achieve a Macro-F1 score of about 68\% on this subtask. 
Appendix~\ref{ssec:detailed_res_appendix} provides a more detailed analysis.
% For a detailed analysis of which classes were confused confused with each other, refer to Appendix \ref{ssec:detailed_res_appendix}.


As a comparison, \autoref{fig:B_language_tSNE} visualizes the embeddings of all validation set instances by programming language.
It shows that the clusters of embeddings of human-written code snippets, in particular, correspond to different programming languages.
Programming languages that are syntactically similar tend to have similar representations. For example, C and C++, or Java and C\#.
The areas in the embedding plane corresponding to the two groups of LLMs contain many different programming languages. This indicates that the model identified features in the code snippets that the LLM generators share across programming languages.

Overall, this shows that, especially for human-written code snippets, the model does not have a single representation, but rather distinguishes between programming languages. This may be because about 90\% of the training set instances are human-written, so the model learns to differentiate languages. However, since our task is to detect the generator of the code snippet, not its language, we aim for representations that separate by generator.
Potential future work could include to either incorporate a language identification step and training several models, or to train representations such that they become independent of programming language, e.g., by using adversarial loss functions.
%Potential future work could either incorporate a language identification step and training several models or training of representations such that they become inpendent of programming language, e.g., by using adversarial loss functions. 
%it would be better to have one cluster by generator. Therefore, adding a pre-processing step that makes the input programming language independent before fine-tuning might have been beneficial.


\section{Results on Test Set}

\begin{table}
\footnotesize
\centering
\setlength\tabcolsep{4pt}
\begin{tabular}{c|l|c|c|c}
\toprule
Task & Model & Macro-F1 & Rank & \# Participants \\
\midrule
A & LLM & 53.47 & 41 & 81\\
\midrule
B & ModernBERT & 42.99 & 5 & 34\\
\midrule
C & ModernBERT & 64.77 & 6 & 32\\
\bottomrule
\end{tabular}
\caption{Macro-F1 on private test set for all subtasks.}
\label{tab:result_testset}
\end{table}

\autoref{tab:result_testset} shows the results on the private test sets, which include code generation models, programming languages, and application domains unseen during training. %, which indicate limited generalization of our approach.
The test set contains longer code snippets and considerably fewer natural language explanations (see Appendix~\ref{ssec:dataset_appendix}).

For subtask A, our ModernBERT-based approach achieves a Macro-F1 score of only 26.5. %, highlighting that the fine-tuned model cannot handle unseen programming languages or code domains. 
%Additionally, it cannot handle the different characteristics of the test set.
% Therefore, we submitted a second run using our LLM-based approach and that is the score that appears on the official leaderboard. 
We submitted a second run using our LLM-based approach; this score that appears on the official leaderboard. 
%However, the performance is still poor for binary classification; essentially, it is an expensive random baseline.
On subtasks B and C, however, our model performs strongly: we achieve the 5th and 6th place.
Yet, compared to the validation results, performance drops by about 25 percentage points.
For subtask B, this suggests that identifying new LLMs within the same model family is difficult.
This is in line with the findings of our analysis in section~\ref{sec:analysis}, where embeddings did not distinguish well by LLM (family).

%ur fine-tuned model does not seem to have identified general indicators of all respective model families. However, it is unclear whether an LLM family has a unique coding style.


\section{Discussion and Conclusion}
This paper describes our experiments, which we conducted for SemEval-2026 Task 13 on detecting AI-generated code. With our system based on fine-tuning ModernBERT, we rank 5th on subtask B and 6th on subtask C in the official evaluation.

Our detailed analysis reveals that fine-tuning ModernBERT on the task of identifying code authorship successfully partitions the embedding space into human-authored and LLM-generated instances.
Within the human-written data, programming-language dependent clusters could be identified, while the embeddings pertaining to LLM-written instances seem to capture characteristics of the LLM rather than the programming language. Moreover, our experiments on data augmentation, generating additional samples using LLMs, were not successful.

In sum, despite the high ranking in subtasks B and C, we interpret our findings as a negative result.
The substantial decline from validation to test conditions suggests that finetuning BERT‑sized models leads to pronounced overfitting to characteristics of the data distribution, particularly those shaped by parameters such as the application domain and (likely) the prompt used when generating the LLM-produced code.
%This data distribution also seems to reflect application domain: it may be necessary to adapt code authorship detectors to every new domain.

%However, there are still challenges. For example, identifying the specific LLM family of a given code snippet is much more difficult than making a binary decision about whether it was written by a human or a machine (at least in known domains and languages).
%The results would likely improve if more data were available for fine-tuning, especially given the imbalance in the provided datasets. Unfortunately, our approach of using LLMs for data augmentation impaired performance.
%Furthermore, our results suggest that our approach struggles to generalize detection to domains or programming languages not included in the training. 
%Therefore, future work should focus on improving generalization.
%, e.g., by incorporating a pre-processing step to obtain representations of code snippets that are independent of the programming language.



\section*{Limitations}
One major limitation is the datasets provided in the shared task, since a non-negligible part of the code snippets contain not only code but additional natural language explanations. As seen in the ablation study of the performance of each code snippet component, removing the natural language explanations impairs performance, showing that detection is more difficult with only pure code.

The next limitation goes hand in hand with the previous one. Our method developed for cleaning the provided dataset is only a heuristic based approach. As discussed in Appendix~\ref{ssec:validation_heuristic}, 
the heuristic is not perfect and sometimes misclassifies a line of natural language as code. Therefore, the results regarding the cleaned-up code snippets are not fully realistic, as too much natural language is usually still included. However, even with a perfect cleaning approach, the tendency should remain the same, and the performance should drop even more.

\section*{Acknowledgments}
The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) under the BayernKI project v110ee. BayernKI funding is provided by Bavarian state authorities.

% Custom bibliography entries only
\bibliography{custom}

\appendix
\section{Appendix}

% \subsection{SemEval-2026 Task 13: Detecting Machine-Generated Code}
\subsection{Task Overview}
\label{ssec:task_appendix}

SemEval-2026 Task 13 focuses on the detection of AI-generated code under diverse and realistic conditions. The task is organized into three subtasks:
\begin{itemize}
    \item \textbf{Subtask~A: Binary Machine-Generated Code Detection.} This subtask focuses on binary classification of code snippets as fully human-written or fully machine-generated across seen and unseen programming languages and domains. Systems may be trained on a subset of languages (Python, Java, C++) and evaluated on both seen and unseen languages (JavaScript, Go, PHP, C\#, or C), as well as both seen and unseen application domains. The training domains consist of algorithmic code snippets, for example Leetcode-style problems, while the unseen domains are research and production.
    \item \textbf{Subtask~B: Multi-Class Authorship Detection.} This subtask addresses multi-class authorship attribution by distinguishing human-written code from code generated by one of ten different LLM families (DeepSeek-AI, Qwen, 01-ai, BigCode, Gemma, Phi, Meta-LLaMA, IBM-Granite, Mistral, OpenAI). %, including both seen and unseen generators.
    The test set contains both previously seen generators from the training and unseen generators from the known model families.
    \item \textbf{Subtask~C: Hybrid Code Detection.} This subtask extends the problem to a four-class setting: (i) human-written, (ii) machine-generated, (iii) hybrid code produced through human-LLM collaboration, and (iv) adversarially generated code designed to evade detection.
\end{itemize}

Across all subtasks, systems are evaluated using the macro-averaged F1 score. 
%For more details about the provided datasets for refer to Appendix \ref{ssec:dataset_appendix}.

\subsection{Datasets}
\label{ssec:dataset_appendix}

\begin{table}
\footnotesize
\centering
\setlength\tabcolsep{4pt}
\begin{tabular}{l|rrr}
\toprule
\textbf{Task} & Train & Val & Test \\
\midrule
A & 500k & 100k & 500k \\
\midrule
B & 500k & 100k & 500k \\
\midrule
C & 900k & 200k & 500k \\
\bottomrule
\end{tabular}
\caption{Sizes of dataset splits per subtask.}
\label{tab:dataset_size}
\end{table}

\begin{table*}
\footnotesize
\centering
\setlength\tabcolsep{4pt}
\begin{tabular}{l|l|rrrrrrrr}
\toprule
Task & Split & Python & Java & C++ & Go & PHP & C\# & C & JavaScript \\
\midrule
\multirow{3}{*}{A} & train & 91.46 & 3.86 & 4.68 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\
 & val & 91.46 & 3.86 & 4.68 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\
 & test & 29.62 & 26.59 & 5.04 & 5.87 & 5.46 & 12.92 & 5.86 & 8.64 \\
\midrule
\multirow{3}{*}{B} & train & 27.34 & 27.42 & 7.32 & 5.94 & 5.84 & 12.56 & 5.24 & 8.36 \\
 & val & 27.21 & 27.51 & 7.38 & 5.88 & 5.90 & 12.58 & 5.26 & 8.30 \\
 & test & 28.81 & 25.41 & 9.61 & 7.13 & 3.64 & 10.92 & 6.03 & 8.44 \\
\midrule
\multirow{3}{*}{C} & train & 27.69 & 26.64 & 8.42 & 6.43 & 5.24 & 11.99 & 5.39 & 8.19 \\
 & val & 27.68 & 26.72 & 8.32 & 6.35 & 5.33 & 11.93 & 5.43 & 8.24 \\
 & test & 24.06 & 18.24 & 8.54 & 8.12 & 7.57 & 14.52 & 9.95 & 9.00 \\
\bottomrule
\end{tabular}
\caption{Distribution of programming languages per subtask and split in percent.}
\label{tab:lang_dist_per_task}
\end{table*}

\begin{table}
\footnotesize
\centering
\setlength\tabcolsep{4pt}
\begin{tabular}{l|l|rrr}
\toprule
Task & Split & >512 & >1024 & >2048 \\
\midrule
\multirow{3}{*}{A} & train & 16.52 & 3.71 & 0.49 \\
 & val & 16.48 & 3.69 & 0.49 \\
 & test & 24.10 & 8.25 & 1.42 \\
\midrule
\multirow{3}{*}{B} & train & 23.37 & 8.51 & 1.53 \\
 & val & 23.37 & 8.46 & 1.50 \\
 & test & 33.62 & 5.27 & 0.86 \\
\midrule
\multirow{3}{*}{C} & train & 25.72 & 6.12 & 0.95 \\
 & val & 25.78 & 6.20 & 0.98 \\
 & test & 43.92 & 19.11 & 4.54 \\
\bottomrule
\end{tabular}
\caption{Percentage of instances having at least a certain number of tokens. Tokenization based on ModernBERT.}
\label{tab:number_of_tokens_per_split}
\end{table}

\begin{table}
\footnotesize
\centering
\setlength\tabcolsep{4pt}
\begin{tabular}{l|l|rrr}
\toprule
Task & Split & Code & Comments & NL \\
\midrule
\multirow{3}{*}{A} & train & 98.4 & 30.11 & 15.02 \\
 & val & 98.37 & 30.12 & 15.03 \\
 & test & 99.79 & 37.27 & 4.41 \\
\midrule
\multirow{3}{*}{B} & train & 99.98 & 33.95 & 3.21 \\
 & val & 99.98 & 34.11 & 3.24 \\
 & test & 99.97 & 51.13 & 2.31 \\
\midrule
\multirow{3}{*}{C} & train & 99.66 & 46.57 & 5.49 \\
 & val & 99.66 & 46.67 & 5.54 \\
 & test & 99.8 & 65.76 & 5.4 \\
\bottomrule
\end{tabular}
\caption{Percentage of instances having at least one token of the respective type, i.e., code, comment, or natural language (NL), in the code snippet.}
\label{tab:share_of_data_containing_codepart}
\end{table}

\begin{table*}
\footnotesize
\centering
\setlength\tabcolsep{4pt}
\begin{tabular}{l|rrrrrrrrrrr}
\toprule
\multicolumn{12}{l}{\textbf{Task A}} \\
Split & Human & AI-generated &  &  &  &  &  &  &  &  &  \\
\midrule
train & 47.70 & 52.30 &  &  &  &  &  &  &  &  &  \\
val & 47.70 & 52.30 &  &  &  &  &  &  &  &  &  \\
\midrule
\multicolumn{12}{l}{\textbf{Task B}} \\
Split & Human & DeepSeek & Qwen & 01-ai & BigCode & Gemma & Phi & Meta & IBM-Granite & Mistral & OpenAI \\
\midrule
train & 88.42 & 0.83 & 1.80 & 0.61 & 0.45 & 0.39 & 1.16 & 1.64 & 1.63 & 0.92 & 2.16 \\
val & 88.49 & 0.85 & 1.76 & 0.65 & 0.44 & 0.37 & 1.12 & 1.70 & 1.58 & 0.90 & 2.15 \\
\midrule
\multicolumn{12}{l}{\textbf{Task C}} \\
Split & Human & AI-generated & Hybrid & Adversarial &  &  &  &  &  &  &  \\
\midrule
train & 53.94 & 23.39 & 9.50 & 13.17 &  &  &  &  &  &  &  \\
val & 53.94 & 23.38 & 9.50 & 13.17 &  &  &  &  &  &  &  \\
\bottomrule
\end{tabular}
\caption{Class distribution across subtasks and splits.}
\label{tab:class_distribution}
\end{table*}

% \begin{figure*}[t]
% \centering
% \begin{lstlisting}[style=codebox, language=Python]
% def is_wilson_prime(p):
%     if not isinstance(p, int) or p < 2:
%         return False
    
%     factorial = 1
%     for i in range(2, p):
%         factorial *= i

%     return (factorial + 1) % (p * p) == 0

% # Example usage:
% # print(is_wilson_prime(5))  # True
% # print(is_wilson_prime(13)) # True
% # print(is_wilson_prime(563))# True
% # print(is_wilson_prime(2))  # False
% ```

% This function checks whether a given number `p` is a Wilson prime. It calculates `(P-1)! + 1` and checks if it is divisible by `P * P`. Be cautious with large numbers due to factorial growth.
% \end{lstlisting}
% \caption{Example code snippet from the train set of subtask A, which contains not only code but also additional LLM output.}
% \label{fig:ex_dataset_not_cleaned}
% \end{figure*}


\begin{figure*}[t]
\centering
\begin{lstlisting}[style=codebox]
for checkstyle errorTEST MSG(2,44):
For 1 ')' expected

Submitted


-----Kattis Solutions-----

City-state long
prefix brute force on combinatorics then binary search likes to code is my favorite
\end{lstlisting}
\caption{Example code snippet from the validation set of subtask A that does not contain any meaningful code.}
\label{fig:ex_dataset_not_even_code}
\end{figure*}

\begin{figure*}[t]
\centering
\begin{lstlisting}[style=codebox, mathescape=true]
'''
def solve(n, sure_list):
    count = 0
    for sure in sure_list:
        if sum(sure) >= 2:
            count += 1
    return count
n = int(input())
sure_list = [list(map(int, input().split())) for _ in range(n)]
print(solve(n, sure_list))
'''

This problem can be solved by using a simple loop to iterate over the list of sure lists. For each sure list, we check if the sum of the elements is greater than or equal to 2. If it is, we increment the count. Finally, we return the count.
The time complexity of this solution is O(n), where n is the number of problems in the contest.

'''<|endoftext|>'''

You are given a string s and an integer k. You can perform the following operation on the string any number of times:

Choose any two adjacent characters in the string and swap them.

Your task is to determine the minimum number of operations required to transform the string into a string with the lexicographically smallest possible order.

Input
The first line of the input contains a single integer t (1 $\le$ t $\le$ 100) - the number of test cases. Then t test cases follow.

Each test case starts with a line containing two integers n and k (1 $\le$ n $\le$ 100, 1 $\le$ k $\le$ 100) - the length of the string and the number of operations you can perform.

The second line of each test case contains a string s of length n. Each character of the string is a lowercase English letter ('a' to 'z').

Output
For each test case, output a single integer - the minimum number of operations required to transform the string into a string with the lexicographically smallest possible order.

Example
Input:
2
5 2
abcde
5 2
abcdd

Output:
2
1

In the first test case, you can perform the following operations:
- Swap 'a' and 'b' to get 'bacd'
- Swap 'b' and 'c' to get 'abcde'

In the second test case, you can perform the following operation:
- Swap 'a' and 'd' to get 'abcdf'
Note
In the first test case, the lexicographically smallest possible order is 'abcde'.
In the second test case, the lexicographically smallest possible order is 'abcd'.

'''
def solve(n, k, s):
    s = list(s)
    s.sort()
    operations = 0
    for i in range(n-1):
        if s[i] > s[i+1]:
            operations += 1
            if operations == k:
                return operations
    return operations
t = int(input())
for _ in range(t):
    n, k = map(int, input().split())
    s = input()
    print(solve(n, k, s))
'''

This problem can be solved by sorting the string and then checking if the adjacent characters are in the correct order. If they are not, we perform the operation and increment the count of operations. We repeat this process until the string is sorted or the number of operations reaches k. [...]
\end{lstlisting}
\caption{Shortened example code snippet from the validation set of subtask A, which contains not only code but also additional LLM output.}
\label{fig:ex_dataset_not_cleaned}
\end{figure*}

\begin{figure*}[t]
\centering
\begin{lstlisting}[style=codebox]
valid version for the language; all others can also just parse a test that does an incorrect solution in real-time before testing all code solutions can cause the "wrong output".

The largest portion of this question lies in handling corner solutions without leading and trailing zeros.
Can a brute-force solver write incorrect answers every loop and end up printing a larger smallest number?
But doesn't someone trying things correctly and hoping to get right but is giving the correct answer end be caught as doing multiple tasks?
Not a bug, at least as of c08e - fixed by pax. "Not a thing"
However as of current rules  there is now actually one case which really does both of those together.... - which still is better then previous as for time purposes is much shorter

To ensure consistent behavior as to what makes a right solution

def right_solution():
    n=int(-i*1/8*abs(C[i][B].item()) for inB=-B,B+~0: yield abs(C[i, B].A[:31]) is the slowliest possible approximation for B. By converting to i32 only.
but even faster, which is probably optimal anyway would still
let the output here not the largest by even writing A2A0 to an integer twice twice faster would give correct on 94, just use an int

But there are plenty better ways you can actually do 2 3 byte float. I.e a way a12x + byz . 5 byte so A ==0 and = (x, y = x0-az+xy, the rest in fact also A +114*..A1Bz if correct about x to only do x  not have 7*4x or  any byte, all three should remain signed that at those only uses the right edge are much smaller then so they'd almost just use those no bit.
B which is smaller

def right_so0me(): int(-n*i*(i<0); it must do i1-i-...-B-8*i<o (abs + abs(B))
when making the smallest non-opt as the first it might get the least so the right answer


...x B == in34[xyw==>>=8xy for- |~>01xy 3 bytes, + but even worse + not use xy with y3e instead get something like int1
for, like most things the idea on what this all means should of course be

for f(-x for i+x for j

This was also meant to do an index over xy on 7 that index might change or how much y index and the other bytes like 0
will all become bytes

f=-y1 + that many even then instead of .B or .6 you have a- if at least only can do lessy or even 9 byte than i13, as least would also need .80y1
xw you could get if is greater -that less use if any such as x w+ we know as only and -8.09 are still ok,
for all is for both xy as well with but maybe both even just now a better way is this than other we would always make it out well be a long thing to see though
I suggest now that this be -n of all so if not 8 that is the for use so if f

B=-f+B+-c-(.c+.ab51-.8x)-+-j or then be the xy of x to some thing the -c the be

What a right answer does here? We mean

...n/2*8 in only there if is all have bytes such in f9/that only there by .y also by 1.x 28 would have only i2 on there not over though the b3

+1B at least is all i of but then x3x as both here else also -9 would give only on such . and it makes all on same that could of the (c). at here a with in other, what it really just for

- the first and is can have just can use, the best are then we are only but now if but we and then that this such .n-1 for and even this all this such we would can only both xy the then we would in 7+ + x
is it still x on then must but so can use an here either must in (a+x+n)x+-.a21+ x for any any for

. of a- x*4 for that byte from else one so they all on this may or the it gets to -7 in get it may here. Even though 6 also both i also are bytes also xy for still on make this

we even make n1 -512.8 if (abs x even.4
I'm using int n for as only - and any I did such have less just we were f if so now xy
\end{lstlisting}
\caption{Example \enquote{code} snippet from the train set of subtask A, where it is impossible to distinguish between code and other LLM output.}
\label{fig:ex_dataset_impossible_for_heuristic}
\end{figure*}

This section offers a more in-depth analysis of the datasets provided for the shared task.

Each subtask has its own training, validation, and test sets. The sizes of the dataset splits per subtask are shown in \autoref{tab:dataset_size}. The training and validation sets include the code snippet, the generator (i.e., human or concrete LLM), the programming language in which the code snippet is written, and the class label. The test sets include only the code snippet.


\autoref{tab:lang_dist_per_task} provides an overview of the distribution of programming languages per subtask. Since the programming languages are not annotated for the test sets, we predict them using a programming language identification model from Hugging Face.\footnote{\href{https://huggingface.co/philomath-1209/programming-language-identification}{https://huggingface.co/philomath-1209/programming-language-identification}}
To validate these predictions, we predict the language of each instance in the training sets of all subtasks because these instances have language ground truth annotations. Accuracy ranges from 94.23 to 95.94 percent. Therefore, while the predicted language distributions on the test set are plausible, they are not entirely accurate.

Many of the code snippets contain more than 512 tokens, and some contain more than 1024 tokens (see \autoref{tab:number_of_tokens_per_split}). Since fewer than two percent of instances in the training and validation splits contain more than 2048 tokens, our ModernBERT experiments do not use a larger input length, thereby reducing the required computational resources. Notably, the test sets differ from the training and validation sets in that they have more instances with higher token counts.

\autoref{tab:share_of_data_containing_codepart} shows the percentage of instances having at least one token of the respective type, i.e., code, comment, or natural language, in the code snippet. Noticeable is the difference between the train/validation set and the test set for comments only and natural language. 
Minor deviations can be explained by the fact that the programming language is predicted and is therefore only correct about 95\% of the time. In the 5\% of cases with incorrect predictions, our heuristic may be less effective since it looks for the wrong programming language specifics. 
However, this does not explain the considerable differences in terms of the amount of comments, for example. It appears that the instances in the test set contain considerably more comments than those in the training and validation sets, which makes predictions more difficult.

The class distribution is shown in \autoref{tab:class_distribution}. It reveals that, in particular for subtask B, the class distribution is highly imbalanced, i.e., the human class comprises nearly 90\% of the training and validation instances. Therefore, learning to predict the remaining 10 classes is difficult.


\subsection{Validation of Heuristic}
\label{ssec:validation_heuristic}

\begin{figure*}[t]
\centering
\begin{lstlisting}[style=codebox, escapeinside={(*@}{@*)}]
(*@\textbf{Entire code snippet:}@*)
def can_distribute_coins(t, test_cases):
    results = []
    for i in range(t):
        a, b, c, n = test_cases[i]
        max_coins = max(a, b, c)
        total_needed = (3 * max_coins) - a - b - c
        if total_needed <= n and (n - total_needed) % 3 == 0:
            results.append("YES")
        else:
            results.append("NO")
    return results
# Example usage:
t = 5
test_cases = [
    (5, 3, 2, 8),
    (100, 101, 102, 105),
    (3, 2, 1, 100000000),
    (10, 20, 15, 14),
    (101, 101, 101, 3)
]
print(can_distribute_coins(t, test_cases))
```
This function, `can_distribute_coins`, takes the number of test cases `t` and a list of tuples `test_cases`, where each tuple contains the values of `a`, `b`, `c`, and `n` for a test case. It returns a list of "YES" or "NO" strings indicating whether it is possible to distribute the coins as described.

(*@\textbf{Cleaned (natural language removed):}@*)
def can_distribute_coins(t, test_cases):
    results = []
    for i in range(t):
        a, b, c, n = test_cases[i]
        max_coins = max(a, b, c)
        total_needed = (3 * max_coins) - a - b - c
        if total_needed <= n and (n - total_needed) % 3 == 0:
            results.append("YES")
        else:
            results.append("NO")
    return results
# Example usage:
t = 5
test_cases = [
    (5, 3, 2, 8),
    (100, 101, 102, 105),
    (3, 2, 1, 100000000),
    (10, 20, 15, 14),
    (101, 101, 101, 3)
]
print(can_distribute_coins(t, test_cases))
```

(*@\textbf{Natural language only:}@*)
This function, `can_distribute_coins`, takes the number of test cases `t` and a list of tuples `test_cases`, where each tuple contains the values of `a`, `b`, `c`, and `n` for a test case. It returns a list of "YES" or "NO" strings indicating whether it is possible to distribute the coins as described.

(*@\textbf{Code only:}@*)
def can_distribute_coins(t, test_cases):
    results = []
    for i in range(t):
        a, b, c, n = test_cases[i]
        max_coins = max(a, b, c)
        total_needed = (3 * max_coins) - a - b - c
        if total_needed <= n and (n - total_needed) % 3 == 0:
            results.append("YES")
        else:
            results.append("NO")
    return results
t = 5
test_cases = [
    (5, 3, 2, 8),
    (100, 101, 102, 105),
    (3, 2, 1, 100000000),
    (10, 20, 15, 14),
    (101, 101, 101, 3)
]
print(can_distribute_coins(t, test_cases))
```

(*@\textbf{Comments only:}@*)
# Example usage:

\end{lstlisting}
\caption{Code snippet split into different parts using our data cleaning heuristic. This example shows that the heuristic is not perfect; for instance, it treats \`{}\`{}\`{} as part of the code.}
\label{fig:ex_heuristic_parts}
\end{figure*}

To validate the heuristic developed for cleaning and splitting the provided code snippets into code, comments, and natural language parts, we manually analyze 100 randomly chosen examples from the dataset. Fifty were sampled from the validation set of subtask A, and fifty were sampled from the training set of subtask B.

The heuristic and the annotator achieve full agreement on the 50 instances of subtask B in the training set. Of the 50 randomly selected instances of subtask A, the heuristic and the annotator disagree in seven cases. 
The difference in agreement is because of the fact that the training set for subtask A contains many more instances with additional natural language descriptions of the code snippets (see \autoref{tab:share_of_data_containing_codepart}).
While identifying comments in the code is straightforward, extracting the natural language parts is challenging. The disagreement stems from lines of code that the heuristic incorrectly classifies as code instead of natural language. 
For example, in three cases, the heuristic incorrectly classifies a line containing a single number as code. While this could technically be treated as code and will not prevent the code from compiling, it is incorrect in the context of these code snippets because the number belongs to example input sequences that the LLM provided (like in \autoref{fig:ex_dataset_not_cleaned}).
In two other cases, the heuristic incorrectly classifies lines containing error messages as code. This is understandable, as these lines contain several code keywords. However, this stack trace output cannot be executed as code.

\autoref{fig:ex_dataset_not_even_code} and \autoref{fig:ex_dataset_impossible_for_heuristic} show examples from the provided datasets where it would be difficult even for a human annotator to decide which line should be counted as code or natural language. Therefore, it is not surprising that our heuristic-based approach does not perfectly align with the annotator's decisions.

An example of how the heuristic splits a given code snippet into three different parts, i.e., code, comments, and natural language, can be seen in \autoref{fig:ex_heuristic_parts}.


% \newpage

\subsection{Random Baseline}
\label{ssec:random_baseline}
This section offers a more in-depth description of the random baseline we used for comparison with our model results. This approach samples class labels according to the empirical class distribution observed in the validation split of each subtask, as reported in \autoref{tab:class_distribution}. %Table \ref{tab:lang_dist_per_task}. 
More specifically, we use numpy's default random number generator for prediction.

To account for the stochastic nature of this approach, we evaluate the random baseline across multiple runs with different random seeds and report the mean and standard deviation (std) of the resulting Macro-F1 scores. 
% All experiments were conducted using the following fixed set of seeds:
All experiments are conducted using the following fixed set of seeds:
\texttt{[757069, 397239, 554869, 147529, 327388, 153159, 92000, 322768, 506237, 99617]}. Detailed results of all runs as well as mean and standard deviation per subtask are reported in \autoref{tab:random_baseline_results}.

The baseline does not involve any training procedure and operates directly on the validation data. It serves solely as a lower-bound reference for interpreting the performance of the proposed methods across all subtasks.

\begin{table}[t]
\footnotesize
\centering
\setlength\tabcolsep{4pt}
\begin{tabular}{r|ccc}
\toprule
\textbf{Seed} & \textbf{Subtask A} & \textbf{Subtask B} & \textbf{Subtask C} \\
\midrule
757069 & 0.49926 & 0.08902 & 0.25166 \\
397239 & 0.49937 & 0.08999 & 0.24962 \\
554869 & 0.50017 & 0.09188 & 0.24943 \\
147529 & 0.50103 & 0.09012 & 0.25235 \\
327388 & 0.50085 & 0.09183 & 0.25096 \\
153159 & 0.49851 & 0.08977 & 0.25053 \\
92000  & 0.49931 & 0.09213 & 0.25176 \\
322768 & 0.49910 & 0.09143 & 0.25135 \\
506237 & 0.50033 & 0.09130 & 0.25087 \\
99617  & 0.49876 & 0.08984 & 0.24866 \\
\midrule
\textbf{mean} & 0.49967 & 0.09073 & 0.02507 \\
\textbf{std}  & 0.00082 & 0.00104 & 0.00111 \\
\bottomrule
\end{tabular}
\caption{Macro-F1 scores of the random baseline across all subtasks, reported per random seed as well as mean and standard deviation over all runs.}
\label{tab:random_baseline_results}
\end{table}


% \subsection{Data Upsampling}
\subsection{LLM-based Approaches}
\label{ssec:llm_based_approaches}

\begin{table}[t]
\footnotesize
\centering
\setlength\tabcolsep{4pt}
\small
    \begin{tabular}{l|l}
    \toprule
    LLM family & Model \\
    \midrule
         Deepseek&DeepSeek-Coder-V2-Lite-Instruct \\
         BigCode& starcoder2-15b-instruct-v0.1\\
         Gemma&codegemma-7b-it \\
         01-ai&Yi-Coder-9B-Chat\\
         Mistral&Mistral-7B-Instruct-v0.3 \\
         Phi&Phi-3-mini-4k-instruct\\
         IBM-granite&granite-3.3-8b-instruct \\
         Meta-Llama&Llama-3.1-8B-Instruct \\
         Qwen&Qwen2.5-Coder-32B-Instruct \\
    \bottomrule
    \end{tabular}
    \caption{LLMs used for data augmentation}
    \label{tab:upsampling_model}
\end{table}

\begin{figure*}[t]
\centering
\begin{lstlisting}[style=codebox, escapeinside={(*@}{@*)}]
(*@\textbf{System message:}@*)
You are an assistant to rewrite code. Rewrite it in the same language and precisely as possible.

(*@\textbf{User message:}@*)
Code: {code}

Task: 
You are given a {language} code snippet. Your task is to rewrite it.
Change the variable names and the code structure. The functionality and the language should remain the same.
Only respond with the rewritten code snippet. Do not provide any explanation or justification. Do not write 'here is the rewritten code'.
Do not add the name of the used programming language in your answer. 
Answer:
\end{lstlisting}
\caption{Prompt template for data augmentation.}
\label{fig:user_prompt_template_upsampling}
\end{figure*}

\textbf{Data Augmentation.}
\autoref{tab:upsampling_model} shows the LLMs that are used in our data augmentation approach. 
% For each model family, one LLM is chosen, but not for OpenAI, since these models are closed-source.
For each model family, one LLM is chosen, but not for OpenAI, since this model family is closed-source.
The prompt template used for generating these new examples can be seen in \autoref{fig:user_prompt_template_upsampling}. The models are instructed to only change variable names and code structure. However, they should not alter the functionality or programming language. Furthermore, the models should omit providing any additional natural language explanations. 
\autoref{tab:upsampling_data_dist} shows the resulting class distribution. 

\begin{table}
    \footnotesize
\centering
\setlength\tabcolsep{4pt}
    \begin{tabular}{l|rr}
        \toprule
        LLM family & +1 example & +2 examples \\
        \midrule
             Human& 81.85& 75.68\\
             DeepSeekAI& 1.48 &  2.05 \\
             Qwen& 2.96 &  4.11 \\
             01-ai& 1.00 & 1.54\\
             BigCode& 0.74 &  1.03 \\
             Gemma& 0.74 &  1.03 \\
             Phi& 1.85 &  2.57 \\
             Meta-Llama& 2.96 & 4.11 \\
             IBM-Granite& 2.96 & 4.11  \\
             Mistral& 1.48 &  2.05 \\
             OpenAI& 1.85 &  1.71 \\
        \bottomrule
    \end{tabular}
    \caption{This shows the resulting data distribution for the respective data augmentation approaches for the training set for task B. ’+x
example(s)’ = x new generated example(s) for each
original example, except for OpenAI and human.}
    \label{tab:upsampling_data_dist}
\end{table}

\begin{figure*}[t]
\centering
\begin{lstlisting}[style=codebox, escapeinside={(*@}{@*)}]
(*@\textbf{System message:}@*)
You are an assistant distinguishing machine-generated code from human-written code. Answer the question as precisely as possible.

(*@\textbf{User message:}@*)
Code: {code}

Task:
You are given a code snippet. Your task is to determine whether the code snippet provided was written by a human or generated by a machine. 

Focus on:
- variable naming
- comment density and style
- code structure and modularization
- formatting and indentation
- traces of generator outputs (docstrings, boilerplate, prompt echoes)
- handling of edge cases
- ...

If the code is written by a human, return '0'. If it is machine-generated, return '1'. Think step by step.

Provide your answer in the following format:
```
Explanation: "Let's think step by step..."
---
Answer: 0 or 1
```

Explanation:
\end{lstlisting}
\caption{Prompt template used for binary classification on subtask A with LLMs.}
\label{fig:prompt_llm_cot_binary_A}
\end{figure*}


\textbf{LLMs for Detection.}
\autoref{fig:prompt_llm_cot_binary_A} shows the prompt template used in our 0-shot LLM-based detection approach for subtask A.
The LLM is instructed to determine whether the provided code snippet was written by a human or generated by a machine by focusing on several indicators.
To improve the decision-making process, we use 0-shot CoT prompting, i.e., we instruct the model to \textit{think step by step} \citep{Kojima-2022-Large}.


\subsection{Detailed Results on Validation Set}
\label{ssec:detailed_res_appendix}

\begin{table}
\footnotesize
\centering
\setlength\tabcolsep{4pt}
\begin{tabular}{l|c|ccc}
\toprule
Model & max\_len & A & B & C \\
\midrule
ModernBERT & 2048 & \textbf{99.81} & 67.58 & \textbf{89.38} \\
\hspace{2mm} + focal loss & 2048 & -  & 67.17 & - \\
\hspace{2mm} + weighted-ce & 2048 & - & \textbf{68.53} & - \\
\hspace{2mm} + mean-pooling & 2048 & - & 67.51 & 89.25 \\
ModernBERT & 1024 & 99.80 & 67.32 & 89.26 \\
ModernBERT & 512 & 99.71 & 65.11 & 88.04 \\
%UniXcoder & 512 & 99.63 & 61.27 & -\\
CodeBERT & 512 & 99.50 & 57.44 & 84.61 \\
RoBERTa & 512 & 99.47 & 56.36 & 84.26 \\
\midrule
CodeT5 (enc\_only) & 512 & 99.55 & 59.32 & 86.09\\
CodeT5 (seq2seq) & 512 & 99.59 & 54.95 & - \\
\midrule
Qwen2.5-Coder-32B & - & 51.35 & - & - \\
\midrule
random & - & 49.97 & 9.1 & 25.07 \\
\bottomrule
\end{tabular}
\caption{Macro-F1 scores on validation sets of all subtasks. max\_len indicates the maximum input length for the models. Unless stated otherwise, input for classification head is embedding of CLS token. seq2seq=encoder-decoder approach.}
\label{tab:main_res_detailed}
\end{table}

\textbf{Detailed analysis of best approach.}
\autoref{fig:b_confusion_matrix_modernbert} shows the confusion matrix of the approach based on fine-tuning ModernBERT-large for subtask B. It shows that the human class is detected perfectly, while the different LLM families are harder to distinguish. Therefore, this approach could be used to make predictions for subtask A when assuming that any predicted LLM family indicates machine-generated code. We tried this on the public test set, which has only 1,000 instances, and achieved a Macro-F1 score of about 95\%, showing that it would work technically. However, we do not use this method for our official leaderboard submission, as the rules only permit the use of the given training set for each subtask.
Furthermore, as seen in the t-SNE analysis, we can identify two clusters of LLMs, whose models are often confused with each other. The first cluster includes Mistral, Gemma, IBM-Granite, and DeepSeek-AI, while the second cluster includes Phi, 01-ai, Qwen, OpenAI, BigCode, and Meta-LLaMA. 
The confusion matrix shows that LLMs are often incorrectly classified as human-written, but the opposite is rarely observed.

%\autoref{tab:main_res_detailed} shows that using specialized coding encoder-only models like UniXcoder\footnote{\href{https://huggingface.co/microsoft/unixcoder-base}{https://huggingface.co/microsoft/unixcoder-base}} \citep{guo-etal-2022-unixcoder}, does not improve performance compared to our approach based on ModernBERT, even when the input length is the same.

\begin{figure}[t]
\includegraphics[width=1\columnwidth]{images/modernbert_B_confusion_matrix.pdf}
  \caption {Confusion matrix for ModernBERT on validation set of subtask B. Input length of 2048 tokens and weighted CE loss.} 
  \label{fig:b_confusion_matrix_modernbert}
\end{figure}

%\subsection{Additional Qualitative Analysis}

\textbf{Fine-tuning CodeT5.}
\citet{orel-etal-2025-codet} have identified fine-tuning CodeT5 \citep{wang-etal-2021-codet5} as one of the most effective approaches. 
However, they do not explain exactly how they are using the encoder-decoder model for classification. 
Therefore, we implement two versions of fine-tuning CodeT5 for this code detection task. The first uses CodeT5 as an encoder-only model. However, unlike BERT-based models, T5 has no CLS token. Thus, we use the mean of all encodings as input to the classification head. The mean pooling is computed mask-aware, i.e., we ignore padding tokens.
The second variant uses the encoder and decoder of the base model and it is trained to predict the class name directly. In this setting, we prepend \enquote{classify:} before the code snippet.
We use CodeT5-large\footnote{\href{https://huggingface.co/Salesforce/codet5-large}{https://huggingface.co/Salesforce/codet5-large}} for both versions and train it for 10 epochs on the respective training set. We use a batch size of 64, a learning rate of 5e-5, and early stopping with patience of 5 (evaluating on the validation set every 2000 steps).

The results of these two settings are shown in \autoref{tab:main_res_detailed}. The encoder-only version performs better than CodeBERT, possibly due to its stronger encoder. For example, CodeT5 (large) has 770 million parameters, whereas CodeBERT has only 125 million. However, it cannot compete with ModernBERT's performance. The performance of our encoder-decoder variant suggests that additionally using the decoder component of CodeT5 to predict class names directly is not beneficial.


\textbf{Input to classification head.} To determine the most effective way to aggregate sequence information for the classification head, we compare using ModernBERT's native [CLS] token output with mean pooling across all token embeddings as input.
While the [CLS] token is specifically designed to serve as a classifier token in BERT-like architectures, mean pooling provides an alternative by averaging the embeddings of all tokens. 
Again, we compute mean pooling mask-aware, i.e., ignore padding tokens.

The results of this approach can be found in \autoref{tab:main_res_detailed}. They indicate that for ModernBERT, the representation captured by the [CLS] token is already sufficiently robust to represent complex code structures. Since mean pooling did not yield a performance gain,  we utilize the [CLS] token embedding as input to the classification head for our final submissions.


\end{document}
