\documentclass[12pt,a4paper]{article}
\usepackage[hyperref]{QL}
\usepackage{times}
\usepackage[utf8]{inputenc}
\usepackage{babel}
\usepackage[T1]{fontenc}
\usepackage{listings}
\usepackage{xcolor}
\usepackage{adjustbox}

\lstset{
  aboveskip=1ex,
  backgroundcolor=\color{gray!25},
  basicstyle=\small\ttfamily,
  belowskip=1ex,
  breaklines=true,
  columns=fullflexible,
  framerule=0pt,
  framexrightmargin=0em,
  framexleftmargin=0em,
  tabsize=2
}

\aclfinalcopy

\newcommand\BibTeX{B\textsc{ib}\TeX}

\title{TueSents at SemEval-2024 Task 8: \\
Predicting the Shift from Human Authorship\\
to Machine-generated Output in a Mixed Text
%Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection - Subtask C. Human-Machine Mixed Text Detection\\
}

\author{Valentin Pickard and Hoa Do \\
        Seminar für Sprachwissenschaft\\
         Eberhard Karls Universität Tübingen, Germany \\
         %Keplerstraße 2, 72074 Tuebingen, Germany\\
         \texttt{\{valentin.pickard, hoa.do\}@student.uni-tuebingen.de}
  }

  
\begin{document}

\maketitle

\begin{abstract}
This paper describes our approach and results for the SemEval 2024 task of identifying the token index in a mixed text where a switch from human authorship to machine-generated text occurs. We explore two BiLSTMs, one over sentence feature vectors to predict the index of the sentence containing such a change and another over character embeddings of the text. As sentence features, we compute token count, mean token length, standard deviation of token length, counts for punctuation and space characters, various readability scores, word frequency class and word part-of-speech class counts for each sentence. class counts. The evaluation is performed on mean absolute error (MAE) between predicted and actual boundary word index. While our competition results were notably below the baseline, there may still be useful aspects to our approach.
\end{abstract}



\section{Introduction}
% What is the task about and why is it important? Be sure to mention the language(s) covered and cite the task overview paper. ~1 paragraph
With the rapid proliferation of Large Language Models (LLMs) that are able to produce fluent texts in response to user queries across a wide range of domains and topics, concerns are raised about the potential misuses of such powerful tools. In spite of their fluency, LLM-generated texts may contain factual errors, inadvertently spreading misinformation. Another common issue occurs in the education system, where students may attempt to pass off the responses of such an LLM as their own work, evading commonly used safeguards against plagiarism. Given the overwhelming volume of potentially machine-generated content, it is desirable to have automated means of detecting such texts to address the above-mentioned issues. In this task \cite{semeval2024task8}, we examine exclusively English mixed texts, where a switch from human authorship to LLM output occurs at most once in a text sample (some samples are entirely machine-generated). To us, this  models a plausible use case, where a human user employs an LLM to finish their work for them. For each sample, the task is to predict the token index at which the authorship change occurs. % What is the main strategy your system uses? ~1 paragraph
We observe, that due to the structure of the samples, we can reformulate the task more generally as trying to detect an authorship change and its location within the sample texts, without explicitly trying to detect the presence of LLM-generated text. This allows us to adapt more traditional, computationally relatively inexpensive approaches to stylometry and authorship identification/attribution. While the task is formulated as prediction of a boundary word, we begin by identifying the boundary sentence in which the authorship change occurs. For each sentence, textual feature vectors are extracted and combined with character n-gram information, those sentence vectors are then fed into a Bidirectional LSTM network \cite{hochreiterschmidhuber1997} which is trained to predict the boundary sentence. %What did you discover by participating in this task? Key quantitative and qualitative results, such as how you ranked relative to other teams and what your system struggles with. ~1 paragraph
We found that our approach performed reasonably well in-domain on the development set, in spite of inevitably introducing some token offset error by only making sentence level predictions and choosing the middle tokens, but failed out-of-domain on the test set, ranking at 26 out of 30 in the competition on subtask C.\par



\section{Background} 
%In your own words, summarize important details about the task setup: kind of input and output (give an example if possible); what datasets were used, including language, genre, and size. If there were multiple tracks, say which you participated in.
In accordance with the task guidelines we do not use external data, but use the English subsets of larger data sets from subtask A and B to extract a character vocabulary. The subtask C dataset comprises a bit over 4000 texts with 505 pre-split into a dev set by the task authors, each text labeled with the index of the boundary word. The following table shows token and sentence counts for train and dev set, when tokenized by splitting on whitespace (U+0020) as in the task baseline model. Sentence splits are determined subsequently on the token lists by identifying sentence-final tokens using our detection regex, this is done to ensure matching the given boundary word labels. 

\begin{table}[ht]
\centering
\resizebox{.5\textwidth}{!}{
\begin{tabular}{l|l|l}
               & train          & dev           \\ \hline
texts          & 3,649           & 505           \\
sentences      & 41,570          & 5,628          \\
tokens (types) & 864,153 (29,593) & 116,221 (8,641) \\
chars          & 5,933,701        & 803,771        \\
avg. sentences & 11.4           & 11.1          \\
avg. boundary  & 3.4            & 3.4          
\end{tabular}
}
\caption{Task data statistics}
\end{table}

We observe that about half of the samples contain 4 - 11 sentences and sentence count per sample ranges from 1 to 76, with 24 samples containing just one sentence, like e.g. \emph{"We have added a 2+ page \textbf{discussion} on the experimental results, highlighting the superiority of the ARC-based models and their impact on the field of deep learning."} (boundary word `discussion' in bold). While on average the author switch occurs in the fourth sentence, in about 15-20\% of the samples the switch occurs in the first sentence. Examining the boundary word position within their respective sentences we found an average offset of -1.6 (train set) or -1.8 (dev set) from the middle of the sentence, i.e. the switch occurs slightly before mid sentence on average.\par

An example text, split in sentences and tokens can be observed below, with the boundary word \textbf{"baseline"} at index 20 highlighted:

% \begin{table*}[t]
% \centering
% \begin{tabular}{lll}
% \textbf{label} & \textbf{start token index} & \textbf{tokens}                                                                                        \\
% 0              & 0                          & ['The', 'paper', 'proposes', 'a', 'method', 'to', 'recognize', 'time', 'expressions', 'from', 'text.'] \\
% 1 &
%   11 &
%   ['It', 'is', 'a\textbackslash r\textbackslash nsimple', 'rule-based', 'method,', 'which', 'is', 'a', 'strong', '\textbf{baseline}', 'for', 'time', 'expression', 'recognition.'] \\
% 0 &
%   25 &
%   ['The', 'authors', 'analyze', 'different', 'datasets', 'and', 'discover', 'that', 'only', 'a', 'small', 'set', 'of', 'words', 'are', 'consistently', 'used', 'to', 'convey', 'time', 'information.']
% \end{tabular}
% \caption{Examples of set of items}
% \label{tab:itemsset}
% \end{table*}


\begin{itemize}
    \item  Format: \emph{label}: (\emph{start token index}) [\emph{tokens}]
    \item 0: (0) [`The', `paper', `proposes', `a', `method', `to', `recognize', `time', `expressions', `from', `text.']
    \item 1: (11) [`It', `is', `a\textbackslash r\textbackslash nsimple', `rule-based', `method,', `which', `is', `a', `strong', `\textbf{baseline}', `for', `time', `expression', `recognition.']
    \item 0: (25) [`The', `authors', `analyze', `different', `datasets', `and', `discover', `that', `only', `a', `small', `set', `of', `words', `are', `consistently', `used', `to', `convey', `time', `information.']
    \item remaining sentences omitted for brevity
\end{itemize}

Interestingly, by using the given tokenization method on whitespace only, we preserve linebreak characters such as in the third token of the second sentence, and obtain empty string tokens in between multiple whitespaces. We choose to include both this `raw' text data as well as a normalized version in our system, since on the one hand, such typographic choices are indicative of authorship changes but on the other hand, we may not be able to rely on their presence in unseen data.
%Here or in other sections, cite related work that will help the reader to understand your contribution and what aspects of it are novel.



\section{System overview}
%Key algorithms and modeling decisions in your system; resources used beyond the provided training data; challenging aspects of the task and how your system addresses them. This may require multiple pages and several subsections, and should allow the reader to mostly reimplement your system’s algorithms.
%Use equations and pseudocode if they help convey your original design decisions, as well as explaining them in English. If you are using a widely popular model/algorithm like logistic regression, an LSTM, or stochastic gradient descent, a citation will suffice—you do not need to spell out all the mathematical details.
%Give an example if possible to describe concretely the stages of your algorithm.
%If you have multiple systems/configurations, delineate them clearly.
%This is likely to be the longest section of your paper.
In our system, we at first sought to compare and combine more traditional textual features with task-specific learned character embeddings, as the former offer the benefit of cheap computation and greater transparency, whereas the latter should allow the system to capture more subtle patterns at the expense of transparency and at a higher computational cost. We choose a relatively straightforward basic architecture for our models, using a BiLSTM over sentence vectors to predict the boundary sentence at which the authorship change occurs. With regards to textual features, we compute for each sentence: token count, mean token length, standard deviation of token length, counts for punctuation and space characters, various readability scores, word frequency class and word part-of-speech class counts. For our character model, we used another BiLSTM with embedding layer over the text characters, adjusting the token labels to the character level. As the character level model did not perform to our expectations on both normalized and raw text, we did not combine it with the textual feature model and decided to use the latter as a stand-alone model.



\section{Experimental setup}
%How data splits (train/dev/test) are used.
We used the provided train/dev split to tune our models. We compared performances of a purely textual feature based model and a character-based model. %Key details about preprocessing, hyperparameter tuning, etc. that a reader would need to know to replicate your experiments. If space is limited, some of the details can go in an Appendix.
We extracted the textual features offline, using the \emph{spacy} \cite{spacy2} library for Python and its \emph{textdescriptives} \cite{Hansen_2023} extension library. We used the \emph{en\_core\_web\_sm (v3.7.1)} pipeline for spacy. Hyperparameters were tuned manually, we settled on single-layer networks of hidden size 16, using Adam optimizer \cite{adam} with learning rate 1e-5, training on batches of size 8 over 100 epochs. For the character-based network we choose an embedding size of 8. %External tools/libraries used, preferably with version number and URL in a footnote. %Summarize the evaluation measures used in the task.
The task is evaluated on mean absolute error (MAE) between predicted and actual boundary word index. To translate our boundary sentence prediction into a token index, we selected the middle token index as default, rounding it down for sentences with an even token count. For the character model, we chose the token containing the predicted character index.\par
%You do not need to devote much—if any—space to discussing the organization of your code or file formats.



\section{Results}
%Main quantitative findings: How well did your system perform at the task according to official metrics? How does it rank in the competition?
On the development set our textual feature model showed somewhat promising results with regards to predicting the sentence containing the boundary word. It predicted the correct sentence in 69.9\% of cases, the adjacent sentence in a further 21.4\% with a sentence index MAE of 0.47. Translating these predictions into token  level predictions using the sentence mid-point yielded a token MAE of 13.5, notably worse than the baseline model's. Our character embedding model did perform notably worse on the development set, with a token MAE of 48.9. We therefore did not pursue it further and abandoned our initial idea of combining it with the textual feature model. On the test set, for the competition, we submitted the predictions of our textual feature model, unfortunately not matching the performance on the development set. We only managed to predict 30\% of boundary sentences correctly, with another 22.6\% predictions of the adjacent sentence, resulting in a sentence MAE of 3.2 and a disappointing token MAE of 59, ranking 26 out of 30 among the participant models in subtask C. %Quantitative analysis: Ablations or other comparisons of different design decisions to better understand what works best. Indicate which data split is used for the analyses (e.g. in table captions). If you modify your system subsequent to the official submission, clearly indicate which results are from the modified system. %Error analysis: Look at some of your system predictions to get a feel for the kinds of mistakes it makes. If appropriate to the task, consider including a confusion matrix or other analysis of error subtypes—you may need to manually tag a small sample for this.
We suspect this drop in performance mainly be caused by introduction of a new text domain in the test set, while development and training samples are exclusively drawn from PeerRead \cite{kang2018dataset} i.e. academic peer reviews, the test set introduces student essays from OUTFOX \cite{koike2023outfox}. This degradation of performance for stylistic textual features is in line with other findings, e.g. the comparisons performed on the M4 dataset \cite{wang2023m4} of which this competition's dataset is an extension. \par



\section{Conclusion}
% A few summary sentences about your system, results, and ideas for future work.
While our model's performance leaves plenty of room for improvement, we can envision the use of simple textual features in a lightweight model, similar to ours as a basic tool in contexts where more powerful models are either unavailable or too expensive to run and the text domain is known in advance. Focusing on sentence level instead of token level predictions allows us to reduce computational effort and we consider it sufficient for many practical applications, where automated detection of LLM generated text is only a first step, such as e.g. examining student essays, where we would expect a teacher to follow up with affected students regarding suspicious spans of text individually.

\bibliography{anthology,QL}
\bibliographystyle{acl_natbib}

\printbibliography

\end{document}
