For subtask A, our intuition was that surface-level and stylistic features would be more effective than semantics in discriminating between human and machine-generated text. To build on this idea, we developed three approaches.

The first approach involved training a character-level LSTM. We expect the stylistic features of the texts to be good indicators of the generator, and working at the character level is known to capture this information well. For example, character n-gram models have been used successfully in the field of authorship attribution, which relies heavily on style \citep{stamatatos2013robustness}.

First, input texts are tokenized at the character-level, all tokens are mapped to their lowercase variants, and lastly numerals and punctuation are mapped to a \verb|<NUM>| and a \verb|<PUNCT>| special token respectively. White-space elements (space, tab, newline) are also mapped to a special token \verb|<WS>|. At this point, the tokenized and transformed inputs are fed through an LSTM, and the representation of the last token is used for prediction. 

The second approach is constructed along the lines of the first in terms of technical setup, but deals with words rather than characters. Large transformer-based solutions benefit from vast amounts of pre-training, but at a heavy computational cost -- using pretrained embeddings as model inputs appeared to be a good compromise between heavy models and training from scratch, as was the case with the character-level LSTM. We used the pretrained Word2vec embeddings from the Wikipedia2Vec \citep{yamada2020wikipedia2vec} project to map texts to vectors, but maintained the other steps, such as mapping numerals, punctuation and white-space to special tokens. 

The third approach is not recurrent in its nature~-- instead, we used the TextDescriptives pipeline \citep{Hansen_2023} through Spacy \citep{honnibal2020spacy} to obtain 66 lingustically motivated features to globally represent the text. Such linguistic features have a long tradition in NLP, for example in the field of readability analysis (for example \citealp{vajjala-meurers-2012-improving}), and have been observed to be valid and cheap-to-compute representations in a variety of settings. Since they are most well known for capturing the style of a text, rather than semantics, they appear to be very well suited for the present task. Additionally, we computed the mean perplexity of the document using GPT-2 \citep{Radford2019} and added it to the feature vectors. This follows the idea that the perplexity of a document assigned by a LLM should be higher for human written texts than for machine generated texts \citep{Chaka2023}. Our third approach computes this global feature vector for the input text, then generates a prediction through a simple MLP. The model consists of 3 linear layers with Tanh activation functions in between and was trained for 2000 epochs with a learning rate of 0.0003. 

Lastly, we formulated a joint model which takes as input the final representations (the last hidden states) of each of the three previous approaches, thus generating a single prediction.

\begin{table}[h!]
    \centering
    \begin{tabular}{llll}
        \hline
        \textbf{Model} & \textbf{Type} & \textbf{L*} & \textbf{H**} \\
        \hline
        Character-level & LSTM & 2 & 512 \\
        Word2vec & LSTM & 2 & 512 \\
        Language features & MLP & 3 & 256 \\
        Joint model & FFN & - & 512 \\
    \end{tabular}
    \caption{Summary of models for subtask A. \textbf{*} Number of layers. \textbf{**} Hidden size.}
    \label{tab:a_models}
\end{table}