\begin{table}[ht]
    \centering
    \begin{tabular}{llll}
        \hline
        \textbf{Model} & \textbf{Dev} & \textbf{Test} & \textbf{Ranking}\\
        \hline
        Baseline & 0.72 & 0.88 & 20\\
        \hline
        Character-level & 0.85 & 0.55 & 127\\
        Word2vec* & 0.82 & 0.72 & 85\\
        Language features* & 0.63 & 0.88 & 21\\
        Joint model* & 0.83 & 0.69 & 96\\
    \end{tabular}
    \caption{Results for SemEval-2024 Task 8, subtask A. Dev and Test columns report the accuracy on the respective data partitions. The ranking column refers to the model ranking in the shared task competition. The scores and ranking of the unofficial submissions were not provided by the organisers and computed by us. There was a total of 137 submissions.\\ \textbf{*} unofficial submissions}
    \label{tab:a_results}
\end{table}

Table \ref{tab:a_results} shows the results for each model on subtask A. On the development set, almost all models outperform the transformer baseline provided by the organisers. The best performing model was the character-level model, with an accuracy of 0.85 -- this was our final submission for the shared task.

While the two recurrent models and the joint model do not differ very much from one another, the FFN built on linguistically motivated global feature vectors sets itself apart in that it is the worst performing model on the development set. 

The character-level model only achieves an accuracy of 0.55 on the test set, while the Word2vec and joint models achieve 0.72 and 0.69 respectively -- all falling short of the baseline. Surprisingly, the language feature model is head and shoulders above the rest when it comes to the test set, with an accuracy of 0.88 that matches the baseline. Our assumption is that this is due to a conceptual difference between development and test set. A possible reason could be that the domain for human-written texts in the test set were student essays only. The development set on the other hand consisted of human-written texts from multiple domains. The linguistic features prevalent in the student essays seem to be more distinctive for classifying the documents compared to the texts from multiple domains.