\input{paper/inserts/03_diagram}

\section{Our Approach}
\label{sec:models}

As we want to analyze whether the argumentative structure of an article can be used to identify deception, we perform a two-step process inspired by \citet{alhindi-etal-2021-fact}.
This allows us to determine whether our model learns from argumentation in the text and provides us with information about the types of argumentation in news articles, which we analyze in Section \ref{sec:analysis}.
We use argumentation types instead of argumentation roles (such as premise and conclusion), as \citet{al-khatib-etal-2016-news} note that the latter encode the strategy an author uses to persuade readers.

We split the articles into an ordered set of sentences and assign them an argumentation type with BERT.
We use four argumentation types: \emph{anecdote}, \emph{assumption}, \emph{testimony}, and \emph{other}.
Table \ref{tab:arg_type} explains the different argumentation types, while Section \ref{sec:dataset_arg} details why these specific ones were chosen.
We use the final transformer layer from this model to generate sentence embeddings, which are fed to a Bi-LSTM model to classify news articles as credible or deceptive.
The architecture of this process is represented in Figure \ref{fig:model}.

\input{paper/inserts/03_arg_types}

While decoder-only models\footnote{Such as OpenAI's GPT line of models.} have been shown to work well with argumentation \cite{el-baff-etal-2024-improving}, they have give mixed results in disinformation detection tasks \citep{llm_fake_news,su-etal-2024-adapting}.
We do not use them for this study to avoid the introduction of artifacts in any part of our pipeline.
We use BERT over similar but larger models to avoid overfitting as our argumentation type dataset is quite small.
Exploratory experiments revealed that BERT struggles with the least represented argument type (see Section \ref{sec:dataset_arg}).
This issue is likely to be more prominent in larger models.

\citet{van_dijk_dimensions_1989} and \citet{yarlott-etal-2018-identifying} note that ordering is important for the argumentative role of text in written news media.
We chose a Bi-LSTM for the deceptive news detection task, as these models will intrinsically take into account the ordering of the argumentation types.

We explain each model and how they are implemented in more detail throughout the rest of this section. The specific hyper-parameters used for our experiments can be found in Appendix \ref{app:hyperparameters}.

\paragraph{Argumentation Type Classifier:}
We fine-tune a BERT model\footnote{\url{https://huggingface.co/google-bert/bert-base-uncased}}
on the argumentation type dataset.
This model is shown individual sentences and must assign an argumentation type to each of them.
We use the output of the \texttt{[CLS]} token from the final transformer layer for classification.
As the latter BERT layers typically learn task-specific features \cite{rogers-etal-2020-primer}, we expect the final layer to encode argumentation-related features for the whole sentence.
This model is then frozen for the rest of the experiments to prevent its weights from changing later on, thus making sure that it retains its knowledge about argumentation types intact.

\paragraph{Deceptive News Classifier:}
Given a news article, we split it into sentences.
These sentences are passed through the now-frozen argumentation type classifier.
We use the output of the final transformer layer corresponding to the \texttt{[CLS]} token as a sentence embedding.
These embeddings are then fed to a Bi-LSTM model to determine whether the article is credible or deceptive.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Datasets}
\label{sec:dataset}

In this section, we describe the different datasets used in our two tasks.
For the argumentation type classification task, we use the Webis-Editorials-16 dataset (Section \ref{sec:dataset_arg}).
For the deceptive news detection task, we use two datasets: one with article-level annotations (Section \ref{sec:dataset_fn}) and another with source-level annotations (Section \ref{sec:dataset_emi}).



All three datasets contain news articles in English collected prior to 2020.
Although the landscape of deceptive news and misinformation is likely to have changed since these articles were originally published, these datasets are still valuable as they only contain human-generated news.
Machine-generated mis- and disinformation is very different from that generated by humans \citep{Tewari_2021} and detecting it is another task in and of itself \citep{Beigi_2024}.
Therefore, we choose established datasets from before content produced by generative language models flooded the web.


\input{paper/inserts/03_datasets}

\subsection{Webis-16 Dataset}
\label{sec:dataset_arg}

The Webis-Editorials-16 dataset (or Webis-16 for short) was originally introduced by \citet{al-khatib-etal-2016-news}.
It consists of news editorials in English from 
three established news sources.
One hundred editorials were selected for each of the three publishers.
The included texts were originally published between December 2014 and January 2015 and were selected such that they would have a length of at least 250 word and had at least five comments.
This dataset does not to distinguish between true and false statements, which is beneficial for our task as it reduces the risk of introducing artifacts into the deceptive news classification task.

Each token in the text was assigned one of eight labels.
Six of these labels correspond to argumentation types, namely \textit{common ground}, \textit{assumption}, \textit{testimony}, \textit{statistics}, \textit{anecdote}, and \textit{other}.
The \textit{continuation} label means that a token has the same argumentation type as the next argumentation type label that appears, thus forming spans of argumentative units.
Some tokens, such as punctuation, are labelled as \textit{non-argumentative} as they do not form part of an argumentative unit, regardless of surrounding tokens.

It is important to note that argumentative units do not necessarily correspond to sentences.
A sentence may contain multiple clauses, each its own argumentative unit.
It is also possible for argumentative units to span two or more sentences.
This poses a problem for our task.
Although token-level classification is useful for studying argumentative units in the context of argumentation types, the difference in granularity can harm our downstream task as it is document-level classification.

Because of this, we cast the argumentation type labels so that each sentence has one and only one.
We do this in the following way:
(i) \textit{continuation} labels take on the same label as the next argumentation type label;
(ii) sentences with more than one argumentation type label are discarded; and
(iii) if all tokens in a sentence not labelled \textit{non-argumentative} share an argumentation type label, the whole sentence gets that label.

One issue that arose during exploratory analysis was that models performed very well on the majority class, but very poorly on under-represented classes.
This was still the case when applying early stopping and keeping the best performing checkpoint.
A model that severely under-performs on one or more of the classes will not allow for good analysis of the data.
To get around this issue, we collapsed some of the minority classes together.
The labels for \textit{assumption}, \textit{anecdote}, and \textit{testimony} were preserved, while \textit{common ground} and \textit{statistics} were grouped into \textit{other}.
This resulted in better performance of the deceptive news classifier and allowed us to conduct a more accurate analysis of argumentation in the articles.
Appendix \ref{app:labels} goes into more detail on how the number of labels was chosen.

\subsection{PolitiFact}
\label{sec:dataset_fn}

FakeNewsNet, originally introduced by \citet{shu_fakenewsnet_2020}, contains the PolitiFact and GossipCop datasets.
They have article-level annotations obtained from their name-sake fact-checking websites.\footnote{\url{https://www.politifact.com/} and \url{http://www.gossipcop.com/}, now defunct.}
The labels are binary and represent verifiable truth.
As our analysis focuses on political news, only the PolitiFact dataset is used in our approach.
It originally contained 948 articles accessible through links provided by the authors to preserve copyright.
Unfortunately, many articles are no longer retrievable due to broken links.

Article length has been shown to be a strong indicator of deceptive news \citep{levi-etal-2019-identifying}.
We filter the dataset to ensure both credible and deceptive articles are within an range of 100 to 800 tokens.
This helps make sure the model learns from argumentative structure rather than length.
Motivation for these bounds can be found in Appendix \ref{app:length}.
The final number of articles after filtering can be found in Table \ref{tab:article_count}.


\subsection{FakeNews-2018 Dataset}
\label{sec:dataset_emi}

The FakeNews-2018 dataset, originally introduced by \citet{francis_misinfowars_2018}, contains over 81,000 political news articles in English collected from various sources from the U.S., Canada, and the U.K. published between 2013 and 2017.
Articles are labelled as credible or deceptive based on the source, according to the factuality and credibility scores from Media Bias/Fact Check,\footnote{\url{https://mediabiasfactcheck.com/}} AllSides,\footnote{\url{https://www.allsides.com/}} and AdFontes Media\footnote{\url{https://adfontesmedia.com/}} to categorize sources as credible or deceptive. 

Some sources labelled as deceptive in the dataset are described as satire.
Even though satirical news differ from non-satirical news \cite{horne_this_2017}, research has shown it is challenging to distinguish satire from deceptive news \citep{horne_this_2017, rubin2015}.
While satirical news are meant to be entertainment, disinformation outlets often present themselves as satire to protect themselves from legal consequences \citep{goldbeck_2018}.
Even when this is not the case, satirical news has the potential to mislead readers through its mimicry of actual news \citep{francis_variation_2024}.


\section{Baselines}
\label{sec:baseline}

%\paragraph{Argumentation Type Classifier:}
We compare our argumentation type classifier against a random classifier and a majority class baseline.
This is done to ensure the model is actually learning from the data and not simply assigning labels arbitrarily.
We focus on both general performance and performance on the lowest scoring label.


%\paragraph{Deceptive News Classifier:}
We compare our approach to three models: a BERT classifier, an SVM using LIWC\footnote{Linguistic Inquiry Word Count, originally introduced by \citet{pennebaker_linguistic_1999}.} features, and FakeFlow \cite{ghanem-etal-2021-fakeflow}.
We choose BERT as it has been shown to perform well for a variety of tasks and is simple to implement. 
Classical machine learning models using LIWC features have been used successfully for deceptive news detection in the past \citep[e.g.][]{Che2018,perez-rosas-etal-2018-automatic}.
We follow the implementation of \citet{horne_this_2017}, using an SVM classifier and the same feature selection process.
The final model used for comparison is FakeFlow, which uses a CNN \cite{kim-2014-convolutional} to model article topics and a Bi-GRU \cite{cho2014learningphraserepresentationsusing} to model emotions in the text.