\documentclass{article}
%Image-related packages
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage[export]{adjustbox}
\usepackage{wrapfig}
\usepackage{natbib}
\usepackage{tabularx}
\bibliographystyle{unsrtnat}

\usepackage[english]{babel}
\usepackage{natbib}


% Language setting
% Replace `english' with e.g. `spanish' to change the document language
\usepackage[english]{babel}

% Set page size and margins
% Replace `letterpaper' with `a4paper' for UK/EU standard size
\usepackage[letterpaper,top=2cm,bottom=2cm,left=3cm,right=3cm,marginparwidth=1.75cm]{geometry}

% Useful packages
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage[colorlinks=true, allcolors=blue]{hyperref}

\title{Probing of pretrained multilingual models on the knowledge of discourse on the material of Universal Dependencies}

\begin{document}
\maketitle

\section{Introduction}

While pretrained language models have shown remarkable performance on various language tasks, there is still much to be explored in terms of their ability to capture broader discourse in documents. Probing tasks which are aimed at testing the understanding of various aspects of the language by neural networks can certainly offer means to analyze these capabilities, but it is important to design appropriate tasks that can capture the relevant aspects of discourse. 
Discourse probing can involve tasks such as identifying the relationships between sentences within a document, the role of one sentence in the document structure, investigating main topics, suitable ending and finding out whether one sentence belongs to the particular paragraph or not. Such tasks can help to shed light on the strengths and limitations of pretrained language models in capturing the nuances of discourse structure.
We frame almost all presented tasks as binary classification problems, but they involve different aspects of Rhetorical Structure Theory, models’ understanding of which is being tested in this study. The peculiarity of our work is that we test multilingual, rather than basic, models on the corresponding data for ten languages, exploring only modern pre-trained transformer models.
All data for our probing tasks was taken from the UD framework.

\section{Related work}

We frame almost all presented tasks as binary classification problems, but they involve different aspects of Rhetorical Structure Theory, models’ understanding of which is being tested in this study. The peculiarity of our work is that we test multilingual, rather than basic, models on the corresponding data for ten languages, exploring only modern pretrained transformer models.
In (\citealp{chen}) discourse probing is conducted via DiscoEval. The main difference between this research and our work is that we do not concatenate vectors for separate sentences, but use the sequence as an input for our models. From the described article we borrowed and adapted the following tasks making them suitable for multiple languages:

\textbf {Sentence Position (SP)} This task tests the model's understanding of linearly-structured discourse. By randomly moving one of the 5 sentences to the first position, the model is asked to accurately predict the correct order within the discourse sequence based on the content of the sentences. 

\textbf{Binary sentence ordering} This task is to identify the correct order between the two contextually codependent sentences. BSO could be useful in testing a model's ability to capture local discourse coherence and understand the relationships between adjacent sentences in a text. 

\textbf {Discourse coherence} Having a sequence of six sentences that form a coherent paragraph, we need to randomly replace one sentence from the coherent sequence with a sentence from another discourse. A model needs to determine whether the resulting sequence of six sentences still forms a coherent document. 

We base our methodology on (\citealp{koto}). The peculiarity of this probing study is that the authors tried to trace differences of understanding of the discourse by models depending on the layer.  As for the tasks that we borrowed from this article, these are:

\textbf {Next sentence prediction } The preceding context consists of 2 to 8 sentences, while the candidates (4 sentences) are always single sentences. Nevertheless, we adopted it as a binary classification task with mixing one of the sentences in a way that researchers in (\citealp{chen}) did. 

\textbf {Sentence ordering} This task is to determine whether the order of sentences in the document is correct. Texts in which from 3 to 7 sentences are mixed within the same sequence are presented as incorrect options. 

\textbf {Cloze story test} Data for this task consists of sequences with 4 sentences in each. Model needs to pick the best ending sentence for all documents. We made our adapted task binary, so for incorrect pairs ‘key:value’ we shuffle ending sentences within all documents. 

\section{Methods}
\subsection{Models}

\textbf {BERT multilingual base model (cased)}. The MLM objective allows the model to learn contextual representations of words by predicting randomly masked tokens within the input text. One of the two BERT objectives is next sentence prediction.

\textbf {XLM-RoBERTa (base-sized model)} Unlike BERT, it was pretrained only with the Masked language modeling (MLM) objective. This version of RoBERTa will precisely perform better on syntax tasks, than on discourse ones. 

\textbf {Multilingual GPT model} The main feature of this neural network is that it uses sparse attention mechanism and, as a result, achieves greater scalability and performance across multiple domains than transformers with multi-head attention. The main difference between models like BERT, RoBERTa and GPT is that the first ones need a larger sequence for masking, that is, generating text, than the GPT.

\textbf {mT5 model} This model’s architecture takes parameters from encoder-decoder transformers, but replaces encoder-decoder attention with a full attention algorithm. 

\subsection{Languages}
Most of the languages in the sample (Table \ref{table:1}) belong to one language family – Indo-European. At the same time, it was found out that the sample (treebank) of the Turkish language is one of the largest in the database. In addition, Turkish is part of one of the largest language families, Altai, so we also included it in the training data. Among the groups of the Indo-European family, some of the most common were chosen: Romance, Germanic and Slavic. The sample also included the Armenian language, since data for this language was huge enough to parse it and no probing studies with its participation had been conducted before. The remaining languages were selected according to the following principle: they either often appeared in NLP works (English, French, Russian) or, conversely, were not or little studied using machine learning – this is the majority of the languages in the sample. This decision was made in order to compare the performance of one model with respect to different languages in order to understand how much the number of examples in a particular language (such as English, the examples in which the vast majority of models in the training sample) determines the understanding of several idioms by a multilingual transformer at once.
\begin{table}[h!]
\centering
\begin{tabular}{||c| c c c c c c c ||} 
 \hline
Language & Russian & Bulgarian & Czech & Serbian & Catalan & French & Latin \\ [0.5ex] 
 \hline\hline
 Binary sentence ordering & 15632 & 17354 & 1230 & 1389 & 1476 & 1468 & 1474\\ 
 \hline
 Cloze story test & 9385 & 67142 & 18437 & 6780 & 47852 & 1201 & 51867\\
 \hline
 Discourse coherence & 3450 & 33567 & 2143 & 2013 & 34701 & 1750 & 21602\\
 \hline
 Next sentence prediction & 12949 & 42781 & 13561 & 4998 & 21952 & 7620 & 13764\\
 \hline
 Sentence ordering & 5302 & 18579 & 9450 & 4356 & 1938 & 2395 & 1027\\
 \hline
 Sentence position & 2790 & 22152 & 7664 & 1732 & 9909 & 1042 & 1395\\
 \hline
 Discourse connective prediction & 14036 & 37620 & 2089 & 1503 & 7605 & 1201 & 3047\\ [1ex] 
 \hline
\end{tabular}
\caption{Number of examples for each language and task}
\label{table:1}
\end{table}

\begin{center}
\begin{tabular}{||c| c c c||} 
 \hline
Language & English & Armenian & Turkish\\ [0.5ex] 
 \hline\hline
 Binary sentence ordering & 1823 & 2094 & 15203\\ 
 \hline
 Cloze story test & 21770 & 46209 & 12064\\
 \hline
 Discourse coherence & 3502 & 29436 & 3972 \\
 \hline
 Next sentence prediction & 16067 & 49673 & 30166 \\
 \hline
 Sentence ordering & 3750 & 19820 & 1960 \\
 \hline
 Sentence position & 7438 & 10347 & 1704 \\
 \hline
 Discourse connective prediction & 8993 & 28049 & 6775\\ [1ex] 
 \hline
 
\end{tabular}

\end{center}

\subsection{Probing methods}

In our study we used all models without performing finetuning. Thus, all models were pre-trained – this is the basic probing approach. This was done because we aim to test the basic models in understanding the discourse. The diagnostic classifier logistic regression was used to assess the quality of the models' performance.
Also we use the [CLS] embedding to get the predictions from transformers following standard practice.

\section{Results} Transformers show the best performance on Cloze Story Test (0.842) and Next sentence prediction (0.849) tasks. These tasks are united by the fact that in both the focus of the prediction is the last sentence of the document.  Due to the fact that an average accuracy for the Discourse coherence task is much lower (0.57), we cannot conclude that good models’ results for these tests correlate with the number of sentences in the input. It can be attributed to the fact that for the DC test there was a sequence of 5 sentences provided, while for CST all documents consisted of 4 sentences.  
\begin{wrapfigure}{l}{0.5\textwidth}
\includegraphics[width=1\linewidth]{Рисунок1.jpg} 
\caption{Averaged accuracy depending on the type of task}
\label{fig:wrapfig}
\end{wrapfigure} 
At the same time, DC task asked the models to determine the coherence of a document in which any of the five sentences could serve as a replaced sentence, except for the first. Meanwhile, in tasks in which the highest accuracy of the models' performance was recorded, the focus sentence is fixed (always the last, only the size of the sequence varied).
Nevertheless, context definitely affects the model’s performance on the task. For example, models perform worse on a task in which it is required to determine the correctness of the order of sentences within a binary sequence (0.61) than on a task containing multiple sequences (0.77). 
Quite unexpected and contrary to hypotheses results were obtained for the task Sentence Position. In the original paper, the BERT-Large accuracy for SP was 0.538, while in our case we got an 0.8 accuracy. Such a difference in the results may indicate the importance of the first position in the sequence, the weight of which in the context of the multi-head attention method is the largest.
The worst average accuracy is for Discourse connective prediction (0.175). Our results for this task are much worse than in the source research. That can be explained by the fact that an average result is affected by too huge and too small values. To sum up, the difference between results that we got and the ones received in the (\citealp{nie}) can be explained by the fact that our data for particular languages is much smaller, therefore, models obtain less connective markers and, as a result, perform worse. 

To find out whether certain languages distort the test results, we will compare the overall average accuracy obtained for all tasks for each language:
\begin{wrapfigure}{R}{0.47\textwidth}
\includegraphics[width=1\linewidth]{Рисунок2.png} 
\caption{Average accuracy for each language}
\label{fig:wrapfig}
\end{wrapfigure}
It can be seen that languages do not vary drastically in their average results. Contrary to expectations, the best results were obtained not for English, but for Latin and Czech, supposedly, due to the fact of bigger data for particular tasks.

To better assess the correlation between the structure of a particular language and the performance of transformers, consider the accuracy averaged over various tasks for each model and each idiom. 
\begin{wrapfigure}{R}{0.83\textwidth}
\includegraphics[width=1\linewidth]{Снимок экрана 2023-08-21 langs.png} 
\caption{Average accuracy depending on the language and type of model}
\label{fig:wrapfig}
\end{wrapfigure}

\textbf {Armenian} BERT and T5 use a bidirectional Transformer, while mGPT uses a left-to-right Transformer (see Figure 3.1). This difference in architecture provides the advantage of the first two models over mGPT, since they can directly access all positions in the sequence simultaneously (parallel receipt of vectors in the encoder), which, as has been proven, provides better accuracy in low-resource languages. 

\textbf {Bulgarian} In this case (see Figure 3.2), there is a distribution common to most tasks (and obtained by averaging the results for both tasks and languages), in which RoBERTa demonstrates the highest accuracy, BERT performs slightly worse, followed by T5, and the worst results are observed for mGPT.

\textbf {English} Results (see Figure 3.5) demonstrated by models for English may show the real distribution of ratings, due to the fact that this language always has the largest number of examples in the training sample. We can assume that BERT potentially has more knowledge about discourse, but either it is more difficult for it to cope with longer sequences, or it has a smaller multilingual base.

\textbf {Catalan} For Catalan we observe extremely unexpected results (see Figure 3.3), as BERT demonstrates the best accuracy (while still lower than the average value for other languages), and mGPT is in second place. A slightly lower average accuracy was demonstrated by T5, and RoBERTa performs the worst. 

\textbf {Czech} RoBERTa’s absolute superiority (see Figure 3.4) may stem from the fact that the compilers of the treebank for the Czech language emphasized long-distance discourse relations in accordance with (\citealp{polakova}). As it has been proven, one of the main advantages of RoBERTa is the ability to analyze large sequences of text (\citealp{conneaucross}).

\textbf {French} The utterance in Romance languages (in comparison with the linear structure of utterance in English) is distinguished by ornateness, the main idea is usually expressed at the beginning and at the end. In this vein, the accuracy of mGPT (see Figure 3.10) can be explained by the sparse attention mechanism which allows each output position to focus on only a subset of input positions, selected based on predefined patterns or rules (\citealp{martins}). 

\textbf {Russian}
For Russian (see Figure 3.7) we observe exactly the same distribution that has already been described for Bulgarian. Since the distribution was almost the same for the Czech language (the difference is that the mGPT showed slightly higher accuracy than T5), it can be assumed that such similarity in the results is explained by the affiliation of the above languages to the same language group. 

\textbf {Latin} In (\citealp{kroon}) it is established that the structure of discourse in Latin is characterized by strong fragmentation in the sense of the distance between discursive units united by various word forms, which are also polysemic. Thus, the high average accuracy of most models (see Figure 3.6) in tasks with the Latin language reflects the ability to build non-trivial connections within the text and understand the general meaning.

\textbf {Serbian} Since Serbian discourse has not been sufficiently studied before, the only factor by which we can explain such a distribution (see Figure 3.8) of model performances is the small amount of data for the language under study. Regarding BERT's superiority over RoBERTa, it can be assumed that it is explained by differences in the token masking procedure - in the case of BERT, it is always a fixed set of tokens when the model is working, which may help in working with low-resource languages.

\textbf {Turkish} RoBERTa achieved the highest performance (see Figure 3.9), surpassing BERT, T5, and mGPT. However, BERT still performed better than mGPT and T5, T5 showed the lowest accuracy among the four models Distinctive features in this case are shared arguments  and properly contained arguments which can both contribute to the complexity of discourse structure (\citealp{turk}). 

Now we are going to examine the correlation between each model's understanding of discourse and different types of tasks.
\begin{wrapfigure}{L}{0.8\textwidth}
\includegraphics[width=1\linewidth]{Снимок экрана 2023-08-21 162239.png} 
\caption{Average accuracy depending on the task and type model}
\label{fig:wrapfig}
\end{wrapfigure}

\textbf {Binary sentence ordering} The uniqueness of this task lies in the fact that mGPT copes with it best (see Figure 6.1), which in all other tasks demonstrates the lowest accuracy rates due to some obvious issues like the lack of some investigated languages in the mGPT’s training data (still this issue does not cause too much problems as multilingual transformers tend to adjust to the new idioms using available data).

\textbf {Cloze story test} In this task (see Figure 6.2) RoBERTa shows the best performance. As mentioned above, we have used some low-resource languages in our research, and it was proved in (\citealp{conneaucross}) that RoBERTa surpasses BERT on cross-lingual classification, but specifically with low-resource languages being used for training data. RoBERTa's superiority over BERT can be explained by the phenomenon called the "generalization gap”, which occurs when a language model's ability to perform well on downstream tasks exceeds its performance on the validation set during training. 

\textbf {Discourse coherence} Even though one of the two main BERT’s objectives is Next sentence prediction, we should keep in mind that the DC task (see Figure 6.3) provides the model with not two, but several sentences as input to determine whether they are coherent. And in this case, the best results obtained by RoBERTa are explained by the fact that this model copes better with long sequences, due to the fact that compared to BERT more extensive training data with lengthier sequence segments is trained.
Results for this task indicate that the type of architecture of the model does not play a crucial role in this case. Although BERT and RoBERTa are encoders, mGPT is a decoder and T5 is an encoder-decoder transformer, we can see that T5 and mGPT-2 have shown almost the same results, which are relatively close to BERT’s accuracy.

\textbf {Next sentence prediction} 
NSP (see Figure 6.5) is a task of the type for which we expect high accuracy of predictions from a model whose main specificity is text generation (mGPT). Hypothetically, in this case, bidirectional self-attention is not required and it is enough to predict the output based only on the previous context. To understand why mGPT still performs the worst, and T5 shows the same results as RoBERTa (thereby neutralizing the importance of having a decoder in the architecture), we need to take into account the differences between generating the next sentence and a single token. Presumably, for the accurate recognition of the next sentence, the context of both the previous and the subsequent sentences plays a decisive role, the full understanding of which is impossible without the encoder (due to the mechanism of bidirectional attention). 

\textbf {Sentence ordering} Again, an unexpected result with BERT performing better than RoBERTa (see Figure 6.7), which may have been caused by differences in the masking procedures for RoBERTa and BERT. In RoBERTa, the masking of 0.15 of tokens is dynamic and changes for each pre-training epoch. Besides, in (\citealp{rothe}) it was demonstrated that BERT performs best with sequence-splitting tasks, indicating that its understanding of sentence ordering exceeds RoBERTa’s one. 

\textbf {Sentence position} In this case (see Figure 6.6), Roberta demonstrates the best results. This task is somewhat similar to the previous one, the difference is that not all proposals are mixed in the SO, but only 4 and another randomly selected, while in the SP all proposals for incorrect options occupy new randomly selected positions. Presumably, in this case, RoBERTa's superiority is explained by the fact that RoBERTa was trained on a much larger corpus of text data than BERT, which allowed it to learn more complex and nuanced patterns in language. Additionally, RoBERTa was trained for longer than BERT, with a larger batch size and more training steps.

\textbf {Discourse connective prediction} For this task where the input consists of two sentences and transformers must predict correct connective RoBERTa unsurprisingly demonstrates the best results (see Figure 6.4). This result can be attributed to the NSP loss being removed in RoBERTa’s architecture and whole input being replaced with full sentences. An obvious problem with mGPT and T5 in solving these kinds of tasks is their generative objective since the sample used for fine-tuning may lack the necessary connectives, in which case the correct answer simply cannot be generated by the model by definition and will eventually be read as incorrect.
\section{Discussion}
\textbf {The influence of the discourse structure in English} The so-called ‘complicated simple sentences’ (\citealp{dagnev}) in Bulgarian generate the heavy complementation, and that is the main difference between Bulgarian (see Figure 3.2) and English rhetorical structure. It can be that the model borrows discourse patterns from the language that prevails in the training sample. Thus, presumably, the fewer languages in the model and the greater the presence of English, the greater the accuracy in those languages whose discursive patterns are similar to patterns in English. The results obtained for Catalan display the same trend (see Figure 3.3) and can be explained by namely right-branching (right-dislocation constructions) which is not often found in English.

\textbf {mGPT's sparse attention mechanism} Due to GPT's performance for French and Russian we can hardly consider that the sparse attention mechanism applied for mGPT helps to cope best with long sequences found in Russian, rather it turns out to be the best in the case when the main topic of the utterance is concentrated at the beginning and end of the text (as in French). At the same time, for Russian Kaplan establishes a structure characterized by situationality, instability of discourse patterns and a constant change of focus of text, which, although in some sense similar to the ornate rhetorical structure in French (both are nonlinear with respect to discourse in English), differs in the lack of integrity according to Kaplan. It can be assumed that this difference is the reason for the strong decrease in the accuracy of the mGPT for Russian compared to French.

\textbf {Models performing similarly with languages belonging to the same group}
The hypothesis that the models act equally (in relation to each other) for languages belonging to the same language group and therefore having common discourse patterns is confirmed by the example of French and Latin. At the same time, this is still a hypothesis, since such a distribution seems to be universal in most cases and has also been recorded for most languages of the Slavic group. This assumption is contradicted by the distribution of model accuracy obtained for Serbian, but in this case it seems appropriate to refer to the lack of resources of this language.

\textbf {Advantages of dynamic masking procedure} In case of Turkish we were talking about shared arguments occur when two distinct discourse connectives use the same text span as their argument. This can create ambiguity or confusion for the reader or listener, as it may not be immediately clear which connective is governing the argument. Properly contained arguments occur when a larger text span that is the argument of one connective contains a smaller text span that is the argument of another connective. For RoBERTa the complexity of text may be potentially overcome via dynamic masking, as in this case the number of potentially different masked versions of each sentence is not bounded like in BERT, therefore the probability of understanding complicated structures gets bigger. At the same time we can see that dynamic masking procedure benefits only in cases where the complicated structure of the text does not change drastically. For instance, in SO task this change could lead to a deterioration in the quality of the model's performance in this case, since the SO task assumes that for incorrect examples all sentences in a sequence are being shuffled. Accordingly, in this case, masking the fixed part of the input can serve as an advantage of BERT. 

\textbf {mT5's superiority over mGPT} In the NSP task we can assume that the results obtained can be explained by the fact that in mT5 the decoder typically produces two additional tokens: the class label and an end-of-sequence token, which can contribute to a better understanding of the connectivity of the final element of the sequence and the previous elements. This hypothesis can be applied to all results in which T5 exceeds mGPT in accuracy.

\section{Conclusion} Our work is devoted to the study of the degree of discourse assimilation by various multilingual models. Despite the fact that many tasks and hypotheses were built on the materials of their predecessors, our research differs from them in that it involves several languages in discourse probing at once and combines completely different tasks that ultimately somehow test the understanding of the model of the whole text. Also, some of our results do not correspond to the conclusions of other researchers and add new information about the understanding of the language by individual models. 
Moreover, we have come to a conclusion that models, on average, perform equally in low-resource and conventional (popular) languages with binary-classification tasks. This result may indicate the presence of certain trends associated with the assimilation of the document structure by models, which apply to all idiolects.
We also identified some characteristics of tasks and training samples that affect the performance of the model, such as the size of the sequence, the number of sentences involved in shuffle, the focus of prediction (the last sentence is often easier to predict than the first) – and this factor is stronger than the significance of the size of the context. The more randomness there is in choosing proposals that will change the position in the document, the better the performance of some models, for example, RoBERTa, since its main principle is masking an unfixed set of tokens. 
Consequently, we have identified certain aspects of tasks that models generally do worse with, such as predicting the connective marker when there is a limited amount of resources, as well as those factors of individual model’s architecture that worsen the results. We also compared the results obtained with the accuracy of the predictions of monolingual models and did not reveal a significant deterioration in the quality of transformers.

\medskip

\bibliography{sample} 


\appendix

\newpage
\onecolumn
\section{Tasks' generation, Examples}
\label{sec:appendixA}
\label{sec:appendixA1}
The ideas for all the tasks were taken from the articles of the predecessors. At the same time, all the previous probing studies, the tasks from which we borrowed, were mainly conducted on the basis of English and they did not use multilingual models. Therefore, we needed to adapt all the borrowed tasks in such a way that they correspond to the treebanks of any language from the database. This is one of the reasons why in our study we did not test the models’ understanding of segmentation into clauses: in each language the division into clauses occurs differently, therefore, we are not allowed to implement a universal code for extracting EDU. 
All tasks, except for the discourse connective prediction, are a binary classification problem. This approach was chosen to better evaluate the accuracy of the models. Taking into account that many of the languages in the sample are not very large and have not been studied sufficiently, their datasets are also small. As a result, if, for instance, in the case of a task for the order of sentences in a sequence, integer answers with an order were submitted to the input, not all numeric sequences would occur in the training sample. Therefore, given that all analyzed models have masking objects, correct and incorrect sequences should be generated by the models themselves. Thus, the correct sequences are marked as 1, the incorrect ones as 0.
\subsection{Discourse connective prediction}
Unlike previous approaches(\citealp{koto}), we did not set a frequency threshold for accounting the connective due to the limited shapes of the data for some languages. Following the approach presented in (\citealp{malmi}), we predict only connectives which occur in the beginning of the sentence, considering this as a base position for an explicit binding marker. This choice is explained by the fact that before testing the understanding of implicit connectives by a multilingual model, we must first pay attention to explicit ones.
\begin{table}[h!]
\centering
\begin{tabular}{ | m{8em} | m{8cm} | m{4cm} | }
\hline
Sent_1 & Sent_2 & Discourse Connective\\\hline
Obviously because I want to vote & If anyone else has voted, what did you guys vote for? & And \\
\hline
\end{tabular}
\caption{Example of a discourse connective prediction task}
\label{table:3}
\end{table}
\subsection{Sentence position}
The position of a sentence within the text can provide context and help to understand the overall structure and purpose of the document. The opening sentences often provide an introduction to the topic, while the following sentences provide more detailed information and support the main idea. (\citealp{chen}) discovered that in the SP task removing the surrounding sentences can make it more challenging to accurately predict the position of the target sentence, as the model has less information to work with. Due to the fact that the context plays a crucial role in a sentence position, we decided to take 5-sentence sequences for our dataset and swap the fourth of them with the other randomly chosen sentence in a sequence. This method was partly proposed by (\citealp{common}), and, although in the described article researchers swap the forth sentence with the first one, we decided not to swap fixed elements of a text, and choose one of them randomly, so we complicated the task, because usually models demonstrate high results in this test. 

\begin{table}[h!]
\centering
\begin{tabular}{ | m{40em} | m{1cm} | }
\hline
Examples & Labels\\\hline
The problem is that customers can find features between low-end camera companies. It's tough to make money branching out when your appeal is in your focus. If they continue to add features so they can justify their likely sky-high valuation. & 1 \\
\hline
The Greater New Orleans Fair Housing Action Center (GNOFHAC) filed a housing discrimination ast week. The complaint, filed with the United States Department of Housing and Urban Development. Thomas Housing Development residents, the City of New Orleans. VI redevelopment of St & 0\\
\hline
\end{tabular}
\caption{Example of a sentence position task}
\label{table:3}
\end{table}

\subsection{Binary sentence ordering}
This task differs from SP in that a much smaller amount of context is supplied to the input, so this test allows us to evaluate the ability of the model to determine the relationship between the minimum context of two sentences. 
\begin{table}[h!]
\centering
\begin{tabular}{ | m{33em} | m{1cm} | }
\hline
Examples & Labels\\\hline
Based on specific intelligence inputs, Army arrested Ghulam Mohiuddin Lone, a LeT man, from Doda district. During the preliminary interrogation, Lone 'confessed' his involvement in the blasts and gave several vital clues & 1 \\
\hline
Salon is clean and girls are nice. I didn't know what I was missing & 0\\
\hline
\end{tabular}
\caption{Example of a binary sentence ordering task}
\label{table:3}
\end{table}
\subsection{Discourse coherence}
In order to evaluate the ability of a model to capture local discourse coherence, it would need to be able to capture characteristics of the entity being discussed or the topic of the sentence group, and perform inference across multiple sentences to determine the coherence of the discourse. This can be a non-trivial task, as it requires the model to have a deep understanding of the underlying meaning and context of the text being analyzed. Connectivity within the document, in accordance with our research and the previous work, is determined from 6 sentences. In our case, this number is fixed. Negative examples are created by replacing one of the sentences with a sentence from another text.
\begin{table}[h!]
\centering
\begin{tabular}{ | m{40em} | m{1cm} | }
\hline
Examples & Labels\\\hline
This idea may seem strange if they are familiar with the King James Version's translation: "In the beginning, God created the heaven and the earth." However, as we have seen, this translation is not correct. Even so, there might seem to be room for the idea of creation made from nothing. It might appear to readers that this idea of creation from nothing is expressed or symbolized in Genesis 1:2 by the mention of "void and vacuum". These two nouns, connected by a conjunction and forming a fixed, com pound phrase, would seem to describe precisely the kind of nothingness that facilitates the concept of creatioт ex nihilo. & 1 \\
\hline
Genesis 1 envisions creation not simply as God making; it is as much as a process of "separation" and differentiation of elements from one another, as we will see in chapter 3. It involves a transformation from an unformed, wate1y mass into the world that sustains human existence with water.Creation is a process in which a deity makes the world, as it came to be. Psalm 33:6-7 nicely expresses this transformation. Let's consider this more closely. & 0\\
\hline
\end{tabular}
\caption{Example of a discourse coherence task}
\label{table:3}
\end{table}
\subsection{Next sentence prediction}
In the source paper there were 3 negative candidates and a single positive one for the next sentence, but we adopted it as a binary classification problem, therefore, for negative examples of sequences we shuffle the last sentence with the other sentence, but not within one document to sustain the text structure.
\begin{table}[h!]
\centering
\begin{tabular}{ | m{33em} | m{1cm} | }
\hline
Examples & Labels\\\hline
It was ok, but the place was old. It was clean, but just a little dumpy. Hard to get into though. & 1 \\
\hline
Horrible customer service. I came in to get a nice gift for my wife. But thankfully there are other flowers shops around & 0\\
\hline
\end{tabular}
\caption{Example of a Next sentence prediction task}
\label{table:3}
\end{table}
\subsection{Sentence ordering}
Originally this task was done by shuffling from 3 to 7 sentences, providing the model with the correct ordering and then predicting it. We reworked it by shuffling all the sentences for the incorrect sequences. This method allows the model to select the most consistent sequences in the dataset and further develop a coherency metric based on NLP analytics (\citealp{barzilay}).
\begin{table}[h!]
\centering
\begin{tabular}{ | m{33em} | m{1cm} | }
\hline
Examples & Labels\\\hline
This is unlike the situation last year in Asia when we evacuated US citizens from areas that were hit by the tsunami - a phenomenon that is much less predictable than the Hezbollah-provoked destruction that rained down on Lebanon. The American-Arab Discrimination Committee is suing Condoleeza Rice and Donald Rumsfeld, charging that they mismanaged the evacuation efforts & 1 \\
\hline
My favorite so far in Bellevue. They have good sushi for a good price & 0\\
\hline
\end{tabular}
\caption{Example of a cloze story test task}
\label{table:3}
\end{table}
\subsection{Cloze story test}
As was described earlier, in this task the model receives a document containing 4 sentences as input and chooses the best completion for the text. We changed this task by making the answers binary and shuffling the last sentences in the sequence for negative samples. We also did not take into account text biases conducting stylistic feature analysis (\citealp{sharma}) as it is harder to trace on a large language data. In (\citealp{common}) it is claimed that cloze story test indeed helps to identify the model’s understanding of the text coherence. If a model performs well on this task, it suggests that it has some level of understanding of the story's narrative structure and can generate coherent and logical endings based on that understanding.
\begin{table}[h!]
\centering
\begin{tabular}{ | m{33em} | m{1cm} | }
\hline
Examples & Labels\\\hline
Heh, yep, I like to wear silk chemises. Also panties even stockings with garter belt .Later on I red somewhere that it's seakness & 1 \\
\hline
You've already asked this . Why would someone post the location of a dealer in a public place? Drop by my house, I can get you some real cheap. Give me an address or something please idk & 0\\
\hline
\end{tabular}
\caption{Example of a cloze story test task}
\label{table:3}
\end{table}
\end{document}


