% Reviews/Questions
%The results with automatic evaluation do not seem  to me to suggest major improvements, and the difference between automatic and human evaluation is not properly explained. 

% - By reading the paper, it is not clear to me why this work has to focus on "complex" question generation
% - Although section 3 gives a detailed description of the datasets used, it is unclear about the data augmentation methods used in this work or any novelty in that aspect.
% - Any explanation about why PCSQ method did not produce good results compared to the baseline method?

% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.

\documentclass[11pt]{article}

% Remove the "review" option to generate the final version.
\usepackage[]{ACL2023}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}

% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out.
% However, it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}


\usepackage{enumitem}
\usepackage{graphicx}
\usepackage{tabularx}


\usepackage{caption}
\newcolumntype{C}[1]{>{\centering\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}
\newcolumntype{L}[1]{>{\raggedright\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}
\newcolumntype{R}[1]{>{\raggedleft\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}

% \usepackage[table]{xcolor}

\usepackage{hyperref}
 % \definecolor{darkblue}{rgb}{0, 0, 0.5}
 %  \hypersetup{colorlinks=true, citecolor=darkblue, linkcolor=darkblue, urlcolor=darkblue}

\usepackage{xstring}
\usepackage{color}
\newcommand{\secref}[1]{\StrSubstitute{\getrefnumber{#1}}{.}{ }}

\usepackage{listings}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{colortbl}
\usepackage{natbib}
\usepackage{multibib}

\usepackage{lingmacros}

% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\title{Complex question generation using discourse-based data augmentation}

% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
%         Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
%         Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \And  ... \And
%         Author n \\ Address line \\ ... \\ Address line}
% To start a seperate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \AND
%         Author 2 \\ Address line \\ ... \\ Address line \And
%         Author 3 \\ Address line \\ ... \\ Address line}

% \author{$^1$Kushnur Binte Jahangir  \and $^{1,3}$Philippe Muller \and $^{1,2,3}$Chloé Braud\\
%        $^1$UT3 - IRIT ; $^2$CNRS ; $^3$ANITI \\ \texttt{firstname.lastname@irit.fr}}

% version with Kushnur gmail adress  khushnur.uiu006@gmail.com
\author{Kushnur Binte Jahangir \\
  UT3 - IRIT \\
  \texttt{khushnur@cse.uiu.ac.bd} \\\And
  Philippe Muller \\
  UT3 - IRIT ; ANITI \\
  \texttt{philippe.muller@irit.fr} \\\And
  Chloé Braud \\
  UT3 - IRIT ; CNRS ; ANITI \\
  \texttt{chloe.braud@irit.fr} \\
  }

\begin{document}
\maketitle
\begin{abstract}
% abstract from Khushnur's report. should be made shorter and updated with most recent human evaluation
Question Generation (QG), the process of generating meaningful questions from a given context, has proven to be useful for several tasks such as question answering  or FAQ generation. 
%has become a significant area of research within the domain of natural language processing. 
While most existing QG techniques %typically 
generate simple, fact-based questions, this research aims to generate questions that can have complex answers (e.g. "why" questions).  %, specifically "why" and "how" questions. 
%by incorporating discourse relations within a text. 
We propose a data augmentation method that uses discourse 
relations to create %generate 
such questions, and experiment on existing English data.
%datasets with more complex, "why"- and "how"-based questions. 
Our approach generates questions based solely on the context without answer supervision, in order to enhance question diversity and complexity.
%By employing an answer-agnostic approach -- without any answer supervision --, and using a Transformer-based model-T5, our research is focused on generating complex questions.
We use an encoder-decoder trained on the augmented dataset to generate either one question or multiple questions at a time, and show that the latter improves over the baseline model when doing a human quality evaluation, without degrading performance according to standard automated metrics.  
\end{abstract}




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}\label{sec:intro}

Question generation is the task of automatically producing varied questions about a document or a set of documents.
It is used to facilitate matching real users' questions looking for information contained in those documents, for instance in the context of Customer Relationship Management or producing FAQs \citep{mass-etal-2020-unsupervised}, in dialogue systems to improve interaction with users \cite{li-etal-2017-learning}, to develop interactive learning for educational purposes \cite{yao-etal-2022-ais,scharpf-etal-2022-collaborative,rao-etal-2023-generation,eo-etal-2023-towards} or as auxiliary tasks for e.g. summarization \cite{pagnoni-etal-2023-socratic}. 
More generically, it can help question-answering (QA) systems by augmenting the amount of instances available for training, as in \cite{duan-etal-2017-question} where automatically generated questions are integrated within a text-based QA system,  or in \cite{bartolo-etal-2021-improving} where they are used as adversarial data to improve robustness.

%as shown in \citet{duan-etal-2017-question,bartolo-etal-2021-improving}.


% ref to applications: 
%also educational \citet{yao-etal-2022-ais,eo-etal-2023-towards,rao23}

%as auxiliary tasks for eg summarization \cite{pagnoni-etal-2023-socratic}


% simple vs complex / 
% draft / from Khushnur's report
% needs to be shorter / with references

% Diversity

%However, t 
 As pointed out in e.g. \cite{sultan-etal-2020-importance,eo-etal-2023-towards}, question diversity is crucial, meaning that a QG system should be able to produce different types of questions, with varied lexical content and associated explicit and implicit answers.
However, the majority of the current research techniques in QG have primarily focused on factoid and multiple-choice questions, where the systems are designed to retrieve factual information or require short-span answers.
% Generating only simple questions limits 
% %Such of these questions do not fully utilize 
% the opportunities for exploring knowledge and understanding that more complex questions could potentially provide.
Since they rely more on reasoning, complex questions 
%Complex questions require more reasoning and explanation and 
might help the user to gain deeper and multiple perspectives on a topic. 
This makes them especially useful in learning environments, complex dialogue systems, and applications that call for a better understanding of text.

%Despite this, 
On the other hand, generating complex questions is a challenging task, as the system must have a grasp of  underlying semantic relationships between different parts of the text. 
This is where discourse relations can play an important role: discourse, or rhetorical, relations are the semantic-pragmatic links between sentences or clauses within a text, describing e.g. causal, temporal or manner connections.
%The semantic connections that exist between phrases or clauses in a text are referred to as discourse relations (e.g. \textit{explanation}, \textit{result}, \textit{manner-means} or \textit{contrast}). 
We assume that including discourse relations into the generation process could help the system 
%can help the QG system 
to produce complicated questions that accurately represent the depth and complexity of the text while also being contextually relevant. 
%by including discourse relations into the process. 
For instance, recognizing a "cause-effect" discourse relation can inspire "why" questions that aim to go deeper into the reasons behind a certain occurrence or circumstance addressed in the text.

In this paper, we present an answer-agnostic QG system, 
%by utilizing 
based on a Transformer-driven model fine-tuned specifically for question generation. 
The emphasis of our QG system is on generating complex questions using discourse relations, with a particular focus on causality related questions to enhance contextual understanding.
Our approach relies on data augmentation: the system is fine-tuned on reference datasets for QA that are reversed to perform the QG task, and augmented with "why" questions that are automatically built from discourse annotated data using simple heuristics. 
By using gold annotated data for discourse, we ensure the quality of our synthetic data.
We use several datasets, the Stanford Question Answering Dataset or SQuAD~\cite{rajpurkar-etal-2016-squad} and Explain Like I’m Five, or ELI5~\cite{fan-etal-2019-eli5} for training a generator, and the Penn Discourse Treebank 2.0 \cite{prasad-etal-2008-penn} for data augmentation. 
% To carry out our question generation tasks, we utilize benchmark datasets like the Stanford Question Answering Dataset (SQuAD)~\cite{rajpurkar-etal-2016-squad} and Explain Like I’m Five (ELI5)~\cite{fan-etal-2019-eli5}, enhancing them through data augmentation, particularly focusing on adding complex questions such as "why" based questions based on discourse relations. We evaluate our generated questions 
We evaluate the results using both automatic evaluation metrics comparing generated questions to existing reference questions about the same paragraphs, and a human assessment of the quality of the generated questions, since automated metrics do not account well for the variety of outputs from answer-agnostic models.  
%Our fine-tuned model with an augmented dataset is able to generate more "why" based questions compared to the baseline model.

% transition to discourse relation 

% descriptions of approach

% contributions summary
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{figure*}
    \centering
    \includegraphics[width=0.8\linewidth]{figures/pipeline.png}
    \caption{Proposed Pipeline For Complex Question Generation Task. }
    \label{fig:method}
\end{figure*}

\section{Related work}
% mostly 
% - datasets with/wo complex questions
% - QG aiming at complex questions beyond factoids: 
%     . multi-hop
%     . reasoning / implicit
% - methods pattern/learnt -> paper on generation of why with syntactict pattern
% - explain distinction answer-aware or answer agnostic
% - main competition ? 
% augmentation ? 
% Improving Reading Comprehension Question Generation with Data Augmentation and Overgenerate-and-rank (Ashok Kumar et al., Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)

Question generation aims at producing relevant questions from documents, that could be a single text or a collection, or other types of inputs such as knowledge bases or images. 
In this paper, we focus on generating questions from a single document, using datasets in which each source text (i.e. context) is associated to question-answer pairs.
In this context, many annotated datasets, primilarly built for QA, have been used for QG with two different settings: answer-aware systems provide the context and the targeted answer to generate the question, while answer-agnostic ones only rely on contexts. 
% %Question generation is a task derived from question-answering: the datasets annotated for the latter have been used to perform the former, using the context containing the answer as input to generate associated questions. 
% %Even if the tasks are different, the datasets come 
% with the same limitation, i.e. 
% %: many existing 
% many datasets for QA/QG are limited, or mainly formed of simple, factoïd questions.
% Even if QG can theoretically produce more diverse questions, its evaluation with automatic metrics is limited to the annotated question-answer pairs, and thus the exploration of complex question generation is challenging.
%thus limiting the work on generating complex questions. 

First systems for QG were rule-based:
%, relying on templates, syntactic and semantic information. 
\citet{heilman-smith-2010-good} proposed to apply syntactic modifications to generate question from declarative sentences, while \citet{dhole2020syn} refined generating patterns using semantic resources.
Interestingly, \citet{agarwal2011automatic} demonstrated the importance of discourse connections for QG by designing patterns also relying on discourse connectives, i.e. specific expressions that can trigger discourse relations (e.g. \textit{because, but, as a result...}), and that also constrain the type of the question to be generated. 
We also rely on syntactic templates and discourse information, but we significantly extend this line of work by using gold discourse annotations and by also including implicit relations.



% \citet{heilman-smith-2010-good} for example proposed a two-stages approach, relying on the ranking of synthetic questions generated by applying syntactic modifications to declarative sentences to change them into questions, an approach mainly limited by the need for labeled training data for ranking. 
% Another heuristic-based approach \cite{agarwal2011automatic} proposed patterns relying on discourse connectives, i.e. specific expressions that can trigger discourse relations (e.g. \textit{because, but, as a result...}).
% They locate parts of the source document containing discourse connectives that are used to infer a question type, and then generate questions based on the excerpt. 
% Their study demonstrated the importance of discourse connections for QG, and we took inspiration from this work, but using the generated questions within a data augmentation setting. 
% Finally, it was proposed in \cite{dhole2020syn} to enhance heuristic-based question generation by combining syntactic rules -- based on universal dependencies --, semantic resources (e.g. VerbNet and PropBank) and specific customized rules, allowing to generate more detailed questions.

Current approaches rely on neural architectures, either RNNs \cite{duan-etal-2017-question,liu2020neural} or Transformers \cite{scialom-etal-2019-self,lopez-2020-transformer,grover2021deep}.
As in our work, \citet{scialom-etal-2019-self,lopez-2020-transformer} proposed an answer-agnostic QG system
%where a Transformer-based 
based on a Transformer architecture
%, with extra features and mechanisms, 
but only evaluated on SQuAD, where complex questions are almost nonexistent.
%, is used to generate questions from source contexts without answer supervision.
% Their system outperfoms \citet{duan-etal-2017-question}'s approach on SQuAD. 
% \citet{lopez-2020-transformer}
% used GPT-2 within both context-aware and context-agnostic settings, and investigate the effect of input size on the quality of the generate questions.
% Although they report competitive results, they also only evaluated their approach on SQuAD, where complex questions are almost unexistant.
Within the same setting,
\citet{grover2021deep} demonstrated the ability of a T5-model to generate relevant and natural questions,
%within an answer-agnostic setting,  
but the authors highlighted the challenge of evaluating generated questions using SQuAD: while the answer-agnostic setting encourages diversity, the generated questions could be far from the reference ones, an issue we address through human evaluation (see Section \ref{sec:human}).
%, since, the absence of answer supervision, while allowing for diversity, leads to generated questions that could be far from the reference questions. 


%Mainly interested with complex questions whose answers span over multiple parts of a document, \citet{fei-etal-2022-cqg} introduced a framework controlling the presence of key entities within generated questions, and demonstrate important improvement on HotpotQA \cite{yang-etal-2018-hotpotqa}.
While these studies successfully applied transformer models such as T5 to QG, they primarily focuses on generic, simple questions, leaving complex questions less explored.
% In addition to the seminal work by \cite{agarwal2011automatic} using discourse connectives, 
Beside \cite{agarwal2011automatic}, discourse information was also leveraged in \citet{stasaski-etal-2021-automatically} where rules are used to extract cause-effect relations in SQuAD: a language model then generates questions on both the cause and effect aspects, and the synthetic questions are evaluated via a QA task.
% used an heuristic-based system \cite{cao2016role} to extract cause-effect relations in SQuAD: a pre-trained model is then fine-tuned to generate questions on both the cause and the effect aspects, and the pairs of synthetic questions and answers are passed to a QA system for evaluation.
Contrary to this work, we use causal relations that are manually annotated to create synthetic data to augment a generic QG model. %, in order to investigate the impact of adding complex questions. 
In addition, relevant to our work is the approach introduced in \cite{lal-etal-2021-tellmewhy}: the authors propose simple transformations based on syntactic templates to create a corpus of "why" questions.
Our heuristics to generate questions are inspired by this work, but our evaluation is not done directly on these synthetic, possibly noisy questions, but on a natural, classic benchmark (e.g. SQuAD). 


%the full potential of using discourse annotations for data augmentation to enhance complxity and diversity of generated questions has not yet been fully explored. 

Also using data augmentation, \citet{ashok-kumar-etal-2023-improving} rely on prompting an LLM using context-answer-question triplets to generate a set of new questions, using varied decoding strategies with the aim of increasing diversity. 
These questions are then ranked, based on perplexity or on a separate model, and the best ranked is added to the training set of a Flan-T5 model fine-tuned on FairytaleQA \cite{xu-etal-2022-fantastic} to generate questions given context-answer pairs.
The evaluation demonstrates that the approach allows to generate questions for which the answer is implicit, i.e. no directly present as text span but need to be inferred. 
Our approach is much simpler, relying on heuristics to generate questions, with a focus on difficult, complex questions while their approach aims at producing generic diversity, with no insight on the types of questions generated.


% SQuAD is the most commonly used for both tasks since it is a large corpus of over 100k question-answer pairs, covering about 500 Wikipedia articles. 
% ELI5 is another popular benchmark dataset that contains long-form answers associated to questions that were asked on Reddit. 
% It covers a vast array of topics and is larger than SQuAD.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Methodology}
\label{sec:methodo}

The pipeline for our question generation task is illustrated in Figure \ref{fig:method} and consists of the following elements:
\begin{itemize}[itemsep=1ex, leftmargin=0.5em, itemindent=0cm]
    \item Two primary datasets, namely SQuAD, and ELI5, serve as a basis for training a model. Since we want to create an answer-agnostic model we only use the context paragraphs and the associated questions as input (ignoring information about the answer).  
    \item The primary datasets are augmented using discourse annotated data, namely the PDTB2 dataset. 
    We extract sentences with specific relations annotated (causal relations). 
    
    \item We apply a manual rule-based approach to derive why-questions from these extracted sentences, relying on their syntactic structures, and add them to the primary datasets, with the original sentences 
    from PDTB2 
    as context paragraphs. 
    
    \item The augmented dataset is then used as an input for fine-tuning an encoder-decoder model from the T5 family, with two different setups: 
    
    \begin{itemize}[itemsep=1ex, leftmargin=1em, itemindent=0cm]
     \item PCSQ (Per Context Single Question), in which each training instance includes a context paragraph and a corresponding single question associated with it. The context and question together serve as a 'training instance' for the T5 model during the fine-tuning process.
     A paragraph can thus appear several times with different questions associated. 
     \item PCMQ (Per Context Multiple Questions), in which each training instance contains a context and all the questions associated with this context.  
     \end{itemize}
     \end{itemize}
     
PCMQ makes for a more complex decoding, but is supposed to encourage question diversity and avoid redundant generations. This setup is made possible because the reference answer for each question is ignored, and so a given paragraph is associated to several different questions in SQuAD.  


\begin{table*}[htbp]
\centering \small
\begin{tabular}{>{\centering\arraybackslash}p{3.5cm}>{\centering\arraybackslash}p{1.5cm}>{\centering\arraybackslash}p{4.5cm}>{\centering\arraybackslash}p{4.5cm}}
\toprule
Sentence/Arg1 & Tense & Question Template & Generated Question \\
\midrule
\textit{ [jaguar} \underline{was} \textit{shocked by mr. ridley's decision]$_{ARG1}$ \underline{because} [...]$_{ARG2}$} & Past &\textbf{Why}\underline{\{aux\}}\textit{\{rest\_arg1\}}? & \textbf{Why} \underline{was} \textit{jaguar shocked by mr. ridley's decision}? \\
\hline
\textit{the beebes' symptoms} \underline{were} \underline{not} \textit{related to the 
carpeting}&Past &\textbf{Why}\underline{\{aux\}}\underline{\{neg\}} \textit{\{rest\_arg1\}}? & \textbf{Why} \underline{were} \underline{not} \textit{the beebes' symptoms related to the carpeting}? \\
\midrule
\textit{frequently, clients} \underline{express} \textit{interest in paintings but do not end up bidding} & Present
& \textbf{Why} \textit{do} \textit{\{rest\_arg1\}}? & *{\textbf{Why} \underline{does} \textit{frequently, clients express interest in paintings but do not end up bidding }?} \\
\bottomrule
\end{tabular}
\caption{Questions generated based on the question templates. Discourse relations link two spans of text ARG1 and ARG2 (explicitly with a marker or implicitly). 
Except for the first example, we only show the first argument of the causal relation (ARG1) as it is the only part used to create the question.
Underlined text in the Sentence/ARG1 column represents verbs, auxiliary verbs, or negation particles extracted from the original sentence. Text in bold in the Question Template column represents fixed elements used in creating the question templates. The generated question column showcases the final questions formed using the respective templates, and incorrect question formations are marked with a star.}
\label{tab:questemplate1}
\end{table*}


%\subsection{Data Augmentation}
%In line with our goal of generating questions based on discourse relations within a text, we adopt a data augmentation strategy. 
Given the scarcity of complex %"why" %and "how" 
%based 
questions in existing datasets, we aim to expand our training examples by integrating more "why" based questions. 
We thus use the PDTB2 dataset which contains documents annotated with discourse relations, including causality relations. These can be signaled by discourse markers, such as "because", "as", and "since", or be {\it implicit}, and the annotation consists of a typical marker that could be inserted. 

We take the sentences from the PDTB2 dataset for both implicit and explicit relations that represent causal relations and produce questions based on some predefined rule-based templates. 
The rules operate on the syntactic structure of a sentence to identify the main verb and auxiliary, and transform it to produce a grammatically correct 
interrogative sentence, 
in a manner similar to how data was produced in the dataset of \cite{lal-etal-2021-tellmewhy}. 
Table \ref{tab:questemplate1} contains some example questions
produced by this procedure. 
More sample of questions generated based on discourse relations is displayed in Appendix \ref{sec:dataAugmentation}.

% The process includes three steps : 
% \begin{itemize}
%     \item Sentence Preparation: For sentence preparation, the focus is on Arg1 (the first argument in a causal relation) before the discourse connective in PDTB2 dataset, aiming to create "why" questions. The selected sentences were converted into lowercase letters for simplification. Certain words at the beginning of Arg1, like "what," "why," etc., were removed to avoid irrelevant content in generated questions. Then, contractions in sentences, such as "won't" and "isn't," were expanded to simplify sentence structure for question template design.
%     \item Handling Grammar: We analyze the sentence's syntactic structure for constructing grammatically accurate questions. Tokenization, POS tagging, and identifying main and auxiliary verbs, and tenses were employed to generate grammatically correct questions. 
%     \item Question Template and Question Generation: The questions are formulated considering the tense and verb structures in the provided sentence. Table \ref{tab:questemplate1} contains some produced questions based on the pre-defined rule-based template from PDTB2 dataset. In Table \ref{tab:questemplate1}, examples a and b are correct, while the incorrect question (example c) is highlighted in red. For some questions, complex sentence structures with multiple verbs lead to generation errors as they fail to fall into predefined rules. More sample of questions generated based on discourse relations is displayed in Appendix \ref{sec:dataAugmentation}.
% \end{itemize} 



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Datasets}
%squad, eli5, pdtb
%why not fantasyQA ? children literature
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
There are numerous datasets available for question generation tasks, including but not limited to NewsQA~\cite{trischler-etal-2017-newsqa}, MS MARCO~\cite{DBLP:conf/nips/NguyenRSGTMD16}, Natural Questions~\cite{kwiatkowski2019natural}, FairytaleQA \cite{xu2022fairytaleqa}, SQuAD~\cite{rajpurkar-etal-2016-squad}, and ELI5~\cite{fan-etal-2019-eli5}. 
Initially, these datasets were %primarily 
designed for question-answering tasks, yet they are now also broadly used in question generation research. 
For the present work we rely on %three 
two datasets, namely SQuAD and ELI5, to perform question generation from a given text.
%for questions associated to a given text.
%The SQuAD dataset is chosen for its diverse range of source paragraphs and questions from Wikipedia, serving as a reliable benchmark for both QA and QG.
%question-answering and generation. 


\textbf{SQuAD} is chosen for its diverse range of source paragraphs and questions from Wikipedia, 
%serving 
it is commonly used as a reliable benchmark for both QA and QG.
%is one of the most commonly used datasets in QA and QG tasks. 
The dataset was produced by Stanford University academics and contains a sizable number of paragraphs that were taken from Wikipedia articles~\cite{rajpurkar-etal-2016-squad}. 
For our experiment, we use the training and development datasets from SQuAD v2.0, which were created by Rajpurkar et al.~\cite{rajpurkar-etal-2018-know} in 2018.\footnote{Retrieved from the GitHub page \url{https://rajpurkar.github.io/SQuAD-explorer/}.}
However, SQuAD focuses mostly on simple factoid questions, so the ELI5 dataset, consisting of more complex questions, is incorporated.
% , offering many more "why" and "how" based questions
% to create more complex questions. 

\textbf{ELI5} which stands for "Explain Like I'm Five", is another popular benchmark dataset used for tasks like QA, QG,
%question-answering, question-generation, 
and other NLP tasks. 
It is sourced from the subreddit r/explainlikeimfive. It provides long-form answers and is available from the Hugging Face website.\footnote{\url{https://huggingface.co/datasets/eli5}.} 
In the ELI5 dataset, each instance consists of a question and user-provided answers on reddit. 
In our context, we consider the answer as the source paragraph and the questions as our system's input.

\textbf{PDTB2.0:} Additionaly, we use the Penn Discourse Treebank Version 2.0 (PDTB2) \cite{prasad-etal-2008-penn} that provides 
%for 
discourse annotated texts.
The PDTB2 is used here to leverage discourse marker-based annotations and produce additional data to augment the training set. %for training. 
Other corpora exist for discourse annotations, but the PDTB is the largest annotated dataset for English including annotations for discourse relations (e.g. \textit{cause, result, manner}), both explicit -- that is triggered by a
%, not only of discourse relations (e.g. \textit{cause, result, manner}), but also 
discourse connective (e.g. \textit{because, as a result, then...}) --, and implicit -- no lexical marker.
Of particular interest, the PDTB2 has annotations of causal relations that we use to create
%generate 
"why" 
%and "how" 
questions.
We use the version provided from the CoNLL 2016 Shared Task~\cite{xue-etal-2016-conll}, 
with level-2 annotations (15 different relation types).

% The Penn Discourse Treebank Version 2.0 (PDTB 2.0)~\cite{prasad-etal-2008-penn}, developed for English language texts provides explicit and implicit discourse relations annotations, which we specifically leveraged for the data augmentation part of our project, particularly those related to causality, making it a valuable source for generating "why"/"how" based questions. 
% We use the version provided from the CoNLL 2016 Shared Task~\cite{xue-etal-2016-conll}, 
% with level-2 annotations (15 different relation types).
% %{\bf XX specify differences with "full" pdtb: relation set / preprocessing etc XX}.  
% %We extracted sentences for both explicit and implicit relations, which have discourse markers such as "because," "since," "as," etc.


 \begin{table*}[ht]
\begin{tabularx}{\textwidth}{|X|}
\hline
\textbf{Input:} Many locals and tourists frequent the southern California coast for its popular beaches, and the desert city of Palm Springs is popular for its resort feel and nearby open spaces.\\
\hline
\textbf{Reference Question:} Other than the desert city why do many locals and tourists frequent southern California?\\
\hline
\textbf{Baseline:} How many locals and tourists frequent the southern California coast?\\

\textbf{SQuAD+ELI5:} What city has a beach?\\

\textbf{+ELI5+PDTB (Exp):} Why do many locals and tourists frequent the southern California coast?\\

\textbf{+ELI5+PDTB (Exp+Imp):} Why do many locals and tourists frequent the southern California coast?\\
\hline
\end{tabularx}
\caption{Example of generated questions by different models in PCMQ approach for SQuAD test data.}
\label{example2}
\end{table*}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiments}

%We experimented the approach with different setup to evaluate the influence of the training data, the model size, and the generating procedure as outlined in Section \ref{sec:methodo}.
Our experiments aim at evaluating the influence of the training data composition, the model size, and the generating procedure as outlined in Section \ref{sec:methodo}. % (i.e. PCSQ \textit{vs} PCMQ).

% training set XX ---
%The dataset distribution was designed with the intention of comparing the performance of various datasets while maintaining consistency in the quantity of training data. 
\paragraph{PCSQ \textit{vs} PCMQ:} %Training sets
%We constructed 
We build the training set differently for the PCSQ and PCMQ setups: for PCSQ -- i.e. one question per paragraph --, we select paragraphs and all questions about them to generate one instance per paragraph-question pair ; for PCMQ, all questions about a paragraph are concatenated in the same instance. 

% We build the training set differently for the PCSQ and PCMQ setups:
% for the training with one question per paragraph (PCSQ), we selected paragraphs and all questions about them to generate one instance per paragraph-question pair. For PCMQ, all questions about a paragraph are concatenated in the same instance.  



\paragraph{Training data composition:}
For the training data, we use SQuAD data as a baseline, and vary the training set by adding either (i) ELI5 data only,
or (ii) ELI5 and the generated questions from the explicit examples of the PDTB, 
or (iii) ELI5 and the generated questions from both the implicit and explicit examples of the PDTB.
% \begin{itemize}
%     \item ELI5 data only,
%     \item or ELI5 and the generated questions from the explicit examples of the PDTB,
%     \item or ELI5 and the generated questions from both the implicit and explicit examples of the PDTB.
% \end{itemize}
%For the training data, we use SQuAD data as a baseline, and various augmentation choices: adding ELI5 data only, or adding ELI5 and some of the generated questions from the PDTB, either restricted to explicit examples, or both implicit+explicit examples. 

For the baseline dataset (SQuAD) we keep approximately $50$k instances for the PCSQ setup, and compare to similarly-sized datasets, by having $20$k instances from SQuAD and $30$k instances from ELI5. 
The additional augmentation from the PDTB is much smaller, with about $1,600$ instances generated from explicit relations, and $1,550$ from implicit relations. 

%For the baseline dataset (SQuAD) we kept approximately 50k instances for the PCSQ setup, and compared to similarly-sized datasets, by having 20k instances from SQuAD and 30k instances from ELI5. The additional augmentation from the PDTB is much smaller, with about 1600 instances generated from explicit relations, and 1550 from implicit relations. 

For the PCMQ setup we cannot hold the number of instances constant without restraining SQuAD too much (there are only $19$k paragraphs in total), so we chose to start from a baseline including all of SQuAD + $30$k ELI5 instances (note that there are much less questions per paragraph in ELI5). We kept the SQuAD-only setup for comprehensiveness, but the PCMQ setup is not entirely fair to this dataset compared to the others. 



% The cumulative training set comprised approximately 50,000 examples (Baseline model) to around 53,000 examples (Augmented models) from SQuAD, ELI5, and PDTB2 datasets. It included the baseline model, SQuAD+ELI5, +ELI5+PDTB (Exp), and +ELI5+PDTB (Exp+Imp) in the PCSQ approach. For the PCMQ approach, the cumulative training set varied from about 19,000 (Baseline model) to around 53,000 (Augmented models), presenting the baseline model, SQuAD+ELI5, +ELI5+PDTB (Exp), and +ELI5+PDTB (Exp+Imp). Test sets included the publicly available SQuAD dataset and a synthetic dataset.
While the training and development sets of SQuAD are publicly available, the test set is not accessible to the public. 
So we divided the development set evenly, allocating $50$\% for validation and the remaining $50$\% for testing. 


\paragraph{Models}
% The experiment was 
 The experiments are conducted using the T5-base model, which is available in the Hugging Face transformers library.\footnote{\url{https://huggingface.co/docs/transformers/index}} 
 The code, written in Python, uses the PyTorch library for fine-tuning the model. 
 This experiment was conducted in a Google Colab Pro environment.
 %\subsection{Fine-Tuning Process}
 The T5 base model has $220$ million parameters. 
 The T5 tokenizer handled data preprocessing, limiting input sequences to $512$ tokens and target sequences to $64$ tokens. The training involves a batch size of $4$, a gradient accumulation size of $32$, and $3$ epochs, employing the Adam optimizer with a learning rate of 1e-4. 

 
 \paragraph{Decoding and post-processing}
 %This section provides an overview of the generated questions for the SQuAD and Synthetic test dataset using both PCSQ and PCMQ formats.
 
To ensure diversity and comprehensiveness in questions, we keep a generation beam of four results for each test paragraph. In the case of PCSQ this ensures we have more than one question to match to the several references in SQuAD. 
% When fine-tuning the model using both PCSQ and PCMQ approaches, the model 
% produces
% %produced 
% four sets of questions for each input paragraph for each approach, configured with a beam size and number of returned sequences set to four. 
% The aim was 
In the PCMQ approach, the model independently 
generates %generated 
varying lengths of questions per set, offering a greater variety compared to PCSQ. We need some post-processing to remove duplicate questions and some not well-formed ones, lacking a '?' mark (incomplete generations), and this impacts the final count of questions obtained for each input text. 
%Examples can be found in Table \ref{tab:pcmq} in Appendix \ref{sec:generatedquestions}). 
Examples of questions generated by different models are shown in Table \ref{example2}.
In Table \ref{tab:pcmq} in Appendix \ref{sec:generatedquestions}, we provide a more complete example of a question generated by the "+ELI5+PDTB (Exp+Imp)" model in the PCMQ setup. 

% \begin{table}[htbp]
%   \centering
%   \begin{tabular}{|p{\linewidth}|}
%     \hline
%     \\
%     Input: The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse ("Norman" comes from "Norseman") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants would gradually merge with the Carolingian-based cultures of West Francia. The distinct cultural and ethnic identity of the Normans emerged initially in the first half of the 10th century, and it continued to evolve over the succeeding centuries.\\ \\
%     Generated Questions: 
%     \begin{enumerate}
%         \item {Who gave their name to Normandy? What was the name of the region in France? Who did the Normans swear fealty to? When did the distinct cultural and ethnic identity of the Norman people emerge?}
%     \end{enumerate}\\
%     \hline
%   \end{tabular}
%   \caption{Example of input text and generated questions in PCMQ format for SQuAD test set. The question is generated by "+ELI5+PDTB2(Exp+Imp)".}
%   \label{tab:pcmq}
% \end{table}

% + results

% \begin{table*}[ht]
% \begin{tabularx}{\textwidth}{|X|}
% \hline
% \textbf{Input:} Many locals and tourists frequent the southern California coast for its popular beaches, and the desert city of Palm Springs is popular for its resort feel and nearby open spaces.\\
% \hline
% \textbf{Reference Question:} Other than the desert city why do many locals and tourists frequent southern California?\\
% \hline
% \textbf{Baseline:} How many locals and tourists frequent the southern California coast?\\

% \textbf{SQuAD+ELI5:} What city has a beach?\\

% \textbf{+ELI5+PDTB (Exp):} Why do many locals and tourists frequent the southern California coast?\\

% \textbf{+ELI5+PDTB (Exp+Imp):} Why do many locals and tourists frequent the southern California coast?\\
% \hline
% \end{tabularx}
% \caption{Example of generated questions by different models in PCMQ approach for SQuAD test data.}
% \label{example2}
% \end{table*}

% \begin{table*}[htbp]
%   \centering
%   \begin{tabularx}{\linewidth}{|X|}
%     \hline
%     {\small Input: The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse ("Norman" comes from "Norseman") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants would gradually merge with the Carolingian-based cultures of West Francia. The distinct cultural and ethnic identity of the Normans emerged initially in the first half of the 10th century, and it continued to evolve over the succeeding centuries.}\\
%     \hline
%     %Generated Questions: 
%     \begin{enumerate}
%         \item Who gave their name to Normandy? What was the name of the region in France? Who did the Normans swear fealty to? When did the distinct cultural and ethnic identity of the Norman people emerge?
%     \end{enumerate}\\
%     \hline
%   \end{tabularx}
%   \caption{Example of input text and generated questions in PCMQ format for SQuAD test set. The question is generated by "+ELI5+PDTB2(Exp+Imp)".}
%   \label{tab:pcmq}
% \end{table*}
\iffalse
details of system, experimental comparison

evaluation methods: choice of reference, automated metrics, human evaluation

analysis of results 
\fi
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% \begin{table*}[htbp]
%     \centering
%     \label{scores}
    
%     \begin{tabular}{@{}llcccccc@{}}
%         \toprule
%         Approach & Training & BLEU-1 & BLEU-2 & BLEU-3 & BLEU-4 & ROUGE-L & METEOR  \\
%         \midrule
%         PCSQ & SQuAD Baseline & \textbf{37.51}&\textbf{25.25} &\textbf{18.54}&\textbf{13.81} &\textbf{45.57} & \textbf{46.13}\\
%         & +ELI5 & 36.77&24.41&17.75&12.99 &45.07 & 45.24	\\
%         & +ELI5+P-E & 37.10&24.74&	18.09&13.40& 45.18 & 45.18\\
%         & +ELI5+P-(E+I) & \textbf{37.51}&25.11&18.36&13.56 &45.53 &	45.53\\
%         \midrule
%         PCMQ & SQuAD alone & 33.48 &21.94 &15.86 &	11.60 &	41.27 &	40.34 \\
 
%         & SQuAD+ELI5 & 33.55&22.14&16.12&11.86&	41.52 &	40.36\\
%         &+ELI5+P-E & 33.78&22.39&16.32&12.03 & \textbf{41.63} & 40.77 \\
%         & +ELI5+P-(E+I) &\textbf{33.89} &\textbf{22.45}&\textbf{16.34}&\textbf{12.07}&\textbf{41.63} & \textbf{40.91}
%  \\
%         \bottomrule
%     \end{tabular}
%         \caption{ Comparative performance of PCSQ and PCMQ approaches with different models. The scores are given in percentages. The highest scores in each metric and approach are highlighted in bold. Here, the baseline model is trained on the SQuAD dataset only. The results are presented for t5-base models. P-E and I stand for PDTB explicit and implicit relations respectively. Note that PCMQ/SQuAD alone is here for reference but not comparable to the other PCMQ setups. }
%         \label{evaluation}
% \end{table*}


\begin{table*}[htbp]
    \centering
    \label{scores}
    
    \begin{tabular}{@{}llcccccc@{}}
        \toprule
        Approach & Training & BLEU-1 & BLEU-2 & BLEU-3 & BLEU-4 & ROUGE-L & METEOR  \\
        \midrule
        PCSQ & SQuAD Baseline & \textbf{37.51}&\textbf{25.25} &\textbf{18.54}&\textbf{13.81} &\textbf{45.57} & \textbf{46.13}\\
        & +ELI5 & 36.77&24.41&17.75&12.99 &45.07 & 45.24	\\
        & +ELI5+P-E & 37.10&24.74&	18.09&13.40& 45.18 & 45.18\\
        & +ELI5+P-(E+I) & \textbf{37.51}&25.11&18.36&13.56 &45.53 &	45.53\\
        \midrule
        PCMQ & SQuAD alone & 33.48 &21.94 &15.86 &	11.60 &	41.27 &	40.34 \\
 
        & SQuAD+ELI5 & 33.55&22.14&16.12&11.86&	41.52 &	40.36\\
        &+ELI5+P-E & 33.78&22.39&16.32&12.03 & \textbf{41.63} & 40.77 \\
        & +ELI5+P-(E+I) &\textbf{33.89} &\textbf{22.45}&\textbf{16.34}&\textbf{12.07}&\textbf{41.63} & \textbf{40.91}
 \\
        \bottomrule
    \end{tabular}
        \caption{ Comparative performance of PCSQ and PCMQ approaches with different models. The scores are given in percentages. The highest scores in each metric and approach are highlighted in bold. Here, the baseline model is trained on the SQuAD dataset only. The results are presented for t5-base models. P-E and I stand for PDTB explicit and implicit relations respectively. Note that PCMQ/SQuAD alone is here for reference but not comparable to the other PCMQ setups. }
        \label{evaluation}
\end{table*}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Automated evaluation}

We generated questions in both approaches on the SQuAD left-out paragraphs and evaluated against the corresponding reference questions using automated evaluation metrics: BLEU (Bilingual Evaluation Understudy)~\cite{papineni-etal-2002-bleu}, METEOR (Metric for Evaluation of Translation with Explicit ORdering)~\cite{lavie2009meteor}, and ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation-longest common subsequences or LCS)~\cite{lin-2004-rouge}.
These metrics are widely adopted in the literature for evaluating question generation. 
The evaluation tasks involved the use of the following library packages: the Natural Language Toolkit (NLTK), ROUGE, and METEOR in calculating BLEU (1 to 4), Rouge-L, and METEOR scores. 
Note that those measures, relying on common ngrams or subsquences between reference and system outputs, are moderately appropriate to our setup, where we try to generate more diverse questions than are present in the reference, without a target answer. 
We address this problem with a human evaluation in Section \ref{sec:human}.



%\subsection{Matching references to predictions} 

A total of 500 paragraphs were chosen from the SQuAD test dataset to assess the question-generation capability of our model. 
These paragraphs consist of multiple reference questions, and correspondingly, our model generates multiple questions for each paragraph. 
To accommodate the presence of multiple references and generated questions per paragraph in the SQuAD dataset, we implemented a mapping approach to find out which reference and generated question pairs are more relevant to each other for the evaluation, especially focusing on one-to-one match between reference and generated questions. 
For both PCSQ and PCMQ approaches, we combined questions generated for each context's four outputs from the beam. 
Using automatic evaluation metrics such as BLEU, ROUGE-L, or METEOR, we then calculated scores for each pair of matched generated and reference questions. 
This filtering resulted in a one-to-one matching between generated and reference questions, ensuring meaningful evaluation of our model's question-generation accuracy. 
%We didn't use any information about the number of reference questions for a given test paragraph. 
% 
Given the decoding procedure, the average number of non-duplicate generated questions was about 3.9 for PCSQ, and 9.5 for PCQM (with small variations depending on the training data).


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Results}
%\subsection{Analysis}

% \begin{table*}[htbp]
%     \centering
%     \label{scores}
    
%     \begin{tabular}{@{}llcccccc@{}}
%         \toprule
%         Approach & Training & BLEU-1 & BLEU-2 & BLEU-3 & BLEU-4 & ROUGE-L & METEOR  \\
%         \midrule
%         PCSQ & SQuAD Baseline & \textbf{37.51}&\textbf{25.25} &\textbf{18.54}&\textbf{13.81} &\textbf{45.57} & \textbf{46.13}\\
%         & +ELI5 & 36.77&24.41&17.75&12.99 &45.07 & 45.24	\\
%         & +ELI5+P-E & 37.10&24.74&	18.09&13.40& 45.18 & 45.18\\
%         & +ELI5+P-(E+I) & \textbf{37.51}&25.11&18.36&13.56 &45.53 &	45.53\\
%         \midrule
%         PCMQ & SQuAD alone & 33.48 &21.94 &15.86 &	11.60 &	41.27 &	40.34 \\
 
%         & SQuAD+ELI5 & 33.55&22.14&16.12&11.86&	41.52 &	40.36\\
%         &+ELI5+P-E & 33.78&22.39&16.32&12.03 & \textbf{41.63} & 40.77 \\
%         & +ELI5+P-(E+I) &\textbf{33.89} &\textbf{22.45}&\textbf{16.34}&\textbf{12.07}&\textbf{41.63} & \textbf{40.91}
%  \\
%         \bottomrule
%     \end{tabular}
%         \caption{ Comparative performance of PCSQ and PCMQ approaches with different models. The scores are given in percentages. The highest scores in each metric and approach are highlighted in bold. Here, the baseline model is trained on the SQuAD dataset only. The results are presented for t5-base models. P-E and I stand for PDTB explicit and implicit relations respectively. Note that PCMQ/SQuAD alone is here for reference but not comparable to the other PCMQ setups. }
%         \label{evaluation}
% \end{table*}

The results presented in Table \ref{evaluation} provide insights into the impact of data augmentation and the effectiveness of PCSQ and PCMQ approaches. 
For PCSQ, the model trained solely on SQuAD slightly outperforms augmented models in all mentioned evaluation metrics, highlighting the effectiveness of focused training on a single dataset. On the other hand, PCMQ, when using everything from SQuAD, ELI5, and the PDTB2 augmentation, outperforms slightly the baseline in BLEU (1 to 4), ROUGE-L, and METEOR.

% redundant with end of section
% Note that since the automated metrics are geared towards lexical overlap with reference questions, 
% they should unfavor approaches that generate "new" questions that were not in the reference with their answers. But we show here that the augmentation doesn't seem to impact the measures with respect to existing reference questions. 

% Comparing PCSQ and PCMQ highlights clear patterns. The PCSQ and PCMQ models might have different capacities to learn from and handle complexity. Given the same augmented examples, it’s possible that the PCMQ model, due to its design to handle multiple questions per context, is better
% equipped to benefit from the diverse data, while the PCSQ model performs best in a more straightforward learning task.

% Substantial differences appear when considering different question types. 
When train with PDTB derived instances, the number of "why" question is higher (+38\% when using explicit and implicit with PCMQ wrt the baseline, +24\% with only explicit). 
In PCSQ, the increase in "why" questions is limited (going from 0 for the baseline to 10 for the full training data), reflecting its lower effectiveness in generating this question type. 
Questions in "how" do not seem positively affected (each system generates almost the same amount), but we did not distinguish simple "how" questions (asking for quantities, i.e "how much/many") and more complex ones.  
We just observed that some generated "how" questions were causal in nature, but more manual analysis is needed to evaluate this precisely. 

% To further investigate the model’s capability to generate complex questions, we focused on the "why" and "how" questions generated by the "+ELI5+PDTB (Exp+Imp)"
% model in PCMQ approach, along with their corresponding reference questions. Table \ref{evaluation(whyHow)}, presents varying quality metrics, with "why" questions displaying scores ranging from 36.59 (BLEU-1) to 11.53 (BLEU-4) to 38.21 (METEOR). The METEOR metric has the highest score among other metrics, suggesting that the generated "why" questions have reasonable linguistic quality and semantic similarity to the reference questions. The "how" questions demonstrate slightly higher scores across all metrics. This suggests that the generated "how" questions are more similar to the reference questions with respect to n-gram precision (based on BLEU scores), the longest common subsequence (ROUGE-L), and overall linguistic quality (METEOR). 

Thus, aligned with our objective, our augmentation techniques effectively increased the number of
generated "why" questions, particularly within the PCMQ models, without detrimentally affecting the quality of the questions generated as a whole, at least according to the automated metrics. %Moreover, automatic
% evaluation metrics like BLEU and ROUGE-L show that the performance of the augmented
% models was either advantageous or comparable to that of the baseline model, while simultaneously achieving success in generating more complex questions.

This is notable since our models are not trained on example answers, meaning they can generate questions about any aspect of the chosen paragraph, for which it is likely the reference does not include any question-answer pair. 

This is why it is important to have a separate, more fine-grained evaluation of the quality of the generated answers, and this is the subject of the following section. 



% \begin{table*}[htbp]
%     \centering
%     \begin{tabularx}{\textwidth}{@{} >{\centering\arraybackslash}X >{\centering\arraybackslash}X >{\centering\arraybackslash}X >{\centering\arraybackslash}X >{\centering\arraybackslash}X >{\centering\arraybackslash}X @{}}
%         \toprule
%         Question Type & Baseline & SQuAD+ELI5 & +ELI5+PDTB (Exp) & +ELI5+PDTB (Exp+Imp) \\ 
%         \midrule
%         how & \textbf{866} & 747 & 781 & 772 \\ 
%         why & 63 & 64 & 78 & \textbf{87}\\ 
%         \bottomrule
%     \end{tabularx}
%     \caption{The table presents the number of "why" and "how" questions generated by various models in PCMQ approach, including Baseline, SQuAD+ELI5, +ELI5+PDTB (Exp), and +ELI5+PDTB (Exp+Imp). All the models are fine-tuned with t5-base.}
%     \label{pcmq_stat_subset}
% \end{table*}
% \raggedbottom


\begin{table}[htbp]
    \small
    \centering
    \begin{tabular}{@{} b{0.35\columnwidth} *{2}{>{\centering\arraybackslash}b{0.25\columnwidth}} @{}}
        \toprule
        Model & how & why \\ 
        \midrule
        SQuAD alone & \textbf{866} & 63 \\ 
        SQuAD+ELI5 & 747 & 64 \\
        +ELI5+P-E & 781 & 78 \\
        +ELI5+P-(E+I) & 772 & \textbf{87} \\ 
        \bottomrule
    \end{tabular}
    \caption{The table presents the number of "why" and "how" questions generated by various models in PCMQ
approach. The results are presented for T5-base models. Here, the baseline model is trained on the SQuAD dataset only.  P-E and I stand for PDTB explicit and implicit relations respectively.  }
    \label{pcmq_stat}
\end{table}








% \begin{table*}[!htbp]
%     \centering
%     \begin{tabularx}{\textwidth}{@{}lXXXXX@{}}
%         \toprule
%         Question Type & Baseline & SQuAD+ELI5 & +ELI5+PDTB (Exp) & +ELI5+PDTB (Exp+Imp) \\ 
%         \midrule
%         how & 176 & 196 & 187 & \textbf{190} \\ 
%         why & 0 & 1 & 10 & \textbf{3} \\ 
%         \bottomrule
%     \end{tabularx}
%     \caption{Number of "why" and "how" questions generated in the PCSQ approach by the Baseline model, SQuAD+ELI5, +ELI5+PDTB (Exp), and +ELI5+PDTB (Exp+Imp). All models are fine-tuned with t5-base.}
%     \label{pcsq_state_why_how}
% \end{table*}
% \raggedbottom

% \begin{table*}[!h]
%     \centering
%     \label{scores}
%     \begin{tabular}{@{}lcccccc@{}}
%         \toprule
%         Question Type & BLEU-1 & BLEU-2 & BLEU-3 & BLEU-4 & ROUGE-L & METEOR  \\
%         \midrule
%         Why & 36.59 & 22.47 & 16.10 & 11.53 & 30.58 & \textbf{38.21} \\
%         How & 38.85 & 25.35 & 17.92 & 12.85 & 33.70 & \textbf{39.96} \\
%         \bottomrule
%     \end{tabular}
%     \caption{Evaluation scores for "why" and "how" questions based on BLEU-1, BLEU-2, BLEU-3, BLEU-4, ROUGE-L, and METEOR metrics. Scores are given in percentage. Results are showcased for the t5-base model, fine-tuned using the SQuAD, ELI5, and PDTB2 datasets, encompassing both explicit and implicit relations.}
%     \label{evaluation(whyHow)}
% \end{table*}
\raggedbottom
\section{Human evaluation}
\label{sec:human}
\begin{table}[htbp]
\centering
\begin{tabular}{lrrr}
\toprule
{Model} & {Bad} & {$\approx$ ok} & {Good} \\
\midrule
Baseline & 39.29 & 10.71 & 50.00 \\
All+P-E & 40.43 & 2.13 & 57.45 \\
All+P-(E+I) & 26.15 & 3.08 & {\bf 70.77} \\
\bottomrule
\end{tabular}
\caption{Human quality assessment of generated questions in \% according to the data that was used to train the generation model (PCMQ setup). Baseline means T5 was only fine-tuned on SQuAD. }
\label{tab:ad}
\end{table}
% or joint with analysis of results
We conducted a human evaluation to assess the quality of questions generated in PCMQ approach by three models: Baseline model, +ELI5+PDTB (Exp), and +ELI5+PDTB (Exp+Imp), all fine-tuned from the T5-base model. 
Two of the authors annotated a subset of randomly selected questions and their context from the SQuAD test dataset using a set of 7 predetermined categories that included subcategories for incorrect questions (more details are provided in Appendix \ref{tab:criteria}); the selection was done by a third author, who kept hidden the system that produced each question. There were 137 annotated questions, some generated by more than one system. 
Adjudications of annotations were done by the two annotators. It turned out some of the error subcategories were quite similar, and the final categories were restricted to three cases: (1) the generated question is good: fluent, and can be answered from the source paragraph, (2) the generated question is almost good: minor disfluency and the answer is in the paragraph, (3) the question is either impossible to understand or too vague, or the paragraph does not contain an answer to the question.  

Cohen's kappa ($\kappa$)  was 0.48 on the 7 original categories, indicating a moderate level of inter-annotator agreement, but % .62 with 3 error kinds, cf google sheet 
was 0.74 when only distinguishing between good questions and all the rest. 

Table \ref{tab:ad} presents the model-wise percentage distribution of the final adjudicated categories, providing insights into the quality assessment of generated questions. The +ELI5+PDTB (Exp+Imp) model exhibits fewer "bad" questions and a substantial increase in "good" questions compared to the baseline, presenting improved question quality with explicit and implicit relation augmentations.


Moreover, from the annotated questions, we determined the distribution of good, almost okay, and bad questions for each question type (e.g., what, why, etc.), see Table \ref{tab:ques-dis}. %\ref{tab:ques-dis}.
We can see for instance that implicit examples help generating more why questions ($32$), but with a cost on the average quality of the questions (61\% of good questions), while using only explicit examples has a much higher quality (79\% of good questions, vs 55\% for the baseline) with less why questions generated ($12$).  
This is done on a small sample of "why questions" so must be taken with a grain of salt. 

% \subsection{Analysis of Complex Question Generation}
% We evaluated our models using a synthetic dataset. Table \ref{complexquestion} presents an example of generated questions in PCMQ approach by the +ELI5+PDTB (Exp) model, highlighting its effective use of discourse relations to create complex questions related to a soccer match cancellation. The questions exhibit the model's ability to utilize cause-effect relationships within the text. However, there are questions, such as question 2, where the generated question lacks contextual accuracy, indicating a need for further improvement of the model.
%\iffalse



\vspace{-0.5\baselineskip} 
\begin{table}[htbp]
    \centering
\begin{tabular}{llrr}
\toprule
 &  & \% correct & nb \\
Type & Model &  &  \\
\midrule
\multirow[c]{3}{*}{How} & ELI5+Exp & 58.33 & 12 \\
 & ELI5+Exp+Imp & 80.95 & 21 \\
 & Baseline & 50.00 & 18 \\
Others & ELI5+Exp & 0.00 & 1 \\
\multirow[c]{3}{*}{What} & ELI5+Exp & 43.75 & 16 \\
 & ELI5+Exp+Imp & 70.00 & 5 \\
 & Baseline & 61.36 & 22 \\
\multirow[c]{3}{*}{When} & ELI5+Exp & 75.00 & 4 \\
 & ELI5+Exp+Imp & 100.00 & 4 \\
 & Baseline & 33.33 & 3 \\
\multirow[c]{2}{*}{Where} & ELI5+Exp & 0.00 & 1 \\
 & Baseline & 75.00 & 2 \\
\multirow[c]{2}{*}{Who} & ELI5+Exp & 100.00 & 1 \\
 & ELI5+Exp+Imp & 100.00 & 3 \\
\multirow[c]{3}{*}{Why} & ELI5+Exp & 79.17 & 12 \\
 & ELI5+Exp+Imp & 60.94 & 32 \\
 & Baseline & 54.55 & 11 \\
\bottomrule
\end{tabular}
   
    \caption{Breakdown of the number of questions in the human evaluation by type, with the \% of correct questions and the number of generated questions. }
    \label{tab:ques-dis}
\end{table}




\section{Conclusion}
We presented an approach based on discourse relation annotations to augment a question generation training set, in the case of a general answer-agnostic question generation system, and with a focus on causal questions. 
Our experiments show that with a small set of additional instances we can make the system generate more causal questions with a good quality, as evaluated by human annotators, and with almost no difference with respect to classic automated metrics for question generation. 
This is only preliminary, as the results would need to be tested on different base question-answer corpora, and more human evaluation would be precious to better separate the roles of the different factors at play here. 
It would also be interesting to investigate the impact of including other discourse relation types to generate different kinds of questions (e.g. "how" questions with relations of the type "goal" or "manner"). 

%good results / human quality
%more human evaluation to separate roles of different aspects (here for lack of time no control of the separate role of ELI5)

%perspectives: test on other datasets with more diversity to start with, and already 
%some why questions (but apart from eli5 other why corpora are automatically created
%\newpage
\section{Limitations}
The proposed approach augments existing datasets and thus depends on the quality and diversity of this basis. 
We are also reliant on existing annotated discourse data, which is costly to produce, and exist only in various quantities for some languages. 
As mentioned in the conclusion, the results would need to be tested on different base question-answer corpora and other languages, and more human evaluation is needed to better separate the roles of the different factors at play here. 
A limitation of our evaluation is the use of automated metrics, which are already known not to be very adequate to compare semantically equivalent questions if they have lexical differences, but are even more inappropriate with the goal to produce diverse questions not tied to existing answers. 
\section*{Acknowledgements}

This work is supported by the Quantum project (ANR-19-CE23-0022).
Our work has benefited from the AI Interdisciplinary Institute ANITI. ANITI is funded by the French ``Investing for the Future – PIA3'' program under the Grant agreement n°ANR-19-PI3A-0004. 
%This work was partially supported by the ANR (ANR-19-PI3A-0004) through the AI Interdisciplinary Institute, ANITI, as a part of France’s “Investing for the Future — PIA3" program. 
This work is also partially supported by the AnDiaMO project (ANR-21-CE23-0020).
Chloé Braud and Philippe Muller are part of the program DesCartes and are also supported by the National Research Foundation, Prime Minister’s Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) program.

% Our proposed question generation models have limitations. Given the nature of the answer-agnostic approach, it may perform poorly in traditional metrics. Moreover, the limitation includes data composition issues and a lack of balanced dataset distributions. These issues can impact the learning and generalization capability of the model. Moreover, the automatic evaluation metrics can't capture the semantics and the context of the generated questions very well. Though we perform human evaluation for our system, performing human annotation on large numbers of questions is time-consuming and costly. Therefore, we need proper evaluation metrics for the overall QG task.


\bibliography{anthology,custom}
\bibliographystyle{acl_natbib}
\onecolumn


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\appendix

\begin{table*}[t]
\centering
\begin{tabularx}{\textwidth}{|X|X|}
\hline
\multicolumn{1}{|c|}{\textbf{Paragraph}} & \multicolumn{1}{c|}{\textbf{Generated Questions}} \\ \hline
\textcolor{magenta}{Due to the heavy rain, the soccer match was canceled\textsuperscript{1}}, and as a result, the \textcolor{blue}{players were disappointed\textsuperscript{3}}. \textcolor{olive}{Since the field was waterlogged, it was unsafe to play\textsuperscript{6}}. The \textcolor{orange}{organizers made the decision to cancel the match\textsuperscript{7}}, and consequently, the \textcolor{purple}{players had to wait for another opportunity to showcase their skills\textsuperscript{4}}. Additionally, the \textcolor{cyan}{spectators were also disappointed\textsuperscript{5} because they were eagerly looking forward to the game.} The cancellation of the match, due to the inclement weather, not only affected the players' morale but also dampened the overall excitement surrounding the event. & 
\textcolor{magenta}{1. Why was the soccer match canceled?} \newline
\textcolor{red}{2. Why was the soccer match canceled due to heavy rain?} \textbf{(Incorrect Question)}\newline
\textcolor{blue}{3. What caused the players to be disappointed?} \newline
\textcolor{purple}{4. What caused players to wait for another opportunity to showcase their skills?} \newline
\textcolor{cyan}{5. Why were spectators disappointed?} \newline
\textcolor{olive}{6. Why was it unsafe to play?} \newline
\textcolor{orange}{7. Who made the decision to cancel the match?} \\ \hline
\end{tabularx}
\caption{Example of generation from one paragraph. The table presents a text passage along with a set of generated questions intended to reflect cause-effect relationships described within the text. Corresponding answers within the text passage are color-coded to match their respective questions, and annotated with superscripts denoting question numbers for clear cross-referencing. The question is generated by +ELI5+PDTB (Exp) model in PCMQ approach.}
\label{complexquestion}
\end{table*}

\section{Sample of Generated Questions from Data Augmentation} \label{sec:dataAugmentation}
The questions generated from the PDTB2 dataset, along with the corresponding discourse relation and discourse connective used in their formulation, are presented in Table \ref{tab:sentenceandquestion}.

\begin{table}[htbp]
\centering
\begin{tabular}{|p{0.9\textwidth}|}
\hline
\textbf{Sentence:} jaguar was shocked by mr. ridley's decision \underline{because} management had believed the government wouldn't lift the golden share without consulting the company first. \textbf{(Explicit Relation) }\\

\textbf{Connective} : because\\

\textbf{Arg1:} jaguar was shocked by mr. ridley's decision

\textbf{Question:} Why was jaguar shocked by mr. ridley's decision? \\
\hline
\textbf{Sentence:} jeastern airlines' creditors have begun exploring alternative approaches to a chapter 11 reorganization \underline{\textbf{,}} they are unhappy with the carrier's latest proposal. \textbf{(Implicit Relation) }\\
\textbf{Connective} : None\\
\textbf{Arg1:} jeastern airlines' creditors have begun exploring alternative approaches to a chapter 11 reorganization

\textbf{Question:} Why have eastern airlines' creditors begun exploring alternative approaches to a chapter 11 reorganization? \\
\hline
\end{tabular}
\caption{Examples of generated questions for both explicit and implicit relation from PDTB2 dataset.}
\label{tab:sentenceandquestion}
\end{table}

\section{Sample of Generated Questions} \label{sec:generatedquestions}
We have included some samples of generated questions from the fine-tuned models on test dataset in Table \ref{tab:pcmq} and \ref{example1}. %, and \ref{example2} respectively. 
In Table \ref{tab:pcmq}, for a given input paragraph, the augmented model in PCMQ approaches generated 4 set of questions where each question set may contain various length questions.

\raggedbottom

\begin{table}
  \centering
  \begin{tabularx}{\textwidth}{|X|}
%  \begin{tabular}{|p{\linewidth}|}
    \hline
    \\
    Input: The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse ("Norman" comes from "Norseman") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants would gradually merge with the Carolingian-based cultures of West Francia. The distinct cultural and ethnic identity of the Normans emerged initially in the first half of the 10th century, and it continued to evolve over the succeeding centuries.
\\
    
    \\Generated Questions: \\
    \begin{enumerate}
        \item {Who gave their name to Normandy? What was the name of the region in France? Who did the Normans swear fealty to? When did the distinct cultural and ethnic identity of the Norman people emerge?}

        \item {What was the name of the Normans in the 10th and 11th centuries? What was Normandy? Who did the Norman people swear fealty to? When did the distinct cultural and ethnic identity of the people emerge?}

        \item {\textcolor{red}{What was the name of the Normans in the 10th and 11th centuries?} \textcolor{red}{What was Normandy? Who did the Norman people swear fealty to?}\textcolor{red}{When did the distinct cultural and ethnic identity of the people emerge? }\colorbox{yellow}{How did the people of Normand}}

        \item {Who gave their name to Normandy? \textcolor{red}{Who did the Normans swear fealty to?}What did the people of Normands mix with? \textcolor{red}{When did the distinct cultural and ethnic identity of the Norman people emerge?}}

    \end{enumerate}
    \\
    \hline
  \end{tabularx}
  \caption{Example of input text and generated questions in PCMQ format for SQuAD test set. The question is generated by "+ELI5+PDTB2(Exp+Imp)". The incorrect questions are red-colored and the yellow-highlighted text present an incomplete question.}
  \label{tab:pcmq}
\end{table}




























\begin{table}
\begin{tabularx}{\textwidth}{|X|}
\hline
\textbf{Input:} Price controls exacerbated the crisis in the US. The system limited the price of "old oil" (that which had already been discovered) while allowing newly discovered oil to be sold at a higher price to encourage investment. Predictably, old oil was withdrawn from the market, creating greater scarcity. The rule also discouraged development of alternative energies. The rule had been intended to promote oil exploration. Scarcity was addressed by rationing (as in many countries). Motorists faced long lines at gas stations beginning in summer 1972 and increasing by summer 1973.\\
\hline
\textbf{Reference Question:} Why was old oil withdrawn from the market?\\
\hline
\textbf{Baseline:} What was withdrawn from the market?\\

\textbf{SQuAD+ELI5:} What did the price control limit?

\textbf{+ELI5+PDTB (Exp):} Old oil was withdrawn from the market creating what?\\

\textbf{+ELI5+PDTB (Exp+Imp):} Why was old oil withdrawn from the market?\\
\hline
\end{tabularx}
\caption{Example of generated questions by different models in PCMQ approach for SQuAD test data.}
\label{example1}
\end{table}





% \begin{table}
% \begin{tabularx}{\textwidth}{|X|}
% \hline
% \textbf{Input:} Many locals and tourists frequent the southern California coast for its popular beaches, and the desert city of Palm Springs is popular for its resort feel and nearby open spaces.\\
% \hline
% \textbf{Reference Question:} Other than the desert city why do many locals and tourists frequent southern California?\\
% \hline
% \textbf{Baseline:} How many locals and tourists frequent the southern California coast?\\

% \textbf{SQuAD+ELI5:} What city has a beach?\\

% \textbf{+ELI5+PDTB (Exp):} Why do many locals and tourists frequent the southern California coast?\\

% \textbf{+ELI5+PDTB (Exp+Imp):} Why do many locals and tourists frequent the southern California coast?\\
% \hline
% \end{tabularx}
% \caption{Example of generated questions by different models in PCMQ approach for SQuAD test data.}
% \label{example2}
% \end{table}







\section{Human Annotation} 
\label{sec:generatedquestions}

\subsection{Sample of Annotated Questions}
We have provided some sample questions that have been annotated by the annotators. The annotated questions can be found in Table \ref{annotated_ques}.
\label{an}
\begin{table}
\small
\begin{tabularx}{\textwidth}{|X|X|X|}
\hline
\textbf{Input Text} & \textbf{Generated Questions} & \textbf{Category} \\
\hline
Price controls exacerbated the crisis in the US. \textcolor{red}{The system limited the price of "old oil" (that which had already been discovered) while allowing newly discovered oil to be sold at a higher price to encourage investment.} Predictably, old oil was withdrawn from the market, creating greater scarcity. The rule also discouraged development of alternative energies. The rule had been intended to promote oil exploration. Scarcity was addressed by rationing (as in many countries). Motorists faced long lines at gas stations beginning in summer 1972 and increasing by summer 1973. &
Why was old oil withdrawn from the market? & 
Good question. Answer is present in the text.  \\
\hline
Highly concentrated sources of oxygen promote rapid combustion. Fire and explosion hazards exist when concentrated oxidants and fuels are brought into close proximity; an ignition event, such as heat or a spark, is needed to trigger combustion. Oxygen is the oxidant, not the fuel, but nevertheless the source of most of the chemical energy released in combustion. Combustion hazards also apply to compounds of oxygen with a high oxidative potential, such as peroxides, chlorates, nitrates, perchlorates, and dichromates because they can donate oxygen to a fire. &
How do compounds with oxidation potential contribute oxygen to? & 
Incorrect question but with relevant words from
the input. \\

\hline
 As indigenous territories continue to be destroyed by deforestation and ecocide, such as in the Peruvian Amazon indigenous peoples' rainforest communities continue to disappear, while others, like the Urarina continue to struggle to fight for their cultural survival and the fate of their forested territories. Meanwhile, the relationship between non-human primates in the subsistence and symbolism of indigenous lowland South American peoples has gained increased attention, as have ethno-biology and community-based conservation efforts. &
 Why do indigenous territories continue to be destroyed by deforestation and ecocide? &
 Grammatically correct but the answer doesn’t
exist. \\
 \hline
\end{tabularx}
\caption{Sample of Annotated Questions by the Annotators. Red-colored text represents the answer texts for the question within the paragraph. The input paragraph is from SQuAD test dataset.}
\label{annotated_ques}
\end{table}



\subsection{Question Category for Annotations}

The annotators assigned each question to one of the seven predetermined categories. Details of each category are provided in Table \ref{tab:criteria}.

\begin{table}
  \centering
    \begin{tabularx}{\linewidth}{|>{\centering\arraybackslash}X|>{\centering\arraybackslash}X|}
      \hline
      \multicolumn{1}{|c}{\textbf{Question Category}} & \multicolumn{1}{c|}{\textbf{Description}} \\
      \hline
Good question. Answer is present in the text. & The answer to the generated question exists in the given sentence/paragraph.\\
\hline
Incorrect question but with relevant words from the input. & The generated question contains some words/phrases from the input, but the question is not grammatically correct and/or does not make sense.\\
\hline
Question and answer are mixed & The generated question contains some part of the
answer.\\
\hline
Grammatical mistake & The generated question is grammatically incorrect.\\
\hline
Grammatically correct but the answer doesn't exist & The generated question is grammatically correct, but the answer to the question does not exist in the input context.\\
\hline
Completely vague  & The generated question is not meaningful, too vague.\\
\hline
Two valid questions are mixed & The generated question contains two questions from different parts of the input. \\
\hline
 \end{tabularx}
  \caption{Description of different category set for question evaluation}
  \label{tab:criteria}
\end{table}


\iffalse
- questions from patterns
- generated questions after learning
- human evaluation 
- error analysis good/bad examples
\fi

\end{document}
