============================================================================ 
CODI 2024 Reviews for Submission #17
============================================================================ 

Title: Complex question generation using discourse-based data augmentation
Authors: Khushnur Binte Jahangir, Philippe Muller and Chloé Braud


============================================================================
                            REVIEWER #1
============================================================================

---------------------------------------------------------------------------
Reviewer's Scores
---------------------------------------------------------------------------
                   Appropriateness (1-5): 5
      Originality / Innovativeness (1-5): 2
           Soundness / Correctness (1-5): 3
             Meaningful Comparison (1-5): 2
                    Recommendation (1-5): 3
               Reviewer Confidence (1-5): 3

Detailed Comments
---------------------------------------------------------------------------
The paper proposes an approach based on discourse relation annotations to augment a question generation training set to improve question-generation qualities. The idea is simple and interesting, however, the method design seems casual and needs more careful considerations. Simply mixing in discourse annotated data is not a fair comparison in terms all experimental evaluations. The results in Table 3 is not consistent, and more results on more data and models should be included.
---------------------------------------------------------------------------



============================================================================
                            REVIEWER #2
============================================================================

---------------------------------------------------------------------------
Reviewer's Scores
---------------------------------------------------------------------------
                   Appropriateness (1-5): 5
      Originality / Innovativeness (1-5): 4
           Soundness / Correctness (1-5): 3
             Meaningful Comparison (1-5): 4
                    Recommendation (1-5): 3
               Reviewer Confidence (1-5): 2

Detailed Comments
---------------------------------------------------------------------------
This paper reports on experiments comparing models for question generation trained on the SQuAD dataset only, or on the dataset augmented with data from ELI5 and from the PDTB (whose causal relations are used to extract 'Why' questions). Two types of evaluation are used: automatic (using a range of metrics including BLEU 1-4, ROUGE-L, and METEOR) and human. Automatic evaluation doesn't seem to indicate any gain using the augmented datasets, but human evaluation indicates a clear improvement. 

Strengths: 
A. the paper contain lots of useful references to work in the area and to models. 
B. the idea of using PDTB to extract why questions is interesting

Weaknesses:
A. The results with automatic evaluation do not seem  to me to suggest major improvements, and the difference between automatic and human evaluation is not properly explained. 
B. Apart from the idea of using PDTB for extracting questions everything else is pretty standard.
---------------------------------------------------------------------------



============================================================================
                            REVIEWER #3
============================================================================

---------------------------------------------------------------------------
Reviewer's Scores
---------------------------------------------------------------------------
                   Appropriateness (1-5): 5
      Originality / Innovativeness (1-5): 3
           Soundness / Correctness (1-5): 4
             Meaningful Comparison (1-5): 3
                    Recommendation (1-5): 4
               Reviewer Confidence (1-5): 4

Detailed Comments
---------------------------------------------------------------------------
This paper presents a work on discourse-based data augmentation for complex question generation. The basic idea is to utilize the discourse relations in the PDTB2 dataset to generate some template-based questions for data augmentation. The experiments demonstrate the performance difference when adding data augmentation. 

This work shows some interesting findings on data augmentation for question generation. Although, it also left some questions. For example, 

- By reading the paper, it is not clear to me why this work has to focus on "complex" question generation
- Although section 3 gives a detailed description of the datasets used, it is unclear about the data augmentation methods used in this work or any novelty in that aspect.
- Any explanation about why PCSQ method did not produce good results compared to the baseline method?
---------------------------------------------------------------------------

