 
This repository contains a demonstrative implementation for pooling-based models, e.g., DeepPyramidion
complementing our paper: `Sparsifying Transformer Models with Trainable Representation Pooling`

## Content of directories
- `code_excerpt` : contains two most important files that may be a good reading for a better understanding 
of how Transpooler and Pyramidion works.
- `configs`: contains configuration files that can be used for reproducing results.
- `data` : contains preprocessed data samples, and also script to download and preprocess the full arxiv dataset.
- `results`: contains our system's outputs and scores calculated with rouge scorer on the test-set.
- `rouge`: contains a rouge-scorer library that provides evaluators.
- `spm`: contains our arxiv-based spm models and vocabulary.
- `full_code_packed.zip`: contains our full implementation(not official) in the fairseq library.
 This is a base for reproduction steps described below.
- `README.md`: the reproducing results guide that you are reading now.
- `run_all.sh`: script for preprocessing and setting up training using downloaded data, configuration files and pre-installed fairseq library.
 Assumes all previous steps in data processing were covered.
 
## Information on reproducibility

What can you reproduce with this tutorial:
 1. Training models from scratch using our code.
 2. Evaluate outputs of our models that we provide.
 3. Evaluating pre-trained models on the summarization tasks.
 4. Reproducing our benchmarks from Appendix B (comparison of Successive Halving Top-k Operator with Iterative Top-k).


## Setup
We provide a code implementation that is based on fairseq library. 
We forked from `fairseq 0.9.0` and directly modified some of its files and added a few more.
The library should be unpacked and installed from requirements.
It requires:
* [PyTorch](http://pytorch.org/) version >= 1.2.0
* Python version >= 3.6
* For training new models, you'll also need an NVIDIA GPU and [NCCL](https://github.com/NVIDIA/nccl)

If you are interested only in *reading* the code of the most crucial implementation, we selected few files and put them into the directory `code_excerpt`.
## Data Preparation
Depending on your interests, you may either recreate all the steps in the data processing pipeline 
or see the expected results on a small subset of 100 examples in the directory `data` that we provided. 

The dataset is already released and can be downloaded and preprocessed using our scripts. 
To obtain and preprocess the arxiv dataset:
1. Download original data with `data/download_arxiv.sh`
2. Flatten jsons into text with `data/preprocess_arxiv.py`
3. Go into the main directory and start `run_all.sh`

Based on `raw`, the other dirs will be populated with data with the `../run_all.sh` scripts.

The `bpe` folder will contain the bpe'ed version of the data, an intermediate step. This may be later removed to save space on disk.
The `bin` folder will contain the binary version of the data that will be used for training and validation (~2.7Gb).
## Training
If you run `run_all.sh`, the training may have already started. 

The folder `configs` contain configuration files that describe each model denoted in the original paper or in the ablation studies. 


## Validation
We provide outputs of our best models that can be validated using the rouge metric implementation from `https://github.com/google-research/google-research/tree/master/rouge`.


1. Enter the `reproducing_results` directory
2. Install the package (it is contained within our submission) by typing: 
`
pip install -r rouge/requirements.txt & 
pip install rouge-score
`
3. Recalculate scores of DeepPyramidion:\
```python -m rouge.rouge --target_filepattern=./results/arxiv_out_gold.tsv  --prediction_filepattern=./results/DeepPyramidion_arxiv_out.tsv --output_filename=./results/DeepPyramidion_arxiv_scores.tsv --use_stemmer=true --rouge_types rouge1,rouge2```

4.  Recalculate scores of our deep Blockwise baseline:\
```python -m rouge.rouge --target_filepattern=./results/arxiv_out_gold.tsv  --prediction_filepattern=./results/DeepBlockwise_arxiv_out.tsv --output_filename=./results/DeepBlockwise_arxiv_scores.tsv --use_stemmer=true --rouge_types rouge1,rouge2```
 
5. You can see the scores in the `./DeepPyramidion_arxiv_scores.tsv` file. Here are our results we reproduced (note, that difference up to 0.0001 w.r.t to the original paper may be due to the scores bootstrapping):
 
```
score_type,low,mid,high
rouge1-R,0.444221,0.446998,0.450162
rouge1-P,0.537431,0.540939,0.544231
rouge1-F,0.469132,0.471500,0.474092
rouge2-R,0.186830,0.189277,0.191787
rouge2-P,0.227076,0.230164,0.232991
rouge2-F,0.197556,0.199953,0.202433
```

## Pretrained models 
Pretrained models will be released after the review. 
If you want to access them earlier, we can upload them to any hosting of your choice that guarantees anonymity and allow for upload of over 1.5Gb files.

## Appendix: Successive Halving Top-k Operator
We provide an additional repository for `Successive Halving Top-k Operator` which can be found in `../student_abstract` directory.
Figures can be easily reproduced following it's README. The code was already publicly released.
