# Divide and Rule: Effective Pre-Training for Context-Aware Multi-Encoder Translation Models

This is our implementation for the experiments of the *Divide and Rule* paper. 

The code for the models is based on [fairseq](https://github.com/pytorch/fairseq), while the scripts for preparing and splitting the data are based on bash and python, respectively.

# Requirements and Installation

[fairseq](https://github.com/pytorch/fairseq) requires the following:

* [PyTorch](http://pytorch.org/) version >= 1.4.0
* Python version >= 3.6
* For training models, you'll also need an NVIDIA GPU and [NCCL](https://github.com/NVIDIA/nccl)

To install and develop locally:
```bash
cd fairseq # this fork!
pip install --editable ./
```

# Replicate results

The steps to reproduce our _K1-d&r_ results on IWSLT En-De are enumerated below:

1. Prepare WMT17 dataset `bash sh/prepare/en-de/wmt17/sent_all.sh`

2. Train K0 on WMT17 `bash sh/run/en-de/wmt17/transfo_base.sh --t=train --sdir=transfo_base` and average 10 best checkpoints `bash sh/run/en-de/wmt17/transfo_base.sh --t=average --sdir=transfo_base`

3. Download IWSLT7 [from WIT3](https://wit3.fbk.eu/2017-01) and unzip it in `./data/en-de/orig/iwslt17`

4. Prepare IWSLT17 `bash sh/prepare/en-de/iwslt17/doc_all.sh standard`

5. Finetune K0 on IWSLT17 `bash sh/run/en-de/iwslt17/transfo_base.sh --t=finetune --sdir=standard/k0 --pretrained=checkpoints/en-de/wmt17/transfo_base/checkpoint.avg_last10.pt`

6. Prepare IWSLT17split `bash sh/prepare/en-de/iwslt17/doc_all.sh split`

7. Train K1 on IWSLT17split `bash sh/run/en-de/iwslt17/han.sh --t=train --cuda=0 --k=1 --sdir=split/k1 --pretrained=checkpoints/en-de/iwslt17/standard/k0/checkpoint_best.pt --datadir=data/en-de/data-bin/iwslt17/split`

8. Finetune K1 on the original IWSLT17 `bash sh/run/en-de/iwslt17/han.sh --t=finetune --cuda=0 --k=1 --sdir=fromsplit/k1 --pretrained=checkpoints/en-de/iwslt17/split/k1/checkpoint_best.pt`

9. Prepare ContraPro `bash sh/prepare/en-de/wmt17/large_pronoun.sh --k=3 --shuffle=False`

10. Evaluate K1 `bash sh/run/en-de/iwslt17/han.sh --t=test-suites --cuda=0 --k=1 --sdir=fromsplit/k1`

11. Display results `bash sh/run/en-de/iwslt17/han.sh --t=results --k=1 --sdir=fromsplit/k1`

The other results presented in the paper can be reproduced with analogous scripts, that can be found in `./sh` .

## Data

Data can be found at these links (most of them are already provided in the scripts):

1. WMT14 [here](http://statmt.org/wmt14/translation-task.html)
2. WMT17 [here](http://statmt.org/wmt17/translation-task.html)
3. IWSLT7 [from WIT3](https://wit3.fbk.eu/2017-01)
4. OpenSubtitles2018 by [Voita et al. (2019)](https://www.aclweb.org/anthology/P19-1116/), available [here](https://github.com/lena-voita/good-translation-wrong-in-context#training-data).


