# DP-NMT

## 1. Normal Training Example

### 1.1. With WMT16 dataset

```python
python main.py \
  --dataset wmt16 \
  --model google/mt5-small \
  --epochs 16 \
  --batch_size 16 \
  --optimizer Adam \
  --learning_rate 0.001 \
  --gradient_accumulation_steps 1 \
```

### 1.2. With custom dataset

```python
python main.py \
  --dataset data/bsd.py \
  --model google/mt5-small \
  --epochs 16 \
  --batch_size 16 \
  --optimizer Adam \
  --learning_rate 0.001 \
  --gradient_accumulation_steps 1 \
```

### 1.3 To continue training from a checkpoint

Keep the same arguments as in previous training and add the following
`--resume_from_epoch` to specify the epoch checkpoint to resume from and modify the checkpoint path `--model`.

```python

python main.py \
  --dataset wmt16 \
  --model checkpoints/2023_06_23-13_21_46/mt5-small \
  --epochs 16 \
  --batch_size 16 \
  --optimizer Adam \
  --learning_rate 0.001 \
  --gradient_accumulation_steps 1 \
  --resume_from_epoch 16  
```

## 2. Private Training Example

```python
python main.py \
  --dataset wmt16 \
  --model google/mt5-small \
  --epochs 7 \
  --batch_size 16 \
  --optimizer Adam \
  --learning_rate 0.001 \
  --gradient_accumulation_steps 32768 \
  --private True \
  --noise_multiplier 13.18 \
  --warmup_steps 4 \
  --custom_dp_dataloader True \
```

Continue training and custom dataset are the same as in normal training.
To disable Poisson sampling in private training, set `--poisson_sampling False`.

## 3. Evaluation

By default, the evaluation while training without teacher forcing is done on the dev set using SacreBLEU. If the training 
is stopped before the end of the last epoch, the evaluation needs to be done separately. To do so, use the arguments
`--generate True` with `--resume_from_epoch` from the stopped epoch to plot the training and validation loss curves. 
Make sure to point to the correct checkpoint path `--model`.

```python
python main.py \
  --dataset wmt16 \
  --model checkpoints/2023_06_25-23_50_22/mt5-small \
  --resume_from_epoch 25 \
  --generate True \
```

Evaluation on the test set is similar and can be done using the argument `--test True`

```python
python main.py \
  --dataset wmt16 \
  --model checkpoints/2023_06_25-23_50_22/mt5-small \
  --resume_from_epoch 25 \
  --generate True \
  --test True \
```

Name of the output file is `result_final_step_test_set.json` and is saved in the checkpoint folder.

BERTScore can be used instead of SacreBLEU by in `evaluate_output.py` script. E.g.,
    
```python
python evaluate_output.py --data checkpoints/2023_06_25-23_50_22/result_final_step.json
```

Name of the output file is `result_final_step_bertscore.json` and is saved in the checkpoint folder. Similar to the test set.

## 4. Compute Epsilon

To compute the epsilon value, for example:

```python
python compute_epsilons.py
    --dataset wmt16 \
    --lang_pair de-en \
    --batch_size 16 \
    --gradient_accumulation_steps 32768 \
    --device_count 2 \
    --epochs 25
```

If `--noise_multiplier` is not specified, the script will compute the epsilon for a range of noise multipliers.
Otherwise, it will compute the epsilon for the specified noise multiplier.
