# Pruning adapter with lottery ticket

1. **Environment Set up:** Environment can be set up using `pip` or `conda` using the comand line
* For pip:	`pip install -r requirements.txt`
* For conda: `conda env create -f environment.yml`

2. **Task setting:** Tasks of GLUE to be trained are listed in file *tasks.txt*, feel free to change it.
3. **Run:** To perform weight pruning on the adapter model with size 128. You can run the following script:
    ```bash
    python main.py \
      --target layer \
      --adapter_size 128 \
      --prune_persent 20 \
      --model_save_dir test \
      --tasks_file tasks.txt \
      --lr 5e-4 \
      --seed 36 \
    ```
    Pruning target can be `base` `weight`, `neuron` or `layer`. (`base` means training adapter without pruning)
4. Models are saved in the directory `model_save_dir` specified. To save space, only the parameters of adapters and classifier are saved in the directory. They are saved in adapter.pkl in every checkpoint directory. The mask of the model is saved as mask.pkl. The file structure of save dir is as follow.
```
model_save_dir
│
└───{task}
│   │
│   └───{lr}-{batchsize}
│       │   init_adapter_clsfier.pkl [Initial weight of model before training]
│       │───Iter0
│       │    │mask.pkl              [Mask used in adapter training]
│       │    │remain_persent.txt    [Remain percentage of parameters]
│       │    │───checkpoint-xxx
│       │    │    |adapter.pkl      [Parameters of adapter and classifier]
│       │    │    |atokenizer_config.json   [Eval result during training]
│       │    │    |...
│       │    │checkpoint-...
│       │───Iter1...
```
5. Code tree structure and the usage of different files
```
Code
│   README.md
│   __init__.py
│   tasks.txt   [Tasks to be run]
│   main.py     [Main script]
└───util
    │   training_glue.py    [Data preprosess & Training model setup]
    │   ttrainer.py         [Trainer for all three pruning methods]
    │   glue.py             
    │   prune.py            [Function for Three pruning methods]
    │   adapter_model.py    [Define of adapter model]
```
6. To load adapter model using adapter.pkl, num_labels is set according to the task.
```python
from util.adapter_model import Adapter, AdapterModel
from trainer import *
model_path = "bert-base-uncased"
adapter_size = 128
model = AdapterModel(model_path, num_labels, adapter_size) 
loadAdapterWithClassifier('adapter.pkl',model)
```

7. Estimate time for runing the model

Adapter Pruning (Maximum model size 141M)
| Tasks | Round | Time                    |
|-------|-------|-------------------------|
| CoLA  | 12    | 0 day 0 h 39 min 24 sec |
| SST2  | 12    | 0 day 5 h 13 min 12 sec |
| MRPC  | 12    | 0 day 0 h 28 min 12 sec |
| STSB  | 12    | 0 day 0 h 26 min 36 sec |
| QQP   | 12    | 1 day 4 h 29 min 12 sec |
| MNLI  | 12    | 1 day 7 h 04 min 36 sec |
| QNLI  | 12    | 0 day 8 h 11 min 24 sec |
| RTE   | 12    | 0 day 0 h 11 min 24 sec |
| Total |       | 3 day 2 h 44 min 00 sec |


