# Continual Prompt Tuning for Dialog State Tracking

## Requirements

- transformers==4.6.1
- torch==1.8.1
- pytorch-lightning==1.2.5


## Datasets

We do experiments on [Schema-Guided Dataset](https://github.com/google-research-datasets/dstc8-schema-guided-dialogue).
We choose 15 services from the dataset and do most experiments on them.
To prepare dataset, download the dataset and put it in `data/` path.

If you are interested in the pre-processing, please check ```utils/preprocess.py``` and ```utils/dataloader.py```.



## Reproduce results in our paper

Our baseline scripts are contained in `baseline_scripts` and 
our prompt-tuning scripts are in `prompt_scripts`.

To reproduce results in `Table 1`, refer to scripts:


|          Methods         | avg. JGA on 5 seeds |                  Script Path                 |
|:------------------------:|:-------------------:|:--------------------------------------------:|
| Fine-tuning              |                14.3 | `baseline_scripts/run_t5_vanilla.sh`         |
| EWC                      |                13.9 | `baseline_scripts/run_t5_ewc.sh`             |
| Replay                   |                58.6 | `baseline_scripts/run_t5_replay.sh`          |
| AdapterCL(20X)           |                49.8 | `baseline_scripts/run_t5_adapter.sh`         |
| AdapterCL(1X)            |                30.6 | `baseline_scripts/run_t5_small_adapter.sh`   |
| Prompt Tuning            |                48.1 | `prompt_scripts/run_train_pt_randinit.sh`    |
| Continual Prompt Tuning  |                59.5 | `prompt_scripts/run_train_cpt_augq.sh`            |
|   w/memory               |                60.7 | `prompt_scripts/run_train_cpt_w_mem_augq.sh` |
|   w/memory & backward    |                61.2 | `prompt_scripts/run_train_cpt_w_mem_bwd.sh`  |
| Multi-Task Prompt Tuning |                64.0 | `prompt_scripts/run_multiprompt.sh`          |


To reproduce results in `Table 2`, refer to scripts:

| Methods | avg. JGA on 5 seeds |                  Script Path                 |
|:-------:|:-------------------:|:--------------------------------------------:|
| 1       |                29.6 | `prompt_scripts/run_train_noMSR.sh`          |
| 2       |                41.8 | `prompt_scripts/run_train_noMSR_cl.sh`       |
| 3       |                48.1 | `prompt_scripts/run_train_pt_randinit.sh`    |
| 4       |                57.6 | `prompt_scripts/run_train_cpt.sh`            |
| 5       |                59.5 | `prompt_scripts/run_train_cpt_augq.sh`       |
| 6       |                60.4 | `prompt_scripts/run_train_cpt_w_mem.sh`      |
| 7       |                60.7 | `prompt_scripts/run_train_cpt_w_mem_augq.sh` |


To reproduce results in `Table 3`, refer to scripts:

| Initialization | avg. JGA on 5 seeds |                 Script Path                 |
|:--------------:|:-------------------:|:-------------------------------------------:|
| RandomInit     |                48.1 | `prompt_scripts/run_train_pt_randinit.sh`   |
| SelectInit     |                54.5 | `prompt_scripts/run_train_pt_selectinit.sh` |
| CLInit         |                57.6 | `prompt_scripts/run_train_cpt.sh`           |


To reproduce results in `Table 4`, refer to `prompt_scripts/run_train_cpt.sh` and set `--dataset_order=30/1/31/32`.  

To reproduce results in `Figure 3`, refer to `prompt_scripts/run_train_cpt_augq.sh` and set `--model_name_or_path` 
to paths to your `t5-base` or `t5-large` pre-trained weights. We use RTX-2080 GPU with 11GB RAM. 
In our experiments, we choose 4 batch size and 4 accumulation steps for `t5-base`
and 2 batch size, 8 accumulation steps for `t5-large`. We use fp32 training for both of them.


## Evaluation 

For evaluation for non-prompt experiments, refer to `gather_res_baseline.py`

For evaluation for prompt-tuning experiments, refer to `gather_res_prompt.py`

For FWT calculation for non-prompt experiments, refer to `gather_res_baseline_fwt.py`

For FWT calculation for prompt-tuning experiments, refer to `gather_res_prompt_fwt.py`




