
# SkipBERT

Code associated with EMNLP 2021 submission


## General Distillation

The code is available at "./general_distillation/".
We implement with DeepSpeed framework for perform distributed training.
We use 8 RTX A6000.

The example bash scripts can be found at "./general_distillation/run_train.sh" and "./general_distillation/run_train-m.sh".

## Task-Specific Distillation

The code is available at "./task_distillation/skipbert".

Before performing task-specific distillation, we shall first:
- Obtain the generally distilled models
- Download the GLUE datasets
- Fine-tune the teacher models

The example bash scripts can be found at "./task_distillation/skipbert/run_distil.sh" and "./task_distillation/skipbert/run_distil-m.sh".
 