# Pretraining
## Data
prepare wiki raw data and put in under `data/wiki-cased/wiki.train.raw using this github`
https://github.com/airsplay/vokenization
`wiki.train.raw` is in the format of raw text file.

## Run
`bash ./scripts/xattn_pretrain.bash 0`

change ${txt_model} to run the experiment on different language encoders.

# How to finetune
## Data

The data is downloaded automatically from the script

## Run
`bash ./scripts/x_run_glue_all.sh` for XMODEL on GLUE
`bash ./scripts/distil_run_glue_all.sh` for VGMODEL on GLUE
`bash ./scripts/run_glue_all.sh` for baseline language encoder on GLUE

