# Adapterfusion pruning with lottery ticket hypothesis 

1. Using Lottery ticket hypothesis to prune the adapter model (weight, neuron, layer)
1. Test the importance of adapter connection by LIA criterion
1. Prune adapter layer using LIA criterion
1. (optional) Use knowledge distillation in adapterfusion to enhance target adapter

Codes:
1. convert.py : convert self created adapter file to adapter-transformers supported format
2. test_af.py : Training adapterfusion with adapters load from the directiory (adapters_for_af / adapters_for_af_base). 
    1. Setting task, load dataset and metric
    2. Create new single adapter model OR Create adapterfusion with adapter stored in directory
    3. Training and evaluating model with trainer
    4. Setting LIA function and add forward hooks to model
    5. Runing eval dataset to calculate the mean of each LIA, then print it out
    6. Tensorboard setup for model graph visualization
4. main.py : Entrence of the program.
5. run_test.sh : A bash script for multiple test runs. 
6. utils.py : util functions
7. config.yml : ALL Configurations for the model.


Use of Code:
1. Convert Custom Adapter format to adapter-transformers format
    1. Get adapter weights file adapter.pkl generated from the `LTH` code.
    2. Rename the file and place it in directory `custom_adapter`
    3. Open `convert.py` file, update the path of adapter file, set list of tasks, type of adatper (Pruned or not)
    4. Run `python convert.py`, it will convert the custom adapter weights file to Adapterhub adapter weight file and saved standard adapters in `adapters_for_af_base` and pruned adapters in `adapters_for_af`
1. Setup your settings in config.yml
1. Run `python main.py -c config.yml train` to train the model OR Setup your config in `run_test.sh` and run the script to run multiple times `bash run_test.py`.

Notice:
1. The code uses Weight&Bias to log the result of the model. If you don't want to use wandb, uncomment first code in `test_af.py` which is `os.environ["WANDB_DISABLED"] = "true"`
1. If you use Weight&Bias, the code will create a new project named `test`. AdapterFusion will be set in group `AF`, and AdapterFusion with LTH will be in group `AF-LTH`. You can filter different task of the runs by the parameter `task` in the project.