Task weighting, which assigns weights on the including tasks during training, significantly matters the performance of Multi-task Learning (MTL); thus, recently, there has been an explosive interest in it. However, existing task weighting methods assign weights only based on the training loss, while ignoring the gap between the training loss and generalization loss. It degenerates MTL's performance. To address this issue, the present paper proposes a novel task weighting algorithm, which automatically weights the tasks via a learning-to-learn paradigm, referred to as MetaWeighting.
Start the training process by running train.py.
You can change the model and Multi-task learning method's config by modify config.py.
models provides multiple model implementations for selection.
Results generated by code are saved in runs (will be automatically generated after training).
You'll need a working Python environment to run the code. The code is based on python 3.6 and pytorch 1.6.
The recommended way to set up your environment is through the Anaconda Python distribution which provides the conda package manager.
The required dependencies are specified in the file requirements.txt.
Run the following command in the repository folder (where requirements.txt is located) to create a separate environment and install all required dependencies in it:
conda create -n env_name python=3.6 # create new environmentsource activate env_namepip install -r requirements.txtBefore running any code you must activate the conda environment:
xxxxxxxxxxsource activate env_name
or, if you're on Windows:
xxxxxxxxxxactivate env_name
This will enable the environment for your current terminal session.
| Parameter | Default & Other Choice | Description |
|---|---|---|
| -d | sentiment [20news] | Datasets of Sentiment Analysis and Topic Classification |
| -m | textcnn [lstm] | Model of feature extractor |
| -t | meta [single/uniform/fudan/mgda/gradnorm /uncertain/tchebycheff_adv/bandit] | Multi-task learning method |
| -g | 0 | Select when there are multiple GPUs |
| -c | None | Remark the result |
| -alpha | 0.1 | Step size for updating weights |
| -split_scale | 0.1 | Query-Split radio |
Run meta on GPU 0 with dataset Sentiment Analysis, alpha=0.1, rho=0.1
xxxxxxxxxxpython train.py
Run meta on GPU 2 with dataset Topic Classification, alpha=0.5, rho=0.1
xxxxxxxxxxpython train.py -d 20news -g 2 -alpha 0.5 -split 0.1
The output will be placed in runs/20news/meta. Use Tensorboard to check the result:
xxxxxxxxxxtensorboard --logdir=RESULT_PATH --port SERVER_PORT --bind_all
This will start the server and open your default web browser to the Tensorboard interface. You can check and download the result in the page.