
# On the Effectiveness of Automated Metrics for Text Generation Systems

This is the code used for the EMNLP2022 submission 
"On the Effectiveness of Automated Metrics for Text Generation Systems"

## Setup and Installation

Developed using `python3.9` on Linux

We recommend running everything in a virtual environment:
```
python -m venv venv
source venv/bin/activate
```

Installing dependencies: `pip install -r requirements.txt`

Download wmt21 data: `./bash_scripts/download_wmt21.sh` **Be sure to run the command
in this folder (project root)**. You might have to make the script executable by running 
`chmod +x ./bash_scripts/download_wmt21.sh`

Build the Rust submodule "toers" by running the provided script: `./bash_scripts/build.sh`
**Be sure to run the command in this folder (project root)**. You might have
to make the script executable by running `chmod +x ./bash_scripts/build.sh`

You will need a running rust installation including cargo. The easiest way
to install Rust for your platform is:
```shell
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup update
```
**Note: curling scripts from the internet into a shell is usually a bad idea.
Do so at your own risk.**


## Reading Guide

The main functions used for the paper can be found in `binary/estrs.py`.

Since computing an approximation of Equation 5 in the paper is relatively slow
we implemented it in Rust: `toers/src/main.rs` and call it from Python.
