
English
=======

Folder structure:

- `gold_and_pred_data`: contains gold ptb dev set and dev predictions for 3 English models from paper.
- `test_suite`: contains annotations for the test suite described in Section 3
- `disco_berkeley_parser_analyser`: contains the code for the extension of the Berkeley parser analyser to discontinuous structures.
- `results`: expected results when running the eval bash script.
    - `{bert,fast,supe}_disco_errors`: eval with test suite for ml-gap+bert, ml-gap+ft, ml-gap
    - `bpa_{bert,fast,supe}/full.error_counts`: error classifications

To run the evaluation on English:

```bash
bash run_eval.sh
```

Dependencies: 

- `discodop` available from command line
- python (3.8) with nltk
- python2 (2.7)


The essential parts of `disco_berkeley_parser_analyser` are taken from the Berkeley parser analyser,
so if you use `disco_berkeley_parser_analyser`, please make sure you also cite the original article:

```bash
Parser Showdown at the Wall Street Corral: An Empirical Investigation of Error Types in Parser Output
Jonathan K. Kummerfeld, David Hall, James R. Curran, Dan Klein
EMNLP 2021
https://www.aclweb.org/anthology/D12-1096/
```

German
======

This archive also contains:

- the predictions of the German models on sentences from [discosuite](https://www.phil-fak.uni-duesseldorf.de/beyond-cfg/resources/discosuite/) (Maier et al 2014),
- an updated version of discosuite,
- scripts to perform evaluation on it.

If you reuse the version of discosuite distributed here, make sure to cite the original article:

```
Discosuite - A parser test suite for German discontinuous structures  
W. Maier, M. Kaeshammer, Peter Baumann, and S. Kübler,  
in Proceedings of the 9th Edition of the Language Resources and Evaluation Conference (LREC'14), Reykjavik, Iceland, 2014.  
https://www.aclweb.org/anthology/L14-1222/
```

Folder structure for `discosuite_results`:

- `discosuite_results/data_german`: gold and pred data for the sentences in discosuite
- `discosuite_results/discosuite_yield_corrected_v1.1.csv`: updated version of discosuite (sentences renumbered, some yields manually corrected)
- `discosuite_results/eval_discosuite.py`: evaluator
- `discosuite_results/tiger_exclude_discosuite_sentences.py`: to remove sentences of discosuite from the tiger training set (to run this one, you'll need the spmrl training set in discbracket format, and discosuite_v1.0)


To run the evaluation on discosuite:

```bash
bash run_eval_discosuite.sh
```
