1. Reproducing the experiments
--------------------

The experiments described in the paper are in the notebok reproduce_experiments.ipynb. 
You can either run the notebook, or open the static html file (reproduce_experiments.html) 
to directly see the results without rerunning. 

Dependencies:
	numpy, matplotlib, seaborn, pandas, stanza

If you miss any of these dependencies, please run any of the following options:

1) run the following command: pip install -r requirements.txt

2) run install.sh

3) run either of the following commands on your terminal for installing the missing dependencies 
	(or run all in case all of them are missing):

	pip install numpy
	python -m pip install -U pip
	python -m pip install -U matplotlib
	pip install seaborn
	pip install pandas
	pip install stanza


2. Estimated time and running
-----------------------------
The reproducing of the experiments can be run on any average laptop. 
The estimated time is around 10 mins.


3. Datasets
-----------

3.1 BenchIE:
========
1. data/gold/benchie_gold_annotations_en_ent_with_sent_id.txt: golden annotations having readable 
	format, whereas the optional tokens are marked with brackets [] (See Table 1 from paper for 
	explanation)
2. daga/gold/benchie_gold_annotations_zh_with_sent_id.txt: same as 
	benchie_gold_annotations_en_ent_with_sent_id.txt, only manually translated on Chinese
3. data/gold/benchie_gold_annotations_de_with_sent_id.txt: same as 
	benchie_gold_annotations_en_ent_with_sent_id.txt, only manually translated on German
4. data/gold/benchie_gold_annotations_en_ent_with_sent_id.txt: similar as 
	benchie_gold_annotations_en_ent_with_sent_id.txt, only adjusted for BenchIE-E. See details
	in Section 3, paragraph "BenchIE-E."

3.2 CaRB:
============
5. data/gold/carb_explicit.txt: all explicit golden annotations from CaRB for the 300 sentences 
	(see paper for details)

3.3 OIE Systems:
============
Explicit OIE extractions by several OIE systems:

6. data/oie_systems_explicit_extractions/clausie_explicit.txt: ClausIE
7. data/oie_systems_explicit_extractions/m2oie_de_explicit.txt: M^2OIE for German language
8. data/oie_systems_explicit_extractions/m2oie_en_explicit.txt: M^2OIE for English language
9. data/oie_systems_explicit_extractions/m2oie_zh_explicit.txt: M^2OIE for Chinese language
10. data/oie_systems_explicit_extractions/minie_explicit.txt: MinIE
11. data/oie_systems_explicit_extractions/openie6_explicit.txt: OpenIE 6
12. data/oie_systems_explicit_extractions/roi_explicit.txt: ROI
13. data/oie_systems_explicit_extractions/stanford_explicit.txt: Stanford OIE

3.4 Input sentences:
================
14. sample300_en.txt: random sample of 300 sentences (see paper for details)
15. sample300_zh.txt: same random sample of 300 sentences as sample300.txt, only manually 
	translated on Chinese
16. sample300_de.txt: same random sample of 300 sentences as sample300.txt, only manually 
	translated on German
