This directory contains data, annotations and scripts associated with

Bender, E.M., D. Flickinger, S. Oepen and Y. Zhang, 2011, `Parser
Evaluation over Local and Non-Local Deep Dependencies in Large
Corpora', Proceedings of the 2011 Conference on Empirical Methods in
Natural Language Processing.

Data files
----------
testitems.txt: 1000 test items, categorized by each of ten phenomenon
testitems.num.txt: The same 1000 test items, with a simple tokenization and
  token numbering applied.
targets.reconciled.txt: Our reconciled annotations for all 1000 items.  
  Each line in this file contains the following @-separated fields:

  item ID @ phenomenon code @ truth-value @ dep. triple @ phenomenon subtype @

annotation-conventions.txt: Notes on the annotation conventions we applied.

Parser output files
-------------------
candc.out
cjreranking.out
enju.out
mst.out
rasp.out
stanford.out

Note: Parser output from the XLE system could not be distributed
due to license restrictions.

Script
------
depeval.pl: The perl script we used to compare the parser output to the
	gold-standard targets.  Usage notes on depeval.pl:

* There is a -v[0-3] option which allows for varying levels of details
  in the output:
   with -v0, or no -v the output is an evaluated version of the test file
   with -v1, the matching parser dependencies will be printed (lines
             starting with "P: ")
   with -v2, in addition to those with -v1, the matching pattern is
             also printed (lines starting with "M: ")
   with -v3, in addition to those with -v2, the matching regex and
             the unmatched patterns are also printed (lines starting 
             with "R: " and "U: " respectively)
  An example:
  $ perl depeval.pl -v2 candc ../testitems.txt 
../targets.reconciled.txt candc.out

* There is another output mode -csv which prints the outputs into a
  CSV ready to be imported into a spreadsheet program.
  An example:
   $ perl depeval.pl -csv candc ../testitems.txt
../targets.reconciled.txt candc.out >candc.csv

summarize: Invokes depeval.pl for the six systems and prints 
summary results to STDOUT

Other
-----
supplementary_materials.pdf provides further numerical results
underlying the claims made in the paper.

