This file explains how to use the code that has been submitted with the paper.

There are five files besides this README, that are part of the submission:

SECTION 0: REQUIREMENTS
-----------------------
To run the files included with this submission, the following are required:
 - Python v2.7 or higher (download and install from http://www.python.org)
 - NLTK v2.0b8 or higher (download and install from http://www.nltk.org)
 - R v2.11 or higher (download and install from http://www.r-project.org/)


SECTION 1: WHAT FILES ARE INCLUDED
----------------------------------

1. compute_agreement_and_bins.py: This is a Python script that takes an input CSV
   file (format described in next section) and computes the majority Turker agreement 
   for each instance and also divides the input file records into three agreement bins:
     (a) Bin 1: All instances where the Turker agreement with the majority rating is greater
         than or equal to 50% (the lowest possible with a binary class) or less than 75%.
         This bin contains the instances of high agreement (clear-cut)
     (b) Bin 2: All instances where the Turker agreement with the majority rating is greater
         than or equal to 75% (the lowest possible with a binary class) or less than 90%.
         This bin contains instances of medium agreement.
     (c) Bin 2: All instances where the Turker agreement with the majority rating is greater
         than or equal to 90% (the lowest possible with a binary class) or less than equal to
         100%. This bin contains instances of low agreement (contentious)

    This script will produce the following four output files:
      (i) 'agreements': a file containing the Turker agreement with the majority rating for each
           instance in the input file. This file has the same number of lines as the input file.
     (ii) 'bin1-50-75.csv': a file containing the input records that belong to Bin 1 (as defined above)
    (iii) 'bin2-75-90.csv': a file containing the input records that belong to Bin 1 (as defined above)     
     (iv) 'bin3-90-100.csv': a file containing the input records that belong to Bin 1 (as defined above)
          
2. compute_precision_recall.py: This is a Python script that can compute both unweighted/weighted precision and recall
   for the systems included in the input file against both expert as well as crowd judgments. The input
   is a CSV file in the format as described in the next section. The output is the precision and recall
   for one or both systems present in the input file. The following command line switches are available:
   
     -c : compute against the crowd judgments (default is to compute against expert judgments)
     -w : compute weighted precision/recall (default is to compute unweighted measures)
     -s : which systems to compute the precision/recall for (possible values "1", "2" or "both")

3. compute_kappa.py: This Python script is very similar to the above precision recall computation script
   except that instead of computing weighted/unweighted precision/recall, it computes Cohen's Kappa
   against the majority rating (crowd or experts). It has the following command line switches:

     -c : compute against the crowd judgments (default is to compute against expert judgments)
     -s : which systems to compute kappa for (possible values "1", "2" or "both")
        
4. plot_agreements.r: This is an R script that can generate the histogram of agreements as shown in Figure 1.
   It takes the 'agreements' file generated by 'compute_agreement_and_bins.py' as the only argument.
   To run this script, do 'Rscript plot_agreements.r <path to agreements file>'. The output will be a pdf
   file called 'agreement-histogram.pdf' that contains the histogram.

5. plot_precision_recall_bins.r: This is an R script that generates the precision/recall-agreement
   curves as shown in the paper in Figure 2. Currently, the values being plotted, the labels and the
   legend is hard coded, so to use this script for a different set of systems and a different data
   (with differently sized bins), you will need to modify the script. To run this script, simply do
   'Rscript plot_precision_recall_bins.r'. The output will be a pdf file called 'prec-recall-agreement.pdf'
   that contains the plot. If you need to generate kappa-agreement plots insead of precision/recall-agreement
   plots, then simply modify the four lines in this file to plot the kappas instead of precision/recall.

SECTION 2: FORMAT OF INPUT CSV FILE
-----------------------------------
The input CSV file needs to be in a specific format. It is a CSV file where each row
contains a preposition instance that has been judged by experts and Turkers and each
row contains the following fields in order: 

- "unitid": a unique instance ID. 
- "prep": the preposition being judged
- "sentence": the sentence containing the preposition
- "preplocation": the location of the preposition in the sentence (word number, starting from 1)
- "sys1pred": the prediction of system 1 for this instance ('1' represents ERROR, '3' represents OK)
- "sys2pred": the prediction of system 2 for this instance
- "internal": the internal or expert judgments for this instance, separated by "|". 
   For example, with three experts this can be the string "3|3|1"
- "crowd": the external or crowd/Turker judgments for this instance, separated by "|"

Note that if you are not planning to use expert judgments at all and only want to use crowd judgments,
you may put an empty string in the "internal" field. If you only have a single system predicion available 
and want to compute the precision and recall for just that system rather than both systems, you should
use the appropriate value for the '-s' switch for 'compute_precision_recall.py' above depending on which field in 
the file you have populated with this system's predictions.

Note also that the bin*.* CSV file above are simply subsets of the original input file and, therefore,
can be used as input files for the 'compute_precision_recall.py' script. In fact, the unweighted precision and 
recall values that are used to plot the precision-agreement curve in Figure 2 were obtained using
'compute_precision_recall.py' with the respective bin*.* CSV files as input files.
