different machine 0.0015424640000000001
word sense 0.001533965
different turkers 0.001400624
standard data 0.001307116
word simi 0.001216514
human evaluation 0.001202124
test set 0.001173678
small test 0.001159247
other candidates 0.001094253
same size 0.001069957
other choices 0.001024741
manual evaluation 9.877599999999999E-4
same source 9.83811E-4
translation task 9.72002E-4
relation coefficient 9.654889999999999E-4
statistical machine 9.568900000000001E-4
machine translation 8.98976E-4
human judgments 8.96541E-4
news translation 8.946889999999999E-4
human intelligence 8.53939E-4
translation quality 8.498259999999999E-4
reference translation 8.4814E-4
ation results 8.460709999999999E-4
natural language 8.455299999999999E-4
language processing 8.42291E-4
automatic metrics 8.316350000000001E-4
suitable method 8.27744E-4
human opponents 8.17233E-4
human translator 8.17233E-4
sense disambiguation 8.1365E-4
source language 8.01905E-4
sufficient accuracy 8.00084E-4
translation systems 7.86453E-4
small amounts 7.858159999999999E-4
source sentence 7.85757E-4
large number 7.82426E-4
lar set 7.77223E-4
small sums 7.67736E-4
high quality 7.67486E-4
tion task 7.600129999999999E-4
translation edit 7.57578E-4
turk web 7.5223E-4
chine translation 7.46096E-4
erence translation 7.46096E-4
related work 7.4527E-4
majority label 7.448299999999999E-4
adequacy scores 7.44415E-4
quality rank 7.42974E-4
particular expertise 7.17167E-4
work snow 7.1415E-4
tag line 7.12126E-4
web interface 6.936080000000001E-4
relation 6.89375E-4
experts number 6.891169999999999E-4
web site 6.8342E-4
web ser 6.8342E-4
agreement figure 6.795939999999999E-4
quality reference 6.78262E-4
multiple reference 6.76666E-4
single judgment 6.70711E-4
expert agreement 6.67813E-4
single expert 6.63195E-4
line agreement 6.60751E-4
complete ranking 6.54829E-4
manual evaluations 6.54314E-4
level ranking 6.513370000000001E-4
machine translations 6.458670000000001E-4
standard judgments 6.43817E-4
simple majority 6.38747E-4
tion quality 6.37837E-4
grand total 6.33909E-4
machine transla 6.33231E-4
machine trans 6.284120000000001E-4
expert judgments 6.27006E-4
nlp tasks 6.250069999999999E-4
ing comprehension 6.23907E-4
evaluation 6.23371E-4
processing tasks 6.231139999999999E-4
mechanical turk 6.161750000000001E-4
complex tasks 6.12338E-4
chance agreement 6.08592E-4
inexpensive way 6.060659999999999E-4
expert annotators 6.00246E-4
various types 5.975539999999999E-4
hension experiments 5.97533E-4
top line 5.96794E-4
reference translations 5.95031E-4
quality labels 5.90921E-4
multiple turkers 5.85662E-4
online labor 5.84244E-4
public forums 5.83325E-4
public eval 5.83325E-4
reading compre 5.83039E-4
majority vote 5.82646E-4
search query 5.79339E-4
human 5.78753E-4
sure quality 5.76942E-4
unweighted weighted 5.74774E-4
relative rankings 5.738360000000001E-4
intelligence tests 5.71683E-4
