summarization task 0.001743423
human summaries 0.001718378
human quality 0.001696322
agreement score 0.0016387100000000003
ranking system 0.0016215140000000001
coverage score 0.001542505
standard score 0.001463911
relevant information 0.001452729
human judges 0.0014451400000000001
automatic summarization 0.001368573
information sources 0.001368455
information retrieval 0.001365141
summarization systems 0.001344012
prevalence score 0.001343561
irrelevant information 0.001336847
tant information 0.001326345
human performance 0.001326281
human judge 0.001323241
document quality 0.0013064980000000001
summarization aspects 0.0012927630000000002
system performance 0.001281211
baseline system 0.001252387
coverage quality 0.0012206370000000001
different subjects 0.00121607
different scenarios 0.001208946
agreement scores 0.001203264
document summary 0.001200888
sentence flow 0.0011976539999999998
ranking algorithm 0.001194614
extractive summarization 0.001181304
automated system 0.001153914
summarization literature 0.001151316
important aspect 0.001149439
important decision 0.001149053
high quality 0.001144804
ranking intelligence 0.001139211
input document 0.001126655
important criteria 0.00110647
quality assessment 0.001102705
important factors 0.001101592
same scenario 0.00110111
relative quality 0.001093406
important themes 0.001091995
important factor 0.0010827509999999999
information 0.00108081
challenging task 0.00107981
plex task 0.00107981
score 0.00107867
important role 0.001070998
important element 0.0010653099999999999
important topics 0.001062166
random ranking 0.00105834
quality criteria 0.001058091
quality report 0.001055082
important skill 0.001051327
same subject 0.001048364
overall agreement 0.001044578
quality judgments 0.001033337
same way 0.001025333
such criteria 0.001017553
ment quality 0.0010081320000000001
individual agreement 0.001005436
quality look 0.00100048
query subject 9.81538E-4
topic models 9.625009999999999E-4
coverage rank 9.60269E-4
average agreement 9.55594E-4
sentence 9.51436E-4
individual judges 9.510160000000001E-4
human 9.3952E-4
words judges 9.37865E-4
good document 9.3761E-4
other analysts 9.35935E-4
tion systems 9.26862E-4
first step 9.23269E-4
summarization 9.0837E-4
first attempt 9.026279999999999E-4
several reports 9.02138E-4
document collection 9.020359999999999E-4
system 8.9445E-4
rank correlation 8.69674E-4
judges concepts 8.6732E-4
several criteria 8.62661E-4
previous research 8.5808E-4
many reports 8.483220000000001E-4
weighted sum 8.36162E-4
individual judge 8.29117E-4
several areas 8.170230000000001E-4
specific topic 8.10166E-4
several collections 8.09241E-4
perfect agreement 8.065570000000001E-4
automatic assessment 8.06106E-4
discordant pairs 8.060249999999999E-4
results figure 7.932550000000001E-4
computational methods 7.91753E-4
content selection 7.89278E-4
individual reports 7.86162E-4
many times 7.85794E-4
summaries 7.78858E-4
future work 7.76964E-4
