The purpose of this task is to identify evidence from a Wikipedia page that can be used to support or refute simple factoid sentences called claims. The claims are generated by humans (as part of the WF1 annotation workflow) from the Wikipedia page about {{entity}}. Some claims are true. Some claims are fake. You must find the evidence from the page that supports or refutes the claim.
Other Wikipedia pages will also provide additional information that can serves as evidence. For each line, we will provide extracts from the linked pages in the dictionary column which appear when you "Expand" the sentence. The sentences from these linked pages that contain relevant supplimentary information should be individually selected to record which information is used in justifying your decisions.
Step-by-step guide:
- Read and understand the claim
- Read the Wikipedia page and identify sentences that contain relevant information.
- On identifying a relevant sentence, press the Expand button to highlight it. This will load the dictionary and the buttons to annotate it:
- If the highlighted sentence contains enough information in a definitive statement to support or refute the claim, press the Supports or Refutes button to add your annotation. No information from the dictionary is needed in this case (this includes information from the main Wikipedia page). Then continue annotating from step 2.
- If the highlighted sentence contains some information supporting or refuting the claim but also needs supporting information, this can be added from the dictionary.
- The hyperlinked sentences from the passage are automatically added to the dictionary
- If a sentence from the main Wikipedia article is needed to provide supporting information. Click “Add Main Wikipedia Page” to add it to the dictionary.
NB: if multiple sentences from the main Wikipedia page are selected, you don’t have to repeat the annotation for all the sentences as this will result in duplicates. Attempting to do so will result will result in this warning (that can be dismissed if new information will be added):
This sentence has already been selected as part of another annotation that uses the original page. Unless you intend to add new information, continuing will result in a duplicate annotation.
- If the claim or sentence contains an entity that is not in the dictionary, then a custom page can be added by clicking “Add Custom Page”. Use a search engine of your choice to find the page and then paste the Wikipedia URL into the box.
- Tick the sentences from the dictionary that provide the minimal amount of supporting information needed to form your decision. If there are multiple equally relevant entries (such as a list of movies), then just select the first.
- Once all required information is added, then press the Supports or Refutes button to add your annotation and continue from step 2.
- If the highlighted sentence and the dictionary do not contain enough information to support or refute the claim, press the Cancel button and continue from step 2 to identify more relevant sentences.
- On reaching the end of the Wikipedia page. Press Submit if you could find information that supports or refutes the claim. If you could not find any supporting evidence, press Skip then select Not enough information
What does it mean to Support or Refute
The objective is to find sentences that support or refute the claim.
You must apply common-sense reasoning to the evidence you read but avoid applying your own world-knowledge by basing your decisions on the information presented in the Wikipedia page and dictionary.
As a guide - you should ask yourself:
If I was given only the selected sentences, do I have stronger reason to believe claim is true (supported) or stronger reason to believe the claim is false (refuted). If I'm not certain, what additional information (dictionary) do I have to add to reach this conclusion.
The following count as valid justifications for marking an item as supported/refuted:
- Sentence directly states information that supports/refutes the claim or states information that is synonymous/antonymous with information in the claim
Claim: Water occurs artificially
Refuted by: ``It also occurs in nature as snow, glaciers ...''
Claim: Samuel L. Jackson was in the third movie in the Die Hard film series.
Supported by: ``He is a highly prolific actor, having appeared in over 100 films, including Die Hard 3.''
- Sentence refutes the claim through negation or quantification
Claim: Schindler's List received no awards.
Refuted by: ``It was the recipient of seven Academy Awards (out of twelve nominations), including Best Picture, Best Director...''
- Sentence provides information about a different entity and only one entity is permitted (e.g. place of birth can only be one place)
Claim: David Schwimmer finished acting in Friends in 2005.
Refuted by: ``After the series finale of Friends in 2004, Schwimmer was cast as the title character in the 2005 drama Duane Hopwood.''
- Sentence provides information that, in conjunction with other sentences from the dictionary, fulfils one of the above criteria
Claim: John McCain is a conservative.
Refuted by: ``He was the Republican nominee for the 2008 U.S. presidential election.'' AND ``The Republican Party's current ideology is American conservatism, which contrasts with the Democrats' more progressive platform (also called modern liberalism).''
Adding Custom Pages
You may need to add a custom page from Wikipedia to the dictionary. This may happen in cases where the claim discusses an entity that was not in the original Wikipedia page
Claim: Colin Firth is a Gemini.
In Original Page: ``Colin Firth (born 10 September 1960)... ''
Requires Additional Information from Gemini: ``Under the tropical zodiac, the sun transits this sign between May 21 and June 21.''
Tense
The difference in verb tenses that do not affect the meaning should be ignored.
Claim: Frank Sinatra is a musician
Supported: He is one of the best-selling music artists of all time, having sold more than 150 million records worldwide.
Claim: Frank Sinatra is a musician
Supported: Francis Albert Sinatra (/sɪˈnɑːtrə/; Italian: [siˈnaːtra]; December 12, 1915 – May 14, 1998) was an American singer
Skipping
There may be times where it is appropriate to skip the claim:
- The claim cannot be verified using the information with the information provided:
- If the claim could potentially be verified using other publicly available information. Select Not Enough Information
- If the claim can't be verified using any publicly available information (because it's ambiguous, vague, personal or implausible) select The claim is ambiguous or contains personal information
NB:Note that claim can be ambiguous even if the exact sentence can be found in Wikipedia (e.g. "The album achieved widespread popularity in America").
- The claim contains typographical errors, spelling mistakes, is ungrammatical or could be fixed with a very minor change
Select The claim has a typo or grammatical error
Flagging
You can submit the claim but flag it for further discussion. This could be because:
- The claim might contain potentially brand-damaging information (but remember that mutated claims are never going to be presented to customers as facts)
- The claim fall between two slightly conflicting guidelines