@inproceedings{gibson-etal-2008-identification,
    title = "Identification of Duplicate News Stories in Web Pages",
    author = "Gibson, John  and
      Wellner, Ben  and
      Lubar, Susan",
    editor = "Evert, Stefan  and
      Kilgarriff, Adam  and
      Sharoff, Serge",
    booktitle = "Proceedings of the 4th Web as Corpus Workshop",
    month = jun,
    year = "2008",
    address = "Marrakech, Morocco",
    publisher = "European Language Resources Association",
    url = "https://preview.aclanthology.org/fix-sig-urls/2008.wac-1.5/",
    pages = "26--33",
    abstract = "Identifying near duplicate documents is a challenge often faced in the field of information discovery. Unfortunately many algorithms that find near duplicate pairs of plain text documents perform poorly when used on web pages, where metadata, other extraneous information make that process much more difficult. If the content of the page (e.g., the body of a news article) can be extracted from the page, then the accuracy of the duplicate detection algorithms is greatly increased. Using machine learning techniques to identify the content portion of web pages, we achieve duplicate detection accuracy that is nearly identical to plain text, significantly better than simple heuristic approaches to content extraction. We performed these experiments on a small, but fully annotated corpus."
}