John Gibson


2008

pdf bib
Identification of Duplicate News Stories in Web Pages
John Gibson | Ben Wellner | Susan Lubar
Proceedings of the 4th Web as Corpus Workshop

Identifying near duplicate documents is a challenge often faced in the field of information discovery. Unfortunately many algorithms that find near duplicate pairs of plain text documents perform poorly when used on web pages, where metadata, other extraneous information make that process much more difficult. If the content of the page (e.g., the body of a news article) can be extracted from the page, then the accuracy of the duplicate detection algorithms is greatly increased. Using machine learning techniques to identify the content portion of web pages, we achieve duplicate detection accuracy that is nearly identical to plain text, significantly better than simple heuristic approaches to content extraction. We performed these experiments on a small, but fully annotated corpus.

2006

pdf bib
Maytag: A Multi-Staged Approach to Identifying Complex Events in Textual Data
Conrad Chang | Lisa Ferro | John Gibson | Janet Hitzeman | Suzi Lubar | Justin Palmer | Sean Munson | Marc Vilain | Benjamin Wellner
Demonstrations