Experiences with Parallelisation of an Existing NLP Pipeline: Tagging Hansard

Stephen Wattam, Paul Rayson, Marc Alexander, Jean Anderson

Abstract
This poster describes experiences processing the two-billion-word Hansard corpus using a fairly standard NLP pipeline on a high performance cluster. Herein we report how we were able to parallelise and apply a traditional single-threaded batch-oriented application to a platform that differs greatly from that for which it was originally designed. We start by discussing the tagging toolchain, its specific requirements and properties, and its performance characteristics. This is contrasted with a description of the cluster on which it was to run, and specific limitations are discussed such as the overhead of using SAN-based storage. We then go on to discuss the nature of the Hansard corpus, and describe which properties of this corpus in particular prove challenging for use on the system architecture used. The solution for tagging the corpus is then described, along with performance comparisons against a naive run on commodity hardware. We discuss the gains and benefits of using high-performance machinery rather than relatively cheap commodity hardware. Our poster provides a valuable scenario for large scale NLP pipelines and lessons learnt from the experience.
Anthology ID:
L14-1539
Volume:
Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)
Month:
May
Year:
2014
Address:
Reykjavik, Iceland
Editors:
Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
4093–4096
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2014/pdf/687_Paper.pdf
DOI:
Bibkey:
Cite (ACL):
Stephen Wattam, Paul Rayson, Marc Alexander, and Jean Anderson. 2014. Experiences with Parallelisation of an Existing NLP Pipeline: Tagging Hansard. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 4093–4096, Reykjavik, Iceland. European Language Resources Association (ELRA).
Cite (Informal):
Experiences with Parallelisation of an Existing NLP Pipeline: Tagging Hansard (Wattam et al., LREC 2014)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2014/pdf/687_Paper.pdf
PDF Search