Lara McConnaughey
2022
Scaling Language Model Size in Cross-Device Federated Learning
Jae Ro
|
Theresa Breiner
|
Lara McConnaughey
|
Mingqing Chen
|
Ananda Suresh
|
Shankar Kumar
|
Rajiv Mathews
Proceedings of the First Workshop on Federated Learning for Natural Language Processing (FL4NLP 2022)
Most studies in cross-device federated learning focus on small models, due to the server-client communication and on-device computation bottlenecks. In this work, we leverage various techniques for mitigating these bottlenecks to train larger language models in cross-device federated learning. With systematic applications of partial model training, quantization, efficient transfer learning, and communication-efficient optimizers, we are able to train a 21M parameter Transformer that achieves the same perplexity as that of a similarly sized LSTM with ∼10× smaller client-to-server communication cost and 11% lower perplexity than smaller LSTMs commonly studied in literature.
2017
The Labeled Segmentation of Printed Books
Lara McConnaughey
|
Jennifer Dai
|
David Bamman
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
We introduce the task of book structure labeling: segmenting and assigning a fixed category (such as Table of Contents, Preface, Index) to the document structure of printed books. We manually annotate the page-level structural categories for a large dataset totaling 294,816 pages in 1,055 books evenly sampled from 1750-1922, and present empirical results comparing the performance of several classes of models. The best-performing model, a bidirectional LSTM with rich features, achieves an overall accuracy of 95.8 and a class-balanced macro F-score of 71.4.
Search
Co-authors
- Jae Ro 1
- Theresa Breiner 1
- Mingqing Chen 1
- Ananda Suresh 1
- Shankar Kumar 1
- show all...