Abstract
Neural language models have seen a dramatic increase in size in the last years. While many still advocate that ‘bigger is better’, work in model distillation has shown that the number of parameters used by very large networks is actually more than what is required for state-of-the-art performance. This prompts an obvious question: can we build smaller models from scratch, rather than going through the inefficient process of training at scale and subsequently reducing model size. In this paper, we investigate the behaviour of a biologically inspired algorithm, based on the fruit fly’s olfactory system. This algorithm has shown good performance in the past on the task of learning word embeddings. We now put it to the test on the task of semantic hashing. Specifically, we compare the fruit fly to a standard binary network on the task of generating locality-sensitive hashes for text documents, measuring both task performance and energy consumption. Our results indicate that the two algorithms have complementary strengths while showing similar electricity usage.- Anthology ID:
- 2022.sustainlp-1.4
- Volume:
- Proceedings of The Third Workshop on Simple and Efficient Natural Language Processing (SustaiNLP)
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates (Hybrid)
- Editors:
- Angela Fan, Iryna Gurevych, Yufang Hou, Zornitsa Kozareva, Sasha Luccioni, Nafise Sadat Moosavi, Sujith Ravi, Gyuwan Kim, Roy Schwartz, Andreas Rücklé
- Venue:
- sustainlp
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 17–28
- Language:
- URL:
- https://aclanthology.org/2022.sustainlp-1.4
- DOI:
- 10.18653/v1/2022.sustainlp-1.4
- Cite (ACL):
- Tanise Ceron, Nhut Truong, and Aurelie Herbelot. 2022. Algorithmic Diversity and Tiny Models: Comparing Binary Networks and the Fruit Fly Algorithm on Document Representation Tasks. In Proceedings of The Third Workshop on Simple and Efficient Natural Language Processing (SustaiNLP), pages 17–28, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
- Cite (Informal):
- Algorithmic Diversity and Tiny Models: Comparing Binary Networks and the Fruit Fly Algorithm on Document Representation Tasks (Ceron et al., sustainlp 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2022.sustainlp-1.4.pdf