Abstract
We present a locality preserving loss (LPL) that improves the alignment between vector space embeddings while separating uncorrelated representations. Given two pretrained embedding manifolds, LPL optimizes a model to project an embedding and maintain its local neighborhood while aligning one manifold to another. This reduces the overall size of the dataset required to align the two in tasks such as crosslingual word alignment. We show that the LPL-based alignment between input vector spaces acts as a regularizer, leading to better and consistent accuracy than the baseline, especially when the size of the training set is small. We demonstrate the effectiveness of LPL-optimized alignment on semantic text similarity (STS), natural language inference (SNLI), multi-genre language inference (MNLI) and cross-lingual word alignment (CLA) showing consistent improvements, finding up to 16% improvement over our baseline in lower resource settings.- Anthology ID:
- 2021.adaptnlp-1.13
- Volume:
- Proceedings of the Second Workshop on Domain Adaptation for NLP
- Month:
- April
- Year:
- 2021
- Address:
- Kyiv, Ukraine
- Editors:
- Eyal Ben-David, Shay Cohen, Ryan McDonald, Barbara Plank, Roi Reichart, Guy Rotman, Yftah Ziser
- Venue:
- AdaptNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 122–139
- Language:
- URL:
- https://aclanthology.org/2021.adaptnlp-1.13
- DOI:
- Cite (ACL):
- Ashwinkumar Ganesan, Francis Ferraro, and Tim Oates. 2021. Locality Preserving Loss: Neighbors that Live together, Align together. In Proceedings of the Second Workshop on Domain Adaptation for NLP, pages 122–139, Kyiv, Ukraine. Association for Computational Linguistics.
- Cite (Informal):
- Locality Preserving Loss: Neighbors that Live together, Align together (Ganesan et al., AdaptNLP 2021)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2021.adaptnlp-1.13.pdf
- Data
- MultiNLI, SNLI