Locality Preserving Loss: Neighbors that Live together, Align together

Ashwinkumar Ganesan, Francis Ferraro, Tim Oates


Abstract
We present a locality preserving loss (LPL) that improves the alignment between vector space embeddings while separating uncorrelated representations. Given two pretrained embedding manifolds, LPL optimizes a model to project an embedding and maintain its local neighborhood while aligning one manifold to another. This reduces the overall size of the dataset required to align the two in tasks such as crosslingual word alignment. We show that the LPL-based alignment between input vector spaces acts as a regularizer, leading to better and consistent accuracy than the baseline, especially when the size of the training set is small. We demonstrate the effectiveness of LPL-optimized alignment on semantic text similarity (STS), natural language inference (SNLI), multi-genre language inference (MNLI) and cross-lingual word alignment (CLA) showing consistent improvements, finding up to 16% improvement over our baseline in lower resource settings.
Anthology ID:
2021.adaptnlp-1.13
Volume:
Proceedings of the Second Workshop on Domain Adaptation for NLP
Month:
April
Year:
2021
Address:
Kyiv, Ukraine
Editors:
Eyal Ben-David, Shay Cohen, Ryan McDonald, Barbara Plank, Roi Reichart, Guy Rotman, Yftah Ziser
Venue:
AdaptNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
122–139
Language:
URL:
https://aclanthology.org/2021.adaptnlp-1.13
DOI:
Bibkey:
Cite (ACL):
Ashwinkumar Ganesan, Francis Ferraro, and Tim Oates. 2021. Locality Preserving Loss: Neighbors that Live together, Align together. In Proceedings of the Second Workshop on Domain Adaptation for NLP, pages 122–139, Kyiv, Ukraine. Association for Computational Linguistics.
Cite (Informal):
Locality Preserving Loss: Neighbors that Live together, Align together (Ganesan et al., AdaptNLP 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/naacl24-info/2021.adaptnlp-1.13.pdf
Data
MultiNLISNLI