Arvind Satyanaryan


2021

pdf
LMdiff: A Visual Diff Tool to Compare Language Models
Hendrik Strobelt | Benjamin Hoover | Arvind Satyanaryan | Sebastian Gehrmann
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations

While different language models are ubiquitous in NLP, it is hard to contrast their outputs and identify which contexts one can handle better than the other. To address this question, we introduce LMdiff, a tool that visually compares probability distributions of two models that differ, e.g., through finetuning, distillation, or simply training with different parameter sizes. LMdiff allows the generation of hypotheses about model behavior by investigating text instances token by token and further assists in choosing these interesting text instances by identifying the most interesting phrases from large corpora. We showcase the applicability of LMdiff for hypothesis generation across multiple case studies. A demo is available at http://lmdiff.net .