Ruoqing Yao


2026

Prior work (Wilcox et al, 2024; Kobzeva et al., 2025) shows that neural language models exhibit filled-gap and unlicensed-gap effects, yet these effects attenuate with intervening clauses, especially with intervening overt complementizers. We conduct attention probing experiments on GPT-2 and identify two specific heads (layer 5, head 2, and layer 8, head 9) whose verb-to-filler attention correlates with filled-gap surprisal. The two heads are sensitive to clausal intervention but not to linear distance, and they show distinct patterns in islands. When intervening overt complementizers appear, head 2 of layer 5’s attention redistributes from the filler to the nearest complementizer, producing an “attend-closest-C” pattern, while head 9 of layer 8 does not. These results may suggest that LMs may have allocated distinct linguistically meaningful representations from the training data to individual attention heads, but they fail to fully learn the correct grammars of FGDs.