Nvidia-led paper traces irrelevant-context errors to a small set of attention heads

A paper from Nvidia and other labs finds that irrelevant text mainly weakens a model's use of the question, rather than directly pushing against the correct answer.

The authors identify a small set of attention heads, which they call intrusion heads, that read unrelated text and reduce support for the right answer. Per the X post by Rohan Paul summarizing the paper, masking about 10 heads on 7B-scale models across four tasks recovered 57% of the accuracy lost to distractors.