New Stanford paper finds that showing a small AI language model more examples cannot teach it a pattern it does not already notice, such as the third letter of a word.
finds that LLMs learn from few-shot examples by looking up stored labels, and only switch that lookup on when labels repeat.
An early signal opens this lookup only when some label repeats. With 1 example per class the model was right about 20% of the time, and with 6 it was right 96%.
Give a model a dozen sentences tagged with made-up labels like foo and bar, and it learns the mapping without weight updates. Earlier work found parts of this, but not how they fit together.
They traced the full path in Qwen3-4B, SmolLM3-3B and Llama-3.2-3B. Each label token stores the class of its sentence, and about 90 later heads match the new sentence against those stored classes.
For few-shot prompts, repeat every label several times and pick classes the model can already tell apart, since examples cannot teach a feature it lacks.
– arxiv. org/abs/2611.05344
Title: "A Circuit for In-Context Learning"