In-Context Learning Beyond Copying: A Training-Time Analysis of Abstractive ICL

Induction Circuit

Previous work showed that LLMs develop induction heads that perform inductive copying by matching patterns and copying them from earlier context. Induction circuits consist of three steps:

1. Previous token heads allow each token to store which token came before it
2. Induction heads attend to the previous token information in earlier contexts, resulting in a "prefix-matching" attention pattern
3. The induction head increases the probability of the attended token in the output

Formally, given input tokens (x₁, ..., x_j), induction circuits operate by searching for tokens that hold information of the current token x_j (i.e., searching x_i+1 where x_i = x_j, i < j). If there is a matching x_i+1, the induction head increases the logit of x_i+1 for the next prediction.

Olsson et al. (2022) hypothesized that these circuits underlie a wide range of in-context learning capabilities. However, subsequent work has demonstrated that induction heads operate in parallel with different components that are more causally important for performance on various ICL tasks. Yin et al. (2025) provide correlational evidence that induction heads transform into other ICL-related heads during training, but it is not clear whether abstractive ICL capabilities can emerge independently from induction heads. In order to understand the implications of the phenomena observed with the emergence of induction circuits, a causal intervention on the training process is necessary.

HAPAX Training Regime

To suppress inductive copying, we apply loss masking to create a training regime where tokens that are predictable by induction heads are excluded from the loss calculation. Specifically, we mask the loss contributions of token positions that contain a matching n-gram within the same context window (where n > 1). Single-token repetitions are not masked because they cannot be predicted by induction. Thus, the first token of any repeated n-gram is left unmasked.

We train Vanilla and HAPAX 1B models from scratch using the GPT-NeoX architecture with the same hyperparameter and training configuration as the Pythia models. We use the Pile dataset for training. The training data consists of 41.7B tokens (19,900 steps), of which 13.2B (31.7%) tokens are masked for the HAPAX model due to loss masking. This means the HAPAX model never receives gradient signals from repeated n-grams. Since tokens with high representational similarity (e.g., "National" and "_National") can still provide a copying signal, we also train a stricter variant, Thresholded-HAPAX, that additionally masks such tokens (52.5% of tokens masked). Because it also removes useful non-copying signal, we treat it as a more aggressive stress test.

Suppression of Inductive Copying

We first measure random repetition performance: the model is given 1000 sequences of random repeated tokens r₁r₂...r_sr₁r₂...r_s-1 and is expected to predict r_s. This synthetic task does not occur in natural language but is solvable through induction heads. The HAPAX model experiences a 66% drop and Thresholded-HAPAX experiences an 89% drop in accuracy relative to the vanilla model at the end of training. Natural text repetition, where the repeated sequence is taken from WikiText, also decreases over training.

On extractive tasks from Todd et al. (2024), HAPAX and Thresholded-HAPAX show reduced performance compared to vanilla on all but one task with statistically significant differences.

Emergence of Abstractive ICL Capabilities

We continue our evaluation with abstractive tasks, where the model needs to generate novel information not contained in the context. We evaluate on 34 abstractive tasks: 26 tasks from Todd et al. (2024) (e.g., Country-Capital task) and 8 word-level translation tasks. Since some tasks have small label spaces, a model can succeed by copying labels from the few-shot demonstrations. We therefore also evaluate a target-excluded setting, where demonstrations never share the target answer.

Rather than observing broad degradation of ICL when inductive copying is suppressed, we find that abstractive capabilities are retained and emerge at similar points in training for HAPAX and Vanilla. Under standard evaluation, HAPAX achieves higher accuracy on 13 of 22 tasks with statistically significant differences. Under target exclusion, HAPAX achieves higher accuracy on 26 of 28 tasks, indicating that part of Vanilla's standard-evaluation accuracy comes from copying target labels present in the context. Thresholded-HAPAX applies heavier masking and outperforms Vanilla on only 7 of 24 significant tasks under standard evaluation, but on 16 of 26 (62%) under target exclusion. Consistent with prior work, function-vector and concept heads still develop under HAPAX, and function-vector heads remain causally important for abstractive ICL.

In-Context Learning Beyond N-gram Copying

We use the token-loss difference metric to understand general ICL capabilities. This metric is defined by the differences of the cross-entropy loss of arbitrary token positions, conventionally using the 500th and 50th token positions. Intuitively, token-loss difference measures improvement across increasing token positions. If the loss at token 500 is lower and TLD > 0, it shows that the model's predictions improved with increasing context.

Yin et al. (2025) provided evidence that the metric is strongly influenced by induction heads but does not correlate well with in-context learning task performance. The vanilla model exhibits a sudden improvement in the metric, while the HAPAX variants appear to worsen over training. As HAPAX models perform worse at exact copying but retain abstractive ICL, our results demonstrate that the sudden increase in token-loss difference for the vanilla model is indicative of the emergence of inductive copying capabilities, but lacks indicative power for the emergence of abstractive ICL capabilities.

To investigate this hypothesis further, we propose evaluating TLD on samples where neither the 500th nor the 50th token can be predicted correctly with inductive copying, alongside a thresholded variant that also excludes positions predictable through highly similar tokens. Contrary to the regular metric, the vanilla model does not exhibit a sudden phase shift on non-extractive samples and is statistically indifferent to HAPAX. Overall, the TLD metric captures the coarse trends of the distribution on which it is computed, so its implications for different types of ICL capabilities depend on the choice of distribution.

Development of Prefix-Matching Under HAPAX

Within vanilla training, the sudden emergence of induction heads with high prefix-matching scores is a well-documented phenomenon. One possibility is that this phase transition is a developmental prerequisite for abstractive ICL, i.e., a necessary transition only after which abstractive capabilities can be learned. We therefore analyze the trajectory of prefix-matching scores in the vanilla and HAPAX models using the random repetition sequence.

In the vanilla model, 12 out of 128 attention heads attain a prefix-matching score over 80%, spike early in training and then decay, while the HAPAX variants display only a single head above 80%. This emergence unfolds gradually over 1800 steps in HAPAX, in contrast to the sudden emergence in vanilla.

Beyond their reduced number and strength, many remaining prefix-matching heads in HAPAX also depart from canonical induction-head behavior. Applying logit lens to the top 10 prefix-matching heads, inclusion of the token to be copied among the promoted tokens worsens from vanilla to HAPAX. This also holds under mean-ablation studies measuring each head's contribution to predicting the target token. Among the top 10 prefix-matching heads in HAPAX, 6 suppress rather than promote the correct token (compared to 2 in vanilla), functionally behaving closer to anti-induction heads than to regular induction heads. Taken together, these findings suggest that abstractive ICL can emerge without the abrupt prefix-matching phase transition observed during vanilla training.

Initialization-Level Prefix-Matching Structure

Under HAPAX, the vanilla phase transitions do not appear and prefix-matching scores are attenuated, but prefix matching is not eliminated entirely. We explore how these patterns can persist when the training objective removes the loss contribution of induction-predictable positions. One possibility involves previous token heads, which can be learned for reasons beyond in-context copying, such as detokenization.

If previous token information is useful for reasons other than in-context copying, later heads may still exploit it and produce prefix-matching patterns without explicit optimization for copying. To test this, we patch the output of layer L4 from the vanilla model at step 5000, which contains previous token heads but no strong prefix-matching heads, into the randomly initialized model and measure whether later randomly initialized heads begin to exhibit prefix matching.

The heads that respond to patching also rank among the top five prefix-matching heads at the final vanilla checkpoint. We observe a similar pattern in HAPAX: two of its top three final prefix-matching heads also increase their scores in this experiment. Since vanilla and HAPAX share the same initialization seed, this overlap suggests that the remaining prefix-matching scores in HAPAX can partly reflect an initialization-level attention tendency, rather than training pressure for copying.

Key Takeaways

Our results suggest that the phase transition observed in standard training is closely tied to the learning of repeated, induction-predictable structure in the data. When this copying signal is suppressed through HAPAX, abstractive ICL capabilities still develop and are retained, despite the absence of the canonical phase transition. More broadly, our approach illustrates how targeted loss masking can disentangle training dynamics that emerge together under standard training, letting us test whether a phenomenon is tied to the capability being studied or to other predictable structure in the data.

Related Work

Our work builds on research investigating induction heads, in-context learning mechanisms, training dynamics, and the role of repetition in language models.

Attention Heads and In-Context Learning

Nelson Elhage, Neel Nanda, Catherine Olsson, et al. A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread, 2021.
Notes: The authors introduce a mathematical framework for understanding transformer circuits and identify induction circuits responsible for inductive copying, where LLMs match patterns and copy them from earlier context.

Catherine Olsson, Nelson Elhage, Neel Nanda, et al. In-context Learning and Induction Heads. Transformer Circuits Thread, 2022.
Notes: The first mechanistic analysis of ICL capabilities in LLMs. The authors find that the development of induction circuits is associated with rapid phase transitions as the model displays prefix-matching and inductive copying capabilities, with a general increase in ICL capabilities following later in training. They hypothesize that induction heads underlie a wide range of ICL capabilities and observe that some heads are also involved in "fuzzy" copying based on semantic similarity.

Kayo Yin, Jacob Steinhardt. Which Attention Heads Matter for In-Context Learning? ICML, 2025.
Notes: The authors show that ablation of traditional induction heads does not damage ICL as much as ablation of function vector heads. They also observe that certain heads displaying high prefix-matching scores early in training later transform into function vector heads, suggesting a potential developmental link between induction heads and other ICL-related heads.

Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, David Bau. Function Vectors in Large Language Models. ICLR, 2024.
Notes: The authors identify function vector heads that trigger ICL tasks and introduce extractive tasks (where the model must directly extract the answer from the input) and abstractive tasks (which require generating new answers rather than copying) that are used within our work to analyze different ICL capabilities.

Sheridan Feucht, Eric Todd, Byron C. Wallace, David Bau. The Dual-Route Model of Induction. CoLM, 2025.
Notes: The authors show that models contain separate concept induction circuits that are causally more important than traditional induction circuits for "fuzzy" copying tasks (e.g., translation).

Training Dynamics and Induction Heads

Stephanie C.Y. Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang, Aaditya K. Singh, Pierre H. Richemond, James L. McClelland, Felix Hill. Data Distributional Properties Drive Emergent In-Context Learning in Transformers. NeurIPS, 2022.
Notes: The authors identify data distributional properties that drive the emergence of in-context learning, showing how skewed Zipfian distributions lead to the emergence of ICL. Importantly, they demonstrate that Zipfian distributions enable both in-context and weight-based learning to co-exist, resolving an initial trade-off between the two learning modes.

Aaditya K. Singh, Ted Moskovitz, Felix Hill, Stephanie C.Y. Chan, Andrew M. Saxe. What Needs to Go Right for an Induction Head? A Mechanistic Study of In-Context Learning Circuits and Their Formation. ICML, 2024.
Notes: The authors use clamping to study subcomponents of induction circuits and their effect on phase changes. Using an optogenetics-inspired causal framework, they identify three underlying subcircuits that interact to drive induction head formation.

Gouki Minegishi, Hiroki Furuta, Shohei Taniguchi, Yusuke Iwasawa, Yutaka Matsuo. Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergence. ICML, 2025.
Notes: The authors study a meta-learning setting to investigate mechanisms in non-exact copying scenarios. They find that this meta-learning ability emerges in multiple training phases, with a different circuit appearing in each phase rather than a single induction-head transition.

Loss Masking and Repetition

Nicolas Zucchet, Francesco D'Angelo, Andrew Kyle Lampinen, Stephanie C.Y. Chan. The Emergence of Sparse Attention: Impact of Data Distribution and Benefits of Repetition. NeurIPS, 2025.
Notes: The authors analyze how training data repetition speeds up emergent behavior in language models, showing the importance of repetition in the data distribution for emergent behavior.

Abhimanyu Hans, John Kirchenbauer, Yuxin Wen, Neel Jain, Hamid Kazemi, Prajwal Singhania, Siddharth Singh, Gowthami Somepalli, Jonas Geiping, Abhinav Bhatele, Tom Goldstein. Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs. NeurIPS, 2024.
Notes: The authors incorporate loss masking strategies to prevent memorization of private information, excluding the loss contributions of certain tokens.

Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, Jason Weston. Neural Text Generation With Unlikelihood Training. ICLR, 2020.
Notes: The authors present unlikelihood loss training to mitigate repetitions in the model. They observe that the resulting models give less repetitive, less dull text while maintaining perplexity, giving superior generations.

How to cite

The paper can be cited as follows.

@misc{sahin2026incontextlearningcopyingtrainingtime,
      title={In-Context Learning Beyond Copying: A Training-Time Analysis of Abstractive ICL}, 
      author={Kerem Sahin and Sheridan Feucht and Adam Belfki and Jannik Brinkmann and Aaron Mueller and David Bau and Chris Wendler},
      year={2026},
      eprint={2511.05743},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2511.05743}, 
}