A Conditional No-Hair Theorem for Transformers: How Language Models Forget Their Prompt

arXiv cs.AIOctober 20, 2026en

arXiv cs.AI

AI Global Wire

arXiv:2610.24133v1 Announce Type: new Abstract: We ask how much an autoregressive transformer retains of its prompt after generating $t$ tokens. Consider two different prompts followed by the same continuation text, so that any difference in their hidden states comes from the prompt alone, and we find that the distance between their hidden states decays approximately as a power law $t^{-\alpha}$, where $\alpha=1$ is the rate at which a fixed prompt's share of uniform attention shrinks and smaller values mean longer memory. Trained models forget fast ($\alpha\approx0.55$--$0.7$), untrained ones slowly ($\alpha\approx0.2$--$0.5$). We prove that this offset is $O\big(P(\log t)^{L-1}/t\big)$ for

This is a short summary published by AI Global Wire. The full article is owned and hosted by arXiv cs.AI — open it there to read it in full.

• Verktyg
• Forskning
• Reglering