Posts tagged "induction-heads"

The Redundancy Trap: Why Single-Head Ablation Lies

August 7, 2026 •20 min read

# The Redundancy Trap *I deleted the two attention heads with the largest positive direct effects in GPT-2 small. The model got **better**. Then I found the same failure across seven models — and in

Why Does Attention Attend There? Tracing the Other Half of the Circuit

June 17, 2026 •3 min read

![QK attributions confirm the induction-head story directly from the weights](qk-induction.png) Attribution graphs gave us a way to see *what* information a model moves between tokens. But they froz

A Mathematical Framework for Transformer Circuits: How LLMs Actually Work

June 5, 2026 •5 min read

window.MathJax = { tex: { inlineMath: [['$','$'], ['\$','\$']], displayMath: [['$$','$$'], ['\\[','\\]']] }, svg: { fontCache: 'global' } }; ![A Mathematical Framework for Transform