tl;dr: Viewing neural network layers as associative memory stores, the author derives transformer architectures by progressively reducing recall interference. A linear outer-product store holds ~d_k memories, MLPs with squared/cubed ReLUs push capacity to d_k^p via nonlinear suppression of low-similarity keys, and self-attention's softmax—equivalent to using all polynomial powers via the exponential—achieves exponential capacity (~1.2M memories at d_k=64). Scaling the softmax sharpness β with log M, as done in YaRN and Qwen, extends this to effectively unbounded context lengths.