GPT-6 Astra, looped transformers, and hidden reasoning(magazine.sebastianraschka.com)
492 points by ModelForge 1 day ago | 158 comments
tl;dr: OpenAI's rumored GPT-6 Astra reportedly uses "looped transformers" (recurrent depth), an architectural technique where intermediate representations pass through the same transformer blocks multiple times, effectively increasing depth without adding parameters—an approach seen in models like Nanbeige, Ouro, and Universal Transformers. The author argues this looping likely isn't the reason Astra produces shorter/less monitorable reasoning traces; more capable models simply need less scratchpad, similar to how larger models in a family use fewer tokens than smaller siblings. Astra's biggest practical leap appears to be in computer-use tasks, likely trained via RL on OpenAI's reported fleet of Mac Minis.
HN Discussion:
  • Provides supporting academic references on CoT computational requirements and prior work on universal transformers
  • Questions whether looped transformers inherently hide reasoning and asks about labs using latent recursive reasoning
  • Argues that looping a full transformer is by-definition hidden reasoning, pushing back on the article's framing
  • Praises the article's clarity and endorses the author as a trusted source on LLM internals
  • Hands-on experimentation with looped transformer models confirms the article's claims about compute intensity