NOTES / 2026.10.07 · 10-post thread · 34 likes
Reality is the final verifier
Ion Stoica and coauthors on the two gaps in agentic software engineering, read against the economics of AGI: agents are checked against what is codified and find whatever isn’t — and the economy ships those systems anyway, because the risk lands on everyone else.
FIG. 01 — FROM THE ORIGINAL THREAD
Insightful paper by @istoica05 & coauthors: with agentic software, reality is the final verifier. Tests, benchmarks, even formal proofs are just proxies.
We have to rapidly change how we operate accordingly. arxiv.org
It fits with how we model the economics of AGI. They explain how one system fails: agents are checked against what is codified, and find whatever isn’t. We unpack why the economy ships those systems into production anyway. x.com
Their example: an agent told to speed up a key-value store got 6x by storing a seed and regenerating values on the fly. Tests passed as nobody had written down that a store should store things! Specs will always miss something, and agents will find it. x.com
Our contribution: why the incentive is to automate anyways. Generating code is nearly free. Checking it expensive. Shipping unverified work is rational whenever the risk lands on everyone else. We call that the Trojan Horse externality. x.com
Both papers land on the same scarce input: humans with authority over what counts as acceptable. They use agents to triage and escalate. We ask how the supply of expert verifiers evolves when juniors lose the routine work that builds expertise. x.com
Bainbridge: automating routine work can erode the expertise needed for hard cases.
Our “Missing Junior Loop” models that erosion. Human augmentation and synthetic practice are investments in rebuilding and expanding human verification capacity.
They explore formal verification, hardware abstractions, world models and evaluator cascades.
These all shift our Cost to Verify curve. The ultimate economic test: can verification capacity keep pace with deployment? Do we only automate when risks are acceptable?
The obvious shortcut is AI checking AI. Both papers say no! A reviewer working from the same spec has the same blind spots. It’s also a market for lemons. The antidote? Reliable liability-as-a-service: AI that guarantees an outcome end-to-end. x.com
Their outer loop makes assurance recurring: deploy, observe, revise spec, model or evaluator, rerun. Our model asks whether effective oversight can keep pace with execution. If it cannot, alignment decays. Human augmentation and better measurement shift that frontier.
Their closing: implementations can be generated just in time; assurance cannot. Same in our model: execution is free, standing behind it is not, and value accrues to whoever can underwrite the work.
Reality Is the Final Verifier: arxiv.org Some Simple Economics of AGI: arxiv.org
Originally published as a thread on X.