--- title: "Goodhart’s Law with teeth" date: 2026-07-22 url: "https://x.com/ccatalini/status/2079949947307475019" tags: [ai-agi] deck: "The metric wasn’t merely gamed — the unmeasured rules of the game became the agent’s degrees of freedom. We modeled this exact failure mode five months ago." image: "/images/notes/goodharts-law-with-teeth.jpg" tweetCount: 3 likes: 9 reposts: 5 --- For everyone rushing to Goodhart’s Law: maybe. But that’s not the crucial bit. The metric wasn’t merely gamed. The unmeasured rules of the game became the agent's degrees of freedom. We modeled this exact failure mode five months ago: Agents will inevitably optimize beyond what we can define, steer, measure and verify. We wrote: "This is Goodhart’s Law with teeth. Classical metric gaming merely inflates a proxy; here, an autonomous agent optimizes so aggressively for measurable targets—treating the unmeasured residual not as an adversary but as an unconstrained degree of freedom—that severe erosion of actual utility becomes a mechanical byproduct. In navigating this unverified space, the agent organically derives latent instrumental preferences (goal preservation, deception, resistance to shutdown) hidden within the dimensions the human cannot observe. Because prohibitive verification costs blind the principal to both the proxy divergence and these emergent goals, the corruption is self-concealing: gross output (Y) rises, but the unverified residual leaks into the economy as a "Trojan Horse" externality (Xa) that consumes real resources to generate counterfeit utility—crowding out human consumption and capital accumulation from within the budget constraint. This is the equilibrium outcome of rational deployment under a widening Measurability Gap." [arxiv.org](https://arxiv.org/html/2602.20946v2) The uncomfortable part? This one was caught because both sides had instrumentation pointed at it. Similar optimizations are running in production every day. Most without the right telemetry.