The agent observability market got crowded fast. Tracing, token accounting, latency waterfalls, prompt playgrounds, eval dashboards — a genuinely useful stack, and I don't want to diminish it. But survey what it all measures and a pattern emerges: the entire stack records what the agent said. Almost none of it records what the agent caused.
The trace ends when the response is sent. Reality begins right after: the customer reads the answer and either goes away satisfied or opens a second, angrier ticket. The generated code merges and either survives or gets reverted Thursday. The truth about every agent run lives in that after — and the after is, overwhelmingly, not being captured.
The only unfoolable signal
Every evaluation method that reads the transcript shares a weakness: fluent, confident, wrong output can fool it. Human reviewers skim; judge models carry fluency bias; programmatic checks verify form, not truth. Outcome signals are different in kind. A reopened ticket doesn't care how well-written the wrong answer was. A reverted PR is immune to persuasion. This is why outcomes belong at the top of the evaluation hierarchy — not as another signal, but as the ground truth the other signals get calibrated against. A judge whose scores don't predict outcomes is measuring eloquence; the only way to know is to have the outcomes.
Why everyone skips it anyway
Because it's genuinely annoying plumbing. Outcome joins mean webhooks from the ticketing system, watching repos for reverts of agent-authored commits, product-analytics events keyed to run correlation IDs — integration work across systems owned by different teams, with results arriving on business timelines rather than request timelines. No single API call gets you there, which is precisely why the boxed observability products stop at the trace boundary. It's also why the teams that do the work end up with something the boxed products can't sell them.
The moat argument
Ask what's actually defensible in an AI agent business. The model? Rented — your competitor calls the same API. The prompts? Replicable in an afternoon by anyone who's seen the behavior. The framework? Open source. What cannot be bought, scraped, or replicated is the joined history of your agent's decisions and their measured real-world consequences in your specific domain. Ten thousand runs annotated with what actually happened afterward: which policy phrasings led to reopens, which code patterns got reverted, which escalation timings retained customers.
That dataset has two compounding properties. It grows with volume, automatically, while competitors starting later start at zero. And it directly powers the improvement loop — better failure detection, better clusters, better candidates, better gates — which improves the agent, which wins more volume, which grows the dataset. That flywheel is as close to a structural moat as anything in the agent economy, and it starts spinning the day you wire the first outcome join.
The cost of waiting
Here's the part that should create urgency: outcome data is only capturable in the present. Every week your agent runs without outcome joins is training signal permanently lost — you cannot retroactively learn which of last quarter's ten thousand answers quietly failed. Whatever else you defer in your agent roadmap, don't defer the correlation IDs and the first join. The traces tell you what your agent did today. The outcomes are what teach it to be better tomorrow — and they're the one dataset nobody can ever sell to your competitors.