The most expensive failures in production AI systems aren't the crashes. Crashes page someone. The expensive failures are the quiet ones: the agent still answers every request, still returns 200, still looks alive on every dashboard you own — and its answers have been getting steadily worse for six weeks.
AI agents degrade silently because the signals that would reveal degradation — answer quality, task success, user trust — are exactly the signals conventional monitoring doesn't measure.
The three forces of drift
Model drift from above. Your agent sits on top of a model you don't control. Providers ship updates, adjust safety behavior, and deprecate versions on their schedule, not yours. I've watched a minor model version change cut an agent's tool-use accuracy by double digits — with zero code changes on our side. Pinning versions only defers the problem: eventually the pinned version is deprecated and you're forced onto new behavior anyway.
Input drift from below. The prompt was written against last quarter's traffic. Then your company launched a new product line, a new pricing tier, a new market. The agent's instructions never mention any of it, so it improvises — confidently. The share of traffic the prompt genuinely covers shrinks month by month.
Prompt rot from within. Every incident adds another instruction: "always check X," "never say Y," "if the customer mentions Z, do W." Eighteen months later the prompt is 4,000 tokens of accumulated scar tissue, and instruction #12 quietly contradicts instruction #31. Each edit was tested against the failure it fixed — none were tested against each other.
Why your dashboards can't see it
Uptime, latency, token spend, error rate: all healthy, all irrelevant. A wrong answer has the same latency profile as a right one. The information that something is wrong exists only in the content of the outputs and the downstream consequences — reopened tickets, reverted PRs, users who stop using the feature. Those live outside the agent's telemetry entirely, which is why teams consistently learn about degradation from an angry customer or an executive screenshot, weeks after the trend began.
The fix is boring, and that's the point
Continuous evaluation. Sample live runs — every run if volume permits — and score them against a fixed, versioned rubric: correctness checks where verifiable, LLM-as-judge scoring where judgment is required, and outcome joins where consequences are trackable. Chart the score like you'd chart p99 latency. Alert on the trend, not the anecdote.
Once that line exists, everything changes. A model update that hurts you shows up in days, with evidence. Input drift appears as a growing cluster of low-scoring runs about topics your prompt doesn't cover (see failure clustering). And prompt rot becomes measurable enough to justify the cleanup nobody could previously prioritize.
Degradation itself isn't preventable — you don't control the model, the traffic, or entropy. What's preventable is not knowing. An agent whose quality is continuously measured can't get quietly worse; it can only get loudly worse, and loud problems get fixed. That measurement is the "Observe" and "Evaluate" half of the closed loop — and it's the half you should build first.