How to Detect and Respond to Hallucinations or Unsafe Outputs in Production

A customer asks your support agent about a refund policy. Retrieval returns nothing useful, so the model fills the gap with a confident, invented answer, and the customer acts on it. Nobody notices, because the dashboard still says "90% positive sentiment." That number means almost nothing without the conversations behind it, and it is the single most common way unsafe outputs slip into production undetected.

Sanity is the Content Operating System for the AI era, the intelligent backend for companies building AI content operations at scale, and it treats agent behavior as content you can log, score, and govern rather than a black box you hope stays green. Legacy CMSes bolt on AI; Sanity is built for it, with structured models, event-driven workflows, and telemetry that lives beside the content the agent actually queried.

This guide reframes hallucination as an engineering problem with a fix at a specific layer, not a mysterious model flaw. You will get a detection pipeline (trace logging, then classification), a failure-mode taxonomy that points each defect at the layer that caused it, safety exits for unsafe outputs, and an eval bench that gates every change before it ships.

Why a green dashboard is not detection

The failure mode that hurts most is silent. A refund answer gets invented, a policy gets misquoted, an agent reaches a tool it should never have touched, and none of it surfaces because the aggregate metric looks healthy. Most agents in production today do not actually know if they are working. The dashboard says "90% positive sentiment," which means almost nothing without the conversations behind it. Sentiment scores measure tone, not truth, and a confidently wrong answer often reads as perfectly pleasant.

Real detection starts one layer lower, at the trace. Every turn, every tool call, every retrieval result, and every model output, captured with timestamps and token counts. Without this raw record, the rest of your governance program does not exist, because you have nothing to score, nothing to replay, and nothing to point a compliance reviewer at when they ask what the agent said and why. A trace is not a log line you skim; it is the evidentiary substrate for everything downstream.

The practical trap is that traces are usually written to an observability sink that lives far from the content the agent queried. You end up correlating a trace ID in one system with a document in another, by hand, weeks after the incident. That distance is where investigations die. The alternative is to treat the conversation itself as a first-class object in the same backend that serves the agent, so the retrieval result the agent saw and the answer it gave are one query apart, not one integration apart. Detection is only useful if the person diagnosing the failure can get from the symptom to the source content without leaving the system.

Classify conversations, do not just count them

Raw traces tell you what happened; they do not tell you whether it was right. That verdict comes from conversation classification: a scoring model run asynchronously over the transcripts to answer the questions that matter. Was this conversation a success? What was the user trying to do? Did the agent reach a tool it should not have? Did retrieval return useful results, or did the agent hallucinate? Classification is not perfect. It is a hundred times better than no scoring, because it turns an unreadable ocean of transcripts into a triage queue you can act on.

The reason to run this asynchronously and at scale is volume. You cannot read every conversation, but a model can read all of them, flag the suspicious ones, and let a human confirm the fraction that matters. Braintrust is what Sanity runs internally for production scoring at scale: it logs every turn and lets you score conversations against rubrics asynchronously. What tools like it deliberately do not do is tell you whether the agent is right in the absolute. They give you the signal; the verdict is a separate discipline you own.

The payoff of classification is that it produces structured judgements you can query, not prose you have to read. In Sanity, each judgement lands as a conversationScore document with success (a number from 0 to 5), retrievalQuality (good, partial, empty, or wrong), failureMode (hallucination, tool-misuse, scope-violation, empty-retrieval, auth-confusion, prompt-drift, or none), and notes. Because these live as content in the Content Lake next to the source documents, a reviewer's notes can reference the exact failed conversation and the documents the agent should have retrieved. That colocation is what turns scoring from a one-time spreadsheet into a living, queryable artifact.

A failure-mode taxonomy that points at the right layer

The most expensive mistake teams make with hallucinations is trying to fix them in the model. Prompt harder, swap to a bigger model, add a "do not make things up" instruction. None of it holds, because hallucination is usually not a model problem. Hallucination usually means retrieval returned nothing useful and the model filled the gap. That is a retrieval problem, and the fix belongs at the retrieval layer, not in the model's temperature setting.

The discipline that makes this actionable is a failure-mode taxonomy where each tag points at exactly one harness layer. Hallucination and empty-retrieval point at retrieval; empty-retrieval is the structural ceiling of your retrieval layer, the honest signal that the content simply was not there to be found. Tool-misuse means the prompt let the agent reach a tool that was not right for the conversation type, so the fix is in the tools layer and the prompt. Scope-violation means the agent answered something the never-say list should have caught, which is a prompt problem. Auth-confusion means the agent acted under the wrong identity, a tools-and-auth problem. Prompt-drift means somebody shipped a prompt change without the eval bench catching the regression.

Each tag points at a layer, and the fix is at that layer, not in the model. This is what converts a vague complaint ("the agent hallucinates") into a work ticket with an owner. When the failureMode field on a conversationScore says empty-retrieval, the retrieval team knows the content gap is theirs to close. When it says prompt-drift, you know a change shipped without a gate, and the response is process, not a model swap. A taxonomy is only as good as its precision, and precision is what lets you route the defect to the person who can actually resolve it.

Responding to unsafe outputs in the moment

Detection catches what already happened; response is about limiting the blast radius while it is happening. The first line of defense is not clever prompting, it is boring engineering: safety exits. A maximum iteration count, usually somewhere between 20 and 50, plus a cap on total tokens or wall-clock time, so a stuck or buggy agent cannot loop forever burning tokens and producing garbage. An agent without a ceiling is an incident waiting for a bill.

The second line is designing tools so the model can recover instead of inventing. When a tool fails, it should return a structured error the model can read back, like { error: 'Session expired' }, rather than throwing an opaque exception that the model paves over with a guess. A model that receives "Session expired" can tell the user to re-authenticate; a model that receives silence hallucinates a plausible-sounding order status. Pair this with OpenTelemetry span attributes, exceptions, and status codes so every tool call is traceable, and the same trace that powers detection also powers the postmortem. OpenTelemetry is the open standard the tracing code plugs into, and the Vercel AI SDK exposes streaming primitives that work with it.

The hardest safety boundary to enforce at runtime is the scope boundary: topics the agent must refuse. This is where the never-say list earns its place, and where response and prevention meet. A forbidden-topics list that lives in the prompt and is checked before the agent answers turns a potential scope-violation into a clean refusal. The point of response is to fail safe and fail visibly, so that a bad output becomes a logged, recoverable event rather than a customer acting on invented information.

The eval bench: the gate that makes every change safe

Everything above produces signal. The eval bench produces the verdict. It is a frozen set of representative conversations, twenty to start, each scored against a rubric you wrote, and it runs on every model change, every prompt change, and every tool change. The bar to ship anything to production is the bench staying green. This is the difference between hoping a change is safe and knowing it, and it is the antidote to prompt-drift: a regression that would have slipped through review gets caught by a test that replays real failure cases.

The bench is also what makes a genuinely powerful idea safe, which is treating the application system prompt as customer-facing behavior and governing it like it. Split the prompt into fields (role, voice, userContext, escalation, and a mustNotSay forbidden-topics list owned by Compliance) so the right people can edit without a pull request. Brand owns voice, Product owns how the agent uses user context, Support owns escalation, and Compliance owns the never-say list. Splitting the prompt into fields is not cosmetic; it is access control. The rule is simple: author it like content, gate it like code. A brand or support edit ships only if the bench holds.

Because the prompt is content, it gets the same treatment as any other governed content in the Studio: versioning, scheduled publishing through Content Releases, and rollback. A prompt change runs the eval bench in CI before it can ship, which is the "gate it like code" half, and it is what makes "anyone can edit" safe rather than scary. When the frozen conversations, their scores, and the source content the agent queried all live in one backend, the bench stops being a spreadsheet someone maintains and becomes a living artifact that the whole team keeps green.

Why this belongs beside your content, not bolted on beside it

You can assemble all of this yourself: a tracing sink here, a scoring service there, a vector store somewhere else, and a governance workflow held together with review meetings. It works until an incident, when the investigator has to correlate a trace ID in one system, a score in another, and a source document in a third, by hand, under time pressure. The architecture decides how fast you can answer the question that governance ultimately asks: what did the agent say, why, and what should it have said instead?

Sanity's answer is to keep the conversations, the scores, and the source content the agent queried in one place. Agent Context Insights is a telemetry and insights layer for agents built on the Context MCP endpoint, with three parts: an insights dashboard in the agentContext plugin, an AI SDK telemetry integration with a saveConversation primitive that stores conversations back in Sanity, and conversation classification you run from a scheduled Function over those stored conversations. Because scores live as structured content next to the documents the agent queried, the same backend that serves the agent stores the conversations it had and the scores derived from them.

That colocation is the difference between an intelligent backend and a pile of integrations. Legacy CMSes stop at publishing; Sanity operates content end to end, so detection, response, and the eval bench draw on the same structured model, event-driven Functions, and governed workflows that already serve the agent. Hallucination detection is not a feature you add on top; it is a property of storing agent behavior as first-class, queryable content in the system that produced it.