How to Log and Monitor LLM Queries and Responses for Governance

Your compliance team asks a simple question about the AI support agent: show me exactly what it told this customer last Tuesday, what data it retrieved to say it, and who was on the hook when it acted on their account. The dashboard says "90% positive sentiment," and that number means almost nothing without the conversations behind it. There is no transcript, no retrieval result, no token count, no trace. The agent is a black box that occasionally speaks on your behalf, and nobody can reconstruct why it said what it said.

Sanity is the Content Operating System for the AI era, the intelligent backend for companies building AI content operations at scale, and it treats this problem as a content problem rather than a monitoring afterthought. Governance is not a dashboard you bolt on after launch. It is trace logging as the non-negotiable floor, asynchronous conversation scoring, failure-mode tags that point to a fix, and an eval bench that decides whether a change ships at all.

This guide walks the layers in order, from logging every turn to auth-forwarded audit trails, and shows why keeping conversations, scores, and the source content the agent queried in one governed system beats stitching them together after the fact.

Trace logging is the floor, not a feature

Most agents in production today do not know if they are working. A sentiment dashboard reports an aggregate number and quietly discards the evidence that would let anyone audit it. When your risk team needs to answer what the agent said, what it retrieved, and how many tokens it burned to say it, an aggregate metric is useless. You need the raw record.

The floor is trace logging: every turn, every tool call, every retrieval result, and every model output, captured with timestamps and token counts. Without this, the rest does not exist. There is nothing to score, nothing to review, and nothing to hand an auditor. This is not a nice-to-have layer you add once the agent is popular. It is the substrate every governance control sits on, because you cannot govern what you did not record.

Treating the trace as structured content changes what it is worth. Instead of logs that expire in an observability tool, each conversation becomes a durable record that lives next to the content the agent queried to produce it. On Sanity, an AI SDK telemetry integration exposes a saveConversation primitive that stores conversations back in Sanity, so the transcript, the retrieval results, and the source documents are all in one Content Lake. The reviewable artifact and the material the model read are no longer in separate systems you have to reconcile. When a regulator or a customer asks what happened, the answer is a query against governed content, not a forensic reconstruction across three vendors.

Scoring conversations so the trace becomes signal

Raw traces answer what happened. They do not answer whether it was any good. At production volume no human reads every transcript, so the second layer is conversation classification: score each conversation with a model running asynchronously over the transcripts. Was this a success? What was the user trying to do? Did the agent reach a tool it should not have? Did retrieval return useful results, or did the model fill the gap with something it invented?

This scoring is not perfect. A model grading a model will miss cases and mislabel edge behavior. But it is a hundred times better than no scoring, and it turns a pile of transcripts into a dataset you can filter, trend, and act on. The practical shape is a structured record. On Sanity a conversationScore document references the conversation and carries a success field (0 to 5, did the agent help the user achieve their goal), a retrievalQuality field (good, partial, empty, or wrong), a failureMode field, and freeform notes. Because it is content, the score inherits version history and permissions, and it lives next to the conversation it grades.

The reason to keep scores in the same backend as the content is that it makes the eval bench a living artifact instead of a one-time spreadsheet. When a surprising production conversation gets scored badly, you already have the transcript, the retrieval result, and the queried source in one place. You promote it into the bench without exporting anything. Observability tooling gives you the signal; scoring turns that signal into a verdict you can reason about, and structured scoring keeps the verdict close to the evidence.

Failure-mode tags that point to the layer, not the model

When an agent misbehaves, the reflexive fix is to edit the prompt and hope. That is how prompts turn into thousand-line apology notes that nobody can reason about. A better discipline attaches a failure-mode tag to every bad conversation and treats each tag as a pointer to a specific layer of the harness, so the fix lands where the problem actually lives.

The mapping is precise. Hallucination usually means retrieval returned nothing useful and the model filled the gap, so it is a retrieval problem, not a model problem. Tool-misuse means the prompt let the agent reach a tool that was wrong for the conversation type, a tools problem. Scope-violation means the agent answered something the never-say list should have caught, a prompt problem. Empty-retrieval is the structural ceiling of your retrieval layer and is the most common root cause. Prompt-drift means somebody shipped a prompt change without the eval bench catching the regression, which points at a missing eval gate. Each tag points to a layer, and the fix is at that layer.

Every failure then becomes a rule. The agent hallucinated a product feature? Add the fact to the retrieved content layer. It leaked a policy detail? Add it to the never-say list in the prompt. It burned thirty tool calls on one conversation? Add a budget to the loop. Over time every prompt line and every eval entry should trace back to a specific failure that happened once. Storing the scores as structured content next to the source the agent queries makes this loop tight: the same backend that serves the agent stores the conversations it had, the scores derived from them, and the content you edit to fix them.

Cost per conversation is the second governance metric

Governance is not only about correctness. It is also about spend, because an agent that answers well while quietly eating your budget is a risk you have to see. The second metric after success rate is cost per conversation, and it decomposes into token economics, model routing, and retrieval cache behavior. Model routing means sending a small model at cheap turns and reserving the large one for hard ones, which only works if you are measuring which turns are which.

The numbers are not small. Agent evals alone can run several times the usage of every other project on a team, because scoring conversations means running models over transcripts at volume. One quote from the field puts it plainly: it absolutely eats through credits. If you are not logging token counts per turn, per tool call, and per retrieval, you cannot see where the spend concentrates, and you cannot tell whether a prompt change that improved success rate tripled the cost of getting there.

This is where trace logging and scoring pay off again. Because the token counts are captured on every turn and the conversation is stored as content, cost per conversation is a query, not a spreadsheet you rebuild each month. You can filter to the conversations tagged with a given failure mode and see what the failures cost you, then prioritize fixes by dollars rather than by anecdote. When a budget cap and a routing change ship together, the eval bench tells you whether quality held while the cost record tells you whether the spend fell. Governing an agent means holding both lines at once, and that is only possible when both are recorded in the same place.

The eval bench is the verdict layer

Observability tools log every turn and let you score conversations against rubrics asynchronously. Braintrust does this for production scoring at scale, OpenTelemetry is the open standard the tracing code plugs into, and the Vercel AI SDK exposes streaming primitives that work with it. What none of these tools do is tell you whether the agent is right. That is the job of the eval bench. The tooling gives you the signal; the bench gives you the verdict.

An eval bench is a frozen set of representative conversations, roughly twenty to start, each scored against a rubric. The rule is simple and strict: any model change, prompt change, or tool change ships only when the bench stays green. When a surprising case shows up in production, you promote it into the bench so the same failure can never regress silently again. This is the gate that catches prompt-drift before it reaches a customer, and it is why prompt-drift is a governance failure rather than a modeling one.

The bench works best as a living artifact rather than a one-time export. Because the conversations, their scores, and the source content the agent queried can all live in one Content Lake, the bench is queryable content you extend in place, not a stale CSV somebody forgot to update. Observability sits alongside your harness, not inside it, and it hands you the raw signal. The bench, grounded in governed content, hands you the decision. Keeping both close means the day you swap models you are not guessing; you replay the bench, read the deltas, and ship or hold on evidence.

Auth-forwarded audit: logging the action against the user

Logging what the agent said is half the audit trail. The other half is logging what the agent did, and crucially, against whom the action is recorded. When an agent updates an account, issues a refund, or reads a restricted record, the audit entry cannot read as the model did this. It has to read as this user did this through the agent, or the trail is worthless the moment a regulator or a security team pulls it.

Auth-forwarding is the mechanism. The user's session token flows through the app, the agent runtime, and the tool layer to the backend API, so three things follow for free. Personalized retrieval, so the agent sees only what the user can see. Personalized action, so the agent does only what the user could do. Traceable audit, so the action is logged against the user, not the model. The side benefit is the important one for governance: the agent inherits your existing security model. Same row-level permissions, same rate limits, same regulatory boundaries. You do not build AI security as a separate discipline; you make sure the token flows.

Instrument the tools so failures are signal too. Structured errors tied to OpenTelemetry spans turn every catch block into something worth logging, so a denied action or a rate-limit trip is captured with the same fidelity as a success. Sanity governs the behavior layer as content as well: splitting the agent prompt into fields (role, voice, escalation, mustNotSay) is access control, and Content Releases let you stage agent behavior the way you stage a website, with drafts, version history, permission gating, audit trails, and rollback. For governance copy, Sanity carries SOC 2 Type II, GDPR, regional hosting and data residency, and a published sub-processor list.