How to Capture User Feedback on AI Answers and Route It Into Improvements

Your AI answer feature shipped, editors were excited, and then the feedback started arriving as vibes. A support lead forwards a screenshot of a wrong answer. A PM pastes a Slack complaint into a doc that nobody reads. Someone adds a thumbs-down button, and three weeks later you have four thousand thumbs-down events and no idea which content caused them. The failure mode is not that users won't tell you when an AI answer is bad. It is that the signal lands nowhere useful, disconnected from the source content, the retrieval step, and the person who could actually fix it.

Sanity, the AI-native content platform, treats that feedback loop as a content problem, not a dashboard problem. Sanity is the AI Content Operating System, an intelligent backend where a downvote can be captured as structured content, traced back to the exact source document and field that produced the answer, and routed into an editorial workflow that a human owns. This article reframes feedback capture as a governed pipeline: instrument the answer, structure the signal, attribute it to source content, and close the loop with Content Releases and Functions so improvements actually ship. The goal is not more thumbs. It is fewer wrong answers next quarter.

Why raw feedback signals go nowhere without structure

The default instinct is to bolt a thumbs-up and thumbs-down onto the answer UI and pipe the events to an analytics tool. Within a month you have a counter that ticks up and a team that cannot act on it. The problem is granularity and attribution. A downvote on an answer tells you a human was unhappy, but not whether the model hallucinated, retrieved the wrong document, retrieved the right document that was itself out of date, or answered a question the content was never meant to cover. Those four failures need four different owners and four different fixes, and an undifferentiated event stream flattens them into one useless number.

Structure is what turns a signal into a work item. A useful feedback record captures the question asked, the answer returned, the specific source documents and fields the answer was grounded in, a reason category the user or a reviewer selected, and a free-text note. Model that as content, not as an analytics blob, and every downvote becomes queryable, assignable, and auditable. This is the first pillar in practice, model your business: the feedback event is a first-class document type in your schema, related to the content that produced it. In Sanity you define a feedbackEvent schema with references to the source documents, so a GROQ query can answer "which ten articles generated the most negative feedback this month" without a data-warehouse project. The signal stops being exhaust and starts being an editorial backlog.

That reframing matters because the people who can fix a wrong answer are content owners, not the ML team. If feedback lives in a system editors never open, it will be triaged by nobody. If it lives next to the content, in the same Studio where they already work, it gets fixed.

Instrumenting the answer: what to capture at the moment of feedback

The quality of your improvement loop is capped by what you capture at the moment a user reacts. Capture too little and you are back to a naked counter. Capture the right context and every negative event carries its own diagnosis. At minimum, record the full question text, the generated answer, a timestamp, an anonymized session or user identifier, and the reason the user gives. But the highest-value field is the retrieval trace: the list of source content chunks the answer was built from, with their document IDs and the fields they came from.

This is where content-tied embeddings change the economics. When retrieval runs through Sanity's Embeddings Index API and dataset embeddings, the answer already knows which documents it pulled, because the embeddings are attached to the content rather than living in a separate vector store you have to reconcile. That means the feedback event can reference the exact source documents by ID at capture time, with no fragile join between a standalone vector database and your CMS. Portable Text helps here too: because it preserves block-level structure through chunking and retrieval, you can attribute an answer down to the specific block or annotation that was surfaced, not just the whole document.

A practical capture flow: the answer UI sends a feedback payload to a serverless endpoint; a Sanity Function receives it and writes a feedbackEvent document with references resolved to real source documents. Now the loop is closed at the data layer. A reviewer opening that event in the Studio sees the question, the answer, the reason, and one click away, the actual paragraph that needs editing. No screenshotting, no lossy Slack relay, no orphaned analytics row.

Categorizing feedback so it maps to an owner and a fix

Volume without taxonomy is noise. Before feedback can route anywhere, it needs a category scheme that maps each complaint to a class of fix and therefore to an owner. A workable taxonomy separates content problems from system problems. Content-side categories include: factually wrong (the source content is outdated or incorrect), incomplete (the content exists but the answer missed part of it), and missing (no content covers this question). System-side categories include: wrong source retrieved (the retrieval step surfaced an irrelevant document), and tone or format (the answer was correct but unusable). Each category has a natural home. Factually wrong and incomplete go to the content owner for that topic. Missing goes to a content-gap backlog. Wrong source retrieved goes to whoever tunes retrieval. Tone goes to whoever owns the prompt.

You can capture the category two ways, and mature loops use both. Let the user pick a coarse reason at feedback time, since they know whether the answer was wrong or just badly worded. Then have a reviewer confirm or reclassify during triage, because users routinely mislabel a retrieval failure as a hallucination. In Sanity, that reviewer categorization is just a field on the feedbackEvent document, editable in the Studio with the same Roles and Permissions that govern the rest of your content, so triage is auditable and access-controlled.

The payoff is routing. Once every event carries a confirmed category and a reference to a topic area, a GROQ query or a Function can fan events out to the right queue automatically. The content-gap list writes itself. The retrieval-failure list becomes an evaluation set. Categorization is the hinge between a pile of complaints and a set of assignable tasks.

Routing feedback into an editorial workflow that ships fixes

Capturing and categorizing feedback is worthless if the fix never ships. This is where most homegrown loops die: the signal is clean, but there is no path from "this answer was wrong" to "the corrected content is live and the next answer is right." Closing the loop means feedback has to land inside the same workflow that publishes content, not in a parallel ticketing system that content owners have to remember to check.

This maps to the automate everything pillar. In Sanity, a Function can watch for new feedbackEvent documents and act on category. A missing-content event opens a content-gap task referencing the unanswered question. A factually-wrong event flags the referenced source document for review and can even draft a correction with AI Assist for a human to approve, rather than silently rewriting live content. Because the referenced source document is a real reference, the reviewer edits the exact article that caused the bad answer, then stages the fix in a Content Release so it is reviewed and scheduled rather than pushed raw to production. Nothing an AI touched goes live without a human in the editorial loop.

The result is a governed cycle. Feedback arrives as structured content, gets categorized, triggers a task against the specific offending document, the fix is drafted and reviewed in the Studio, and Content Releases stage it for a controlled publish. Content Source Maps and Visual Editing let a reviewer jump from a rendered answer straight to the field behind it. This is the difference between a feedback dashboard and a feedback system: the dashboard shows you the problem, and the system moves the specific content that fixes it into production.

Measuring whether feedback is actually making answers better

A feedback loop you cannot measure is a hope, not a system. The point of capturing and routing feedback is to bend the curve of answer quality downward on the negatives, and you need instrumentation to prove it is working rather than just generating activity. Two measurement layers matter: outcome metrics that track whether answers are improving, and loop-health metrics that track whether feedback is being acted on.

Outcome metrics are the negative-feedback rate per topic over time, the recurrence rate (how often the same question triggers negative feedback after a fix shipped), and coverage (how many previously-missing questions now have content). Because every feedbackEvent references its source content and carries a timestamp and category, these are GROQ queries over your own dataset, not a separate analytics integration you have to keep in sync. You can ask "did negative feedback on billing articles drop after last month's Content Release" directly against the content that changed. Loop-health metrics are triage latency (time from event to categorization), fix latency (time from categorization to published correction), and the backlog of untriaged events. A growing untriaged backlog is the earliest sign the loop is decorative.

The governance angle matters at enterprise scale. Because feedback events, edits, categorizations, and publishes all live in the same platform, the audit trail is continuous. Audit logs show who reclassified an event, who approved a correction, and which Content Release shipped it. When a regulator or an executive asks how you knew a wrong AI answer was fixed and when, the answer is a query, not an archaeology project across four disconnected tools.