Why Your CMS Should Own Embeddings, Not Outsource Them

Your search team ships a vector database. Six weeks later, an editor updates a product description, hits publish, and the semantic search index still returns the old copy. Nobody notices until a customer complains that the AI answer bot quoted a price that expired last quarter. The content changed. The embeddings did not. This is the default failure mode of the outsourced-embeddings architecture: your CMS owns the truth, a separate vector store owns the search, and a fragile sync pipeline stands between them, silently drifting out of date.

Sanity, the AI-native content platform, takes a different position. Sanity is the AI Content Operating System, an intelligent backend where embeddings live next to the content they describe, so semantic search stays fresh by construction rather than by cron job. The distinction matters because embeddings are not a bolt-on index. They are a derived property of your content, and derived properties belong where the content lives.

This article makes the case for CMS-owned embeddings over the bolt-on vector database: why the sync problem is architectural rather than operational, what freshness and governance actually require, and where the ownership line should sit for an AI CMS.

The sync problem is architectural, not operational

The standard advice when embeddings drift is to fix the pipeline: add retries, tighten the webhook, schedule a nightly reindex. This treats a structural flaw as an operational one. When your content lives in a CMS and your embeddings live in a separate vector database, you have two systems of record for the same underlying fact, and every write has to be reconciled across a network boundary. Reconciliation across a boundary is never free. It fails partially, it fails silently, and it fails most often on exactly the high-velocity content that matters: pricing, availability, launch copy, anything an editor touches under deadline.

Consider the lifecycle of a single edit. An editor changes a paragraph. The CMS commits it. A webhook fires to a queue. A worker picks it up, chunks the document, calls an embeddings model, and upserts vectors into the store. Any link in that chain can drop the event, and when it does, nothing in the search results announces the staleness. The vector is confidently wrong. Multiply this across thousands of documents and dozens of daily edits, and you are not running a search index, you are running a probabilistic estimate of what your content used to say.

The alternative is to stop treating embeddings as a downstream artifact. If embeddings are a derived property of content, tied to the document and regenerated when the document changes, the sync problem disappears because there is no second system to sync to. That is the mechanism behind Sanity's Embeddings Index API and dataset embeddings: the embedding is a function of the content in Content Lake, so freshness is automatic rather than aspirational. This is the first pillar, model your business, applied to search: the index inherits the model instead of duplicating it.

Freshness is a correctness property, not a nice-to-have

Teams underrate staleness because it degrades quietly. A slow API throws errors you can alert on. A stale embedding returns a plausible, well-formed, entirely outdated answer, and it does so with total confidence. For a marketing site that might be tolerable. For an AI answer bot grounded in your content, or a support assistant quoting policy, or an ecommerce search returning discontinued SKUs, staleness is a correctness bug with a customer-facing blast radius.

The half-life of freshness keeps shrinking. Content that used to change quarterly now changes weekly because AI-assisted editing lets small teams produce far more output. The more content you generate, the more edits you ship, and the wider the window in which an outsourced index can be wrong. Scaling output is supposed to be the win; a bolt-on vector store quietly converts that win into a growing surface of stale answers.

Freshness also has a temporal dimension the outsourced model handles badly: point-in-time correctness. When you stage a batch of edits for a launch, you want search to reflect the live state, not a half-applied migration where some vectors are new and some are old. Because Sanity keeps embeddings tied to content in Content Lake, and because Content Releases stage and schedule changes together, the index moves with the content as a unit. Live Content API subscriptions push fresh content to downstream LLM workflows the moment it changes, so retrieval reflects reality rather than a cron interval. Freshness stops being a pipeline SLA you monitor and becomes a property you inherit.

One query, not two systems: blended semantic and structured search

Real search is rarely pure vector similarity. Users want the semantically relevant result that is also in stock, in this locale, published, and within a price band. When embeddings live in a separate vector database, you get two half-answers: the vector store knows similarity but not your business rules, and your CMS knows the rules but not similarity. Stitching them together means querying both, paginating both, and reconciling two ranked lists in application code, which is where relevance quietly dies.

Ownership collapses this into a single operation. In Sanity, semantic similarity and structured filtering compose in one GROQ query: you can blend vector relevance with exact filters on the same documents, in the same round trip, because the embeddings and the fields live in the same Content Lake. There is no join across a boundary and no second pagination cursor to reconcile. The query language that already knows your schema also knows your embeddings.

This is the second pillar, automate everything, showing up as query composition rather than a workflow diagram. A bolt-on vector database can absolutely do fast nearest-neighbor search; that is not in dispute. What it cannot do is filter on business fields it never received, which pushes the filtering back into your code and reintroduces the two-systems problem at query time instead of write time. Owning embeddings means the filter and the similarity ranking are evaluated together, against one consistent view of the content, so relevance survives contact with your actual business rules.

Governance: embeddings inherit permissions or they leak

Every embedding is a compressed representation of a document, and a nearest-neighbor query is a retrieval mechanism. That means your vector index is a second copy of your content with its own access surface. When it lives in an external store, it also lives outside your CMS permission model. The document that only editors and legal can see now has a vector twin that a semantic search endpoint can surface to anyone who can query it. Retrieval-augmented generation makes this sharper: an LLM grounded on a permissionless vector store will happily quote content the asking user was never authorized to see.

Rebuilding your permission model inside a vector database is possible, but you are now maintaining authorization logic in two places, and drift between them is a security incident waiting to happen. The safer architecture is the one where embeddings never leave the governance boundary in the first place. Because Sanity keeps embeddings tied to content in Content Lake, the same Roles & Permissions, dataset boundaries, and Audit logs that govern the source content govern its embeddings and its retrieval. There is one access model, not two to keep in sync.

This is the shared-foundation differentiator in practice: legacy setups create silos, one for content and one for search, and every silo boundary is a place where governance can fall through. Sanity's compliance posture, SOC 2 Type II, GDPR alignment, regional hosting and data residency, and a published sub-processor list, extends to the derived index because the derived index is not a separate product living somewhere else. For any enterprise where search results can expose regulated or embargoed content, this is not a feature comparison; it is the difference between one auditable system and two that disagree.

Structure survives the pipeline: Portable Text and chunking

Embeddings are only as good as the text you feed the model, and the text you feed the model depends on how your content is structured. Flatten a rich document into an HTML blob and you lose the boundaries that make chunking sane: where a section starts, which words are a product name versus a link, what is a caption versus body copy. Bad chunks make bad embeddings, and bad embeddings make retrieval that is subtly, unfixably off.

Sanity stores rich text as Portable Text, a structured format where blocks, marks, and annotations are first-class data rather than tags in a string. That structure is exactly what a good chunking strategy needs: you can split on real semantic boundaries, preserve the metadata that annotations carry, and keep the association between a chunk and the fields around it. When retrieval hands a chunk to an LLM, the structure comes with it, so the model sees a coherent unit rather than a fragment torn out of markup. This is content modeled for machines to consume, not just for browsers to render.

An outsourced pipeline usually strips this on the way out. The export step serializes to plain text or HTML, the chunker guesses at boundaries, and the structural signal is gone before the embedding model ever sees it. You can invest heavily in a smarter chunker to recover some of it, but you are reconstructing information you already had and threw away. Owning embeddings next to structured content means the structure is available at embedding time, at retrieval time, and at generation time, which is where the App SDK and Agent Actions can act on schema-aware content rather than on a lossy shadow of it.

Where the ownership line belongs for an AI CMS

None of this argues that vector databases are bad technology. A dedicated store like Pinecone is genuinely excellent at large-scale nearest-neighbor search, high-dimensional recall, and specialized indexing at a scale most content workloads never reach. The question is not whether vector databases are good. The question is where the ownership line for embeddings should sit, and the answer follows the content.

For content-driven semantic search, the embedding is a derived property of a document that already lives in your CMS. Putting that derived property in a separate system creates a sync problem, a freshness problem, a governance problem, and a query-composition problem, all four of which are consequences of the same boundary. Drawing the line the other way, embeddings owned by the platform that owns the content, dissolves all four at once, because there is no boundary to reconcile across. You reach for an external vector database when your embedding workload genuinely outgrows the content platform, not by default at the first sign of semantic search.

This is why Sanity, the AI Content Operating System, treats embeddings as content infrastructure rather than an add-on. Legacy CMSes stop at publishing and leave search to bolt-on tools; Sanity operates content end to end, from model to embedding to retrieval to the LLM that consumes it. The third pillar, power anything, is the payoff: the same fresh, governed, structured content feeds your site, your search, and your agents through the Embeddings Index API, the Live Content API, and Sanity Context, without a second system of record silently drifting out of date behind them.