How to Implement Rate Limiting for LLM Content Generation

Your content team wires an LLM into the publishing pipeline to draft product descriptions, and three weeks later you get the invoice: a runaway loop re-generated the same 4,000 SKUs eleven times overnight because a retry handler had no backoff. Or worse, an editor triggers a bulk translation across 40 locales, the provider returns 429s halfway through, and half your catalog ships in English while the job silently dies. Rate limiting is the unglamorous plumbing that separates an LLM content workflow you can trust in production from one that burns budget, corrupts data, and pages you at 2am.

Sanity is the AI-native content platform built to keep these workflows governed, and its Content Operating System for the AI era treats generation as a first-class, reviewable operation rather than a fire-and-forget API call. That framing matters here: rate limiting is not just a networking concern, it is a content-governance concern about who can trigger what, how often, and with what review gate.

This guide reframes rate limiting for LLM content generation as a layered discipline. You will size limits against provider quotas and cost, enforce them at the workflow boundary with queues and backoff, and wire the whole thing into an editorial system that can stage, review, and roll back what the model produces.

Why LLM generation breaks the assumptions your CMS was built on

Traditional content operations were paced by humans. An editor writes an article, hits publish, and the system handles one write at a time at human speed. LLM generation inverts that: a single click can fan out into hundreds or thousands of concurrent model calls, each one metered, each one billable, and each one capable of failing independently. Your CMS was never designed to be a load generator against a third-party API with hard quotas and per-minute token ceilings.

The failure modes are specific. Providers enforce requests-per-minute and tokens-per-minute limits, and when you cross them you get 429 responses, not a graceful queue. Naive retry logic turns one rate-limit error into a retry storm that makes the problem worse. Bulk operations (translate every page, summarize every product, regenerate every meta description) are exactly the workloads that blow through quotas, because they batch thousands of calls with no natural pacing between them.

Then there is cost. Token-based pricing means an unthrottled loop does not just fail loudly, it can succeed expensively. A misconfigured job that regenerates content it already generated is invisible until the bill arrives. Under Sanity's first pillar, model your business, the fix starts in the data model: represent a generation job as content, with a status, a trigger source, and a token budget, so the system can reason about how much work is in flight before it dispatches the next call. Rate limiting becomes a property of your content model, not an afterthought bolted onto an HTTP client.

Layer one: size your limits against provider quotas and real cost

Before you write a single line of throttling code, you need numbers. Every LLM provider publishes rate limits along at least two axes, requests per minute and tokens per minute, and often a third, concurrent requests. These tiers change as your account matures, so treat them as configuration, not constants. Pull them into a settings document your team can update without a deploy.

Size your workflow limits below the provider ceiling, not at it. If your provider allows 3,500 requests per minute and 90,000 tokens per minute, budgeting to 60 percent of each leaves headroom for the interactive traffic your editors generate while a bulk job runs. Running both an AI Assist rewrite in the Studio and a Functions-driven translate-on-publish batch against the same provider key means they share a quota, and if you have not accounted for both, one starves the other.

Cost sizing is the second axis. Estimate tokens per operation (prompt plus expected completion), multiply by the number of items in a bulk run, and set a hard token budget per job. A job that would exceed the budget should refuse to start, not discover the limit mid-run. This is where representing the job as content pays off: the budget, the estimate, and the running total all live on the job document, queryable with GROQ, visible in the Studio, and auditable after the fact.

Layer two: enforce with queues, concurrency caps, and exponential backoff

Sizing tells you the ceiling. Enforcement keeps you under it. The core pattern is a queue with a bounded worker pool. Instead of firing every generation call the moment a bulk job starts, you enqueue units of work and let a fixed number of workers pull from the queue, so concurrency never exceeds a value you chose deliberately. Concurrency caps are the single most effective control because they turn an unbounded fan-out into a steady, predictable stream.

Pair the queue with a token-bucket or leaky-bucket limiter to smooth the rate over time. A token bucket refills at your target requests-per-minute and drains one token per call, so bursts are absorbed up to the bucket size and sustained throughput settles at the refill rate. This is what keeps a 5,000-item job from slamming the provider in the first ten seconds.

When a 429 does arrive, and it will, respect it. Read the provider's retry-after header when present, and otherwise apply exponential backoff with jitter: wait, double the wait on each failure, and add randomness so a fleet of workers does not retry in lockstep. Cap the retries and route exhausted items to a dead-letter state rather than looping forever. In Sanity, Functions give you serverless hooks (translate-on-publish, enrich-on-publish, moderate-on-publish) that are natural places to house this logic, and because a Function runs per document change, you can pace the fan-out at the source instead of letting an external script hammer the API. Agent Actions, being schema-aware, let the generation call itself validate against your content model, so a throttled retry never writes a malformed document.

Layer three: govern who can trigger generation and how much

Rate limiting is not only about protecting the provider. It is about protecting your organization from itself. An intern with the ability to trigger a regenerate-all-descriptions job is a bigger risk than any 429. Governance closes that gap by scoping who can start expensive operations, how large those operations can be, and whether they require review before anything ships.

The controls are organizational and technical at once. Roles and Permissions decide which editors can invoke bulk generation versus single-document assists. A per-user or per-role quota (this account may trigger N generation jobs per day, or M total tokens) prevents a single actor from consuming the shared budget. And an approval gate on large jobs means a run above a threshold enters a pending state that a lead must release.

This is where the fifth differentiator lands: rigid CMSes force you to scale people to keep up with content demand, while Sanity scales output by letting automated generation run under governed limits. Content Releases let LLM-generated changes stage together, get reviewed as a set, and schedule or roll back atomically, so a throttled bulk translation lands as one reviewable release rather than 40 loose edits trickling into production. Combined with Audit logs, you get a record of who triggered what, when, and against which budget, which is exactly the evidence a compliance review asks for. Sanity carries SOC 2 Type II, GDPR alignment, regional hosting for data residency, and a published sub-processor list, so the governance story extends past your own limits to the platform underneath them.

Make rate-limit events observable, not silent

The worst rate-limiting bug is the one you never see. A bulk job that quietly drops 30 percent of its items to 429s and marks itself complete is more dangerous than one that crashes, because the crash pages you and the silent failure ships a half-translated catalog that nobody notices until a customer does. Observability is the difference between a limit that protects you and a limit that hides damage.

Instrument three things. First, per-job counters: items attempted, succeeded, retried, and dead-lettered, all stored on the job document so they survive the process that produced them. Second, rate-limit telemetry: how often you hit 429s, which limit (requests or tokens) you tripped, and the average backoff duration, so you can retune your sizing next run. Third, alerts on the states that matter, a job entering dead-letter, a budget exceeded, or a run stalled longer than expected.

Because Sanity's Content Lake exposes real-time subscriptions, a monitoring surface can watch job documents change and react the moment a run stalls or overruns its budget, rather than polling on a timer. The Studio itself becomes the dashboard: editors see a job's progress, its retry count, and its dead-lettered items in the same interface where they review the output, so the person who triggered the work is the person who sees it fail. That closes the loop between the automation and the human accountable for it, which is the whole point of treating generation as governed content rather than a background script.

Put it together: a governed generation pipeline end to end

Trace a real bulk operation through all three layers. An editor requests new meta descriptions for 3,000 products. First the governance layer checks their role and the job size against the daily quota; the run exceeds the auto-approve threshold, so it enters a pending release for a lead to confirm. On approval, the sizing layer estimates tokens (3,000 items times an average prompt-plus-completion cost) and confirms the job fits the token budget, refusing and flagging it if it would not.

Enforcement takes over. The 3,000 items enter a queue drained by a bounded worker pool capped at, say, eight concurrent calls, metered by a token bucket refilling at 60 percent of the provider's tokens-per-minute. A 429 triggers exponential backoff with jitter; three exhausted retries send an item to dead-letter. Throughout, per-job counters update on the job document in the Content Lake, and a real-time subscription drives the Studio progress view.

The output does not go straight to production. Generated descriptions land in a Content Release, staged as structured Portable Text so the rich-text structure survives review and any downstream LLM reuse. The editor reviews the batch, the dead-lettered items are re-queued or handled by hand, and the release schedules or publishes atomically. This is Sanity's second pillar, automate everything, expressed concretely: the model does the work, the limits keep it safe, and the editorial system keeps a human in the loop. AI is wired into the data model, the editor, and the delivery layer, not bolted on with a plugin, which is why the rate-limiting logic and the review gate live in the same governed system instead of in a fragile external script nobody owns.