Rate Limiting AI Features: Stop One User From Draining Your Budget

Rate-limit AI features at the user, session, and endpoint level, not just by trusting your LLM provider's account-wide API limit. A single compromised account, a scraping bot, or a client-side bug that loops on retries can turn a predictable monthly bill into a five-figure surprise before your spend alert even fires.

Most teams find out about this the hard way. They ship an AI feature, wire it to OpenAI or Anthropic, set a spend notification, and assume that's coverage. It isn't. Spend alerts are a smoke detector, not a sprinkler system: they tell you the kitchen is on fire, they don't put it out. By the time you get the email, the damage is already on your card.

Why provider-level limits aren't enough

Every LLM provider ships rate limits, but they're scoped to your whole account, not to your individual users. OpenAI's rate limit docs describe requests-per-minute and tokens-per-minute ceilings that apply globally to your API key. That protects the provider's infrastructure, not your business. If one user in your app fires 500 requests in a loop, that traffic counts against the same bucket as every other legitimate user, and it's your product that degrades or your bill that spikes, not theirs.

This isn't hypothetical for us. Our own outreach tooling scrapes each prospect's site with a self-hosted Firecrawl instance and a local model, then makes one LLM call that extracts the relevant facts and drafts the email. If that call got stuck in a retry loop, or a bug fed it the same prospect list twice, OpenAI's account-level limit wouldn't stop it — it would just start throttling every other request on that API key, including unrelated ones, while the loop kept burning tokens underneath. The provider's ceiling protects OpenAI's servers, not our budget or our other traffic.

This matters more for AI features than for typical API endpoints because the cost-per-request variance is enormous. A normal REST call costs fractions of a cent regardless of what the user does. An LLM call can cost 50x more depending on prompt length, context window, and whether the model reasons through multiple steps. A chatbot that lets users paste arbitrary text, or an agent that can call tools in a loop, has a much wider blast radius than a typical CRUD endpoint.

What to actually rate-limit

Think in layers, not one number:

• Per-user request rate: requests per minute per authenticated user, using something like a token bucket so short bursts are allowed but sustained abuse isn't.
• Per-user token spend: cap total tokens (input + output) per user per day or per billing cycle, separate from request count, since one long prompt can cost more than a hundred short ones.
• Per-session tool calls: if you're running an agent that can call tools or make sub-requests, cap the number of steps per session. This is exactly why our outreach pipeline is deliberately a single call — scrape, then one shot that extracts facts and drafts the email — instead of a multi-step "extract, then summarize, then draft" chain. Every extra step in a chain is another place a stuck loop can start; a one-call design gives a runaway process nowhere to hide. A loop that calls a tool 40 times because it never finds a stopping condition is a common and expensive failure mode in chains that don't have that discipline.
• Per-IP and per-device fallback: for unauthenticated or free-tier surfaces, add a secondary limit that doesn't depend on account identity, since that's the first thing an abuser will fake.

Enforce all of this server-side, in the same service that holds your provider API key. Client-side throttling is a UX nicety, not a security control. If your rate limiting logic is checkable and bypassable from the browser, it isn't protecting your budget.

Build it as a gate, not an afterthought

The cleanest pattern is a spend and rate gate that sits between your application and the LLM call, so every request, regardless of which feature triggered it, passes through the same check. That gate should track:

1. Current usage against the user's limit (Redis with a sliding window or token bucket works well for this)
2. A hard ceiling that returns a clear error rather than silently queuing or retrying
3. A way to raise limits for specific verified users without redeploying

Retries deserve special attention. A naive retry-on-failure wrapper around an LLM call, combined with a rate limit error, can spiral: the request fails, the client retries, the retry also gets rate-limited, and now you have a thundering herd hitting your gate every second. Back off exponentially and cap total retries per request, not just per minute.

If you're also juggling multiple providers for cost or reliability reasons, the same gate is the right place to enforce limits before routing, which is one more reason to centralize this logic rather than sprinkling checks across every AI endpoint. We've covered the tradeoffs of that routing layer separately in our piece on LLM cost optimization and in multi-LLM provider failover.

A pattern that also cuts cost, not just risk

The cheapest request is the one you don't make. In our own outreach tooling, we scrape each prospect's site with a self-hosted Firecrawl instance and a local model, and we found that a single call that extracts the relevant facts and drafts the email in one shot beats a multi-step chain of "extract, then summarize, then draft" on both cost and latency. Fewer calls per unit of work means your rate limits are naturally harder to hit and your ceiling per user goes further. Before you reach for aggressive throttling, check whether your feature is making more LLM calls than it needs to accomplish the task. Rate limiting caps the damage from a bad actor; call-count discipline shrinks the damage a normal user can do just by using the product as intended.

Where this fits with vendor keys

If you're using a managed AI vendor rather than your own provider keys, ask them directly how they isolate one customer's traffic from another's, and whether limits are configurable per account or fixed. This is one of the questions worth adding to any vendor conversation, alongside the pricing model questions we outline in BYOK vs managed LLM keys. If a vendor can't answer clearly, assume the isolation doesn't exist and plan your own gate on top regardless.

The minimum you should ship before launch

If you only do three things before an AI feature goes live: put a per-user token cap in front of every LLM call, cap tool-call loops in any agentic flow, and make sure retries back off instead of compounding. None of this requires exotic infrastructure, a Redis instance and a few dozen lines of middleware cover most products. It's cheap insurance against the one failure mode that turns an otherwise well-run AI feature into a budget line item you have to explain.

If you're scoping an AI feature and want a second opinion on the guardrails before it ships, let's talk.