Rate limits and cost budgets for AI features in production

A meal-planning app adds a "plan my week" button backed by a cloud model. In testing, a request costs a fraction of a cent. Three weeks after launch, a prompt change adds the user's full pantry and recipe history to the context, a few hundred users tap the button several times a day, and one account turns out to be a script replaying the request all night.

None of those events is unusual. Together they produce a bill that is larger than the app's revenue for the month, and nobody noticed until the invoice arrived.

AI features have a marginal cost per use that most app features do not. A list screen costs the same whether the user scrolls once or a hundred times. A model call costs money every time, and the amount depends on input you may not control. That needs to be designed for before launch, in the same place you design authentication and error handling.

1. Measure what one request costs

You cannot set a sensible limit without knowing what a request costs at the median and at the tail.

Most hosted models bill input and output tokens separately, often at different rates. For each AI operation, log:

• the operation name, such as summarizeNote or planWeek
• the model and provider route
• input tokens and output tokens as reported by the provider
• the computed cost at your current price table
• a pseudonymous user identifier and plan tier
• latency and outcome

Log token counts and cost only. Keep prompt content out of this pipeline; it has different privacy rules.

After a week, look at p50 and p95 cost per operation. The tail is usually driven by context size: long documents, retrieved history, or a conversation that keeps growing. That is where most of the money goes, and it is also where a per-request cap does the most good.

Set an explicit output limit on every call. Leaving max_tokens at the provider default means the model decides how expensive the response is.

2. Keep keys and enforcement on the server

An API key shipped inside the app binary should be treated as public. Extracting it takes an afternoon, and from then on your quota belongs to whoever has it.

Route model calls through your own backend or a proxy that holds the provider key. The client asks for an operation; the server decides whether to run it, builds the request, and calls the provider. Any limit enforced only in the app is a suggestion that a modified client can ignore.

The server then needs to know who is calling. A signed-in account is the simplest answer. For apps that allow anonymous use, Apple's App Attest lets your server check that requests come from a legitimate instance of your app on a real device. It raises the cost of scripted abuse. It does not tell you who the user is, and a determined attacker with real devices can still generate traffic, so per-identity limits still apply.

Give the client an operation name and its parameters, never a free-form prompt it can fill. An endpoint that accepts arbitrary prompts and forwards them with your key is a general-purpose model API that you pay for.

3. Budget by operation, in cost units

A single requests-per-minute limit across all AI endpoints treats a one-line rewrite and a week-long meal plan as the same thing. They can differ by 50x in tokens.

Define limits per operation and count in tokens or cost units instead of requests. A compact policy might look like this:

summarizeNote:
  max input tokens: 6,000
  max output tokens: 400
  burst: 10 requests / minute / user
 
planWeek:
  max input tokens: 12,000
  max output tokens: 2,000
  burst: 3 requests / minute / user
 
Daily allowance (all operations, cost units):
  free: 50
  pro: 1,000
 
Global ceiling:
  alert at 150% of trailing 7-day hourly average
  stop non-essential operations at daily hard cap

The numbers are illustrative. The policy has four layers: a per-request cap, a short-window burst limit, a per-user allowance tied to the plan, and a global ceiling that protects the business when everything else fails.

Reject oversized input before calling the model. Truncating silently produces worse answers that users report as quality bugs.

4. Reserve before the call, settle after

Output tokens are unknown until the response finishes, and users fire concurrent requests. A counter that increments after each call lets ten parallel requests through when the user had budget for one.

Use a reserve-and-settle pattern:

1. Estimate the maximum cost from the input size and the output cap.
2. Atomically reserve that amount against the user's allowance. Reject if it does not fit.
3. Call the provider.
4. Settle with the actual usage and release the difference.
5. If the call fails before the provider charges anything, release the full reservation.

The reservation has to be atomic in whatever store holds the counters. A read-check-write sequence across two round trips has the same race as no reservation at all.

Streaming responses need the same treatment. If the user cancels mid-stream, settle what the provider reports for the partial response; the tokens already generated are usually billed.

5. Tie allowances to entitlements the server verifies

Paid plans usually come with larger allowances. The server should decide the tier from verified purchase state, never from an isPro flag the client sends.

Check subscription status on the backend through the App Store Server API, or through your billing service's server-side API and webhooks. Cache the result with a short expiry so a refund or lapsed subscription takes effect within minutes, and so a slow entitlement lookup does not sit in front of every model call.

Size the free allowance with the product in mind as well as the cost. A free tier that hits a hard wall on the second request teaches users that the feature is a paywall trap. A small allowance that lets someone complete the core task a few times shows them what they would be paying for. That is a pricing decision; make it deliberately and write down what it costs per free user per month.

6. Degrade on purpose

Every limit needs a designed response. A spinner that never resolves, or a generic "Something went wrong", turns a budget control into a support ticket.

When a user is close to their allowance, say so before they start a long operation. When they hit it, show when it resets and what they can still do. Reasonable options include:

• a smaller or cheaper model for the same operation
• an on-device route, if the feature has one
• a manual path through the same screen
• an upgrade offer, when the user is on a plan where that makes sense

A cheaper fallback still has to respect the user's privacy choices. If they agreed to send a note to one provider, a quota-driven switch to a different provider needs the same disclosure and consent as the original route.

Provider rate limits are a separate case from your own. An HTTP 429 from the provider means you are over its quota. Retry with exponential backoff and jitter, honor any Retry-After header, and cap the attempts. Every retry of a request that reached the model can cost money, so do not let a retry loop in the client stack on top of a retry loop in the server.

7. Watch spend like an error rate

Spend belongs on the same dashboard as crash rate and latency, with alerts on it.

The useful views are:

• cost per operation per day
• cost per active user, split by plan
• the top users by spend over the last 24 hours
• average input and output tokens per operation, to catch context growth

Alert on rate of change. An hourly spend that is three times the trailing average is worth waking up for, even if the absolute number is small. Set a hard spending limit in the provider console as well, where the provider supports one. It is a blunt last line of defense, but it is outside your own code and survives your bugs.

Treat prompt and model changes as cost changes. Adding retrieved context to a prompt can double the input tokens of every call. Ship those changes behind a remote flag, compare cost per operation before and after on a small cohort, and keep a kill switch per operation so you can turn off one expensive feature without disabling the rest of the app.

8. Test the limits before users do

Quota code is easy to write and easy to get subtly wrong. Test it the way you would test payment code.

Cover at least these cases:

• concurrent requests from one user that together exceed the allowance
• a provider failure after reservation, which must release the budget
• cancellation mid-stream, which must settle partial usage
• an expired subscription that should drop the user to the free tier
• the global ceiling tripping while requests are in flight
• a provider 429 with and without Retry-After

Run a scripted client against a staging backend that uses a mock provider, and confirm that the limits hold and the counters reconcile with the request log. Then check the client: every limit response should map to a specific screen state with its own copy.

If I had to pick one rule, it would be that no AI call reaches a provider without an operation name, an owner, and a budget the server enforces. Apps that skip it tend to find out their limits from the invoice.