Stop runaway workloads
Enterprise Tier
A rate limit holds one consumer to a token ceiling per hour, so a runaway workload hits a wall at a chosen volume instead of at three weeks of spend. Ceilings are set through the management API on a project or an API key.
A rate limit today is a token ceiling per hour. The 2026 Q4 release adds request ceilings, per-minute windows, and the Admin Console screen that sets both on a project or an API key. Those are documented ahead of release in Set rate limits on projects and keys. Everything on this page is available now.
A rate limit is flow control, not a spend cap. It protects shared capacity by bounding the volume of traffic; it does not stop a consumer from spending a budget over the course of a month. The two are complementary, and choosing between them is covered in Choose the right cost control for each workload; the spend caps themselves are set in Set a budget and track spend against it.
Persona: Platform operator working through the management API, with the Console's usage and traffic view (the developer-side monitoring surface) for confirming the effect.
Estimated time: 15 to 25 minutes for an initial pass; ongoing as workloads evolve.
When this guide applies
Rate limits are the right concern in any of these situations:
| Situation | Control that fits |
|---|---|
| A consumer drives large token volumes (long contexts and verbose completions) that strain upstream capacity | An hourly token ceiling on its key |
| No single key in a project should take more than its share, and keys come and go | An hourly token ceiling on the project |
| A new or untrusted integration should be bounded before its real traffic shape is known | A deliberately tight ceiling at the key scope |
| A retry loop or runaway agent is sending far more requests than a healthy client would | A token ceiling bounds the damage; ending the loop means addressing the key, as in Contain a leaked key before it drains the budget. Request ceilings arrive in the 2026 Q4 release |
| Spend over a month should be capped regardless of traffic volume | A budget, not a rate limit. See Set a budget and track spend against it |
Outcomes
By the end of this guide:
- The available counters (input, output, or total tokens per hour) and subjects (an API key or a project) are understood.
- At least one rate limit is applied at the subject appropriate to the workload it protects.
- The caller-side experience of a hit limit, an HTTP
429response, is understood, along with the back-off behavior applications are expected to implement. - A way to confirm whether limits are being hit is in place.
Prerequisites
- Administrator access to the management API: typically the
super_adminrole, or a role with permission to manage limits. - At least one provider and model already provisioned. See Provision models and providers.
- Coordination with the developer teams that own the affected keys. Limits are enforced inline, so a misconfigured limit produces visible production impact.
Step 1: choose the counter
A rate limit counts tokens over an hour. The counter can cover input tokens, output tokens, or the two together.
| Counter | What it caps |
|---|---|
| Input tokens per hour | Prompt volume: long contexts and large documents |
| Output tokens per hour | Completion volume: verbose or unbounded generation |
| Total tokens per hour | Both together, as one figure |
Total tokens is the general-purpose ceiling. The separate input and output counters fit workloads whose load is lopsided: a summarization pipeline that sends large prompts and returns short answers is bounded by its input volume, and a generation workload by its output.
There is no request counter and no per-minute window today; both arrive in the 2026 Q4 release. A flood of small calls accumulates tokens slowly, so a token ceiling catches a retry storm late. Where the runaway is a misbehaving or compromised key rather than a legitimately heavy workload, the faster tool is containment.
Step 2: choose the subject
The subject determines which traffic the ceiling is measured against. The subjects used for create today are a project and an API key.
| Subject | What the limit governs | Typical use |
|---|---|---|
| An API key | All traffic presented with one key | Bounding a specific application or integration to its expected envelope |
| A project | Traffic under one project | Bounding a piece of work whose keys share a project |
A project limit is each key's limit, not the project's total. A project set to 100,000 tokens an hour gives every key in that project 100,000 tokens an hour of its own. Where several limits cover one key, the smallest number per counter is the one in force.
Tag and organization scopes are available through the management API only. They are not the subjects this guide configures, and the Admin Console screen arriving in the 2026 Q4 release offers a project or an API key. Limits already set on a tag continue to apply to the keys they match.
Separately from operator-set policies, each API key can also carry Console-side hourly token limits (Total, Input, and Output tokens per hour, counted in fixed hourly windows aligned to the UTC clock), configured by the developer who owns the key. That key-owner control is documented in Route requests across providers under What the API Keys page shows. It does not replace an administrator rate-limit policy, and a developer cannot see or change the administrator policies that cover the key.
Step 3: set the limit through the management API
Rate limits are created and adjusted through the management API. There is no Admin Console screen for them until the 2026 Q4 release. The operations are documented in the Policy API reference, starting with CreateRateLimitPolicy. Two behaviors of the current API are worth knowing:
- The API accepts a
periodvalue of daily, weekly, or monthly, and the gateway ignores it: the count is always kept per hour. - A limit left unset for a given subject is not enforced for that subject. A key with no ceiling of its own is still covered by any project ceiling (or any tag or organization ceiling set through the API) that applies to it.
A changed limit reaches the gateway the way any configuration change does; the delivery window is described in Configuration propagation.
Step 4: size and adjust limits
Sizing is the part of this work that takes the most judgment. Two failure modes are worth avoiding.
- Limits set too tight. Normal traffic hits the ceiling, healthy clients start receiving
429responses, and the workload looks, from the consumer's side, as though Agent Router is failing. The cost of this failure mode is immediate and visible. - Limits set too loose. A runaway consumer is never actually constrained, and the limit becomes a number that never fires. The cost surfaces later, as overloaded upstreams or a degraded experience for other consumers sharing the same capacity.
The dependable approach is to size each limit from observed traffic. The expected peak is read from the usage surface, see Monitor traffic and usage, the highest legitimate hourly token volume over a representative window is taken as the baseline, and the limit is set comfortably above that baseline but well below any level that would constitute overload. Setting a limit at roughly two to three times the observed peak is a common starting point, adjusted for how bursty the workload is.
For a brand-new workload with no traffic history, a deliberately tight limit is the safer starting point. Real traffic surfaces the true shape quickly, and the limit is relaxed once the baseline is known. Relaxing a limit that proved too tight is a low-risk adjustment; discovering that a generous limit allowed an overload is harder to recover from.
Some workloads are quiet most of the time and very loud occasionally: end-of-month batch runs and scheduled report generation. A limit sized for the quiet baseline fires on the burst. The cleanest answer is usually to isolate the bursty work on its own key with its own, more generous limit, which keeps the steady-state key tight and the usage attribution clean. The operational complication of issuing a second key is typically smaller than the complication of explaining why a single key's traffic shape is irregular.
Step 5: monitor whether limits are being hit
A limit that never fires and a limit that fires constantly are both worth knowing about: the first may be set too loosely to matter, and the second is likely throttling healthy traffic. Both are visible in Agent Router's usage and traffic surfaces.
The signal to watch for is the rate of 429 responses, broken down by the subject the limit is set on. A workflow that pairs well with this guide:
- Open the usage and traffic view. See Monitor traffic and usage.
- Apply a time range that matches the cadence of the workload under review.
- Break the traffic down by the subject the limit is set on: by key or by project.
- Look at the proportion of requests returning
429. A small, occasional fraction during peaks is normal; a sustained high fraction indicates the limit is too tight for the legitimate workload.
When a limit is firing more than expected, the choices are to raise it if the traffic is legitimate, see Step 4, or to address the consumer if the traffic is not, by working with the owning team or, in the case of a suspected-compromised key, following Contain a leaked key before it drains the budget.
What a caller experiences when a limit is hit
When a request would breach an active limit, the gateway rejects it with an HTTP 429 Too Many Requests response rather than forwarding it upstream. The rejection applies only to the request that crossed the threshold; once the window advances, traffic flows again. A 429 is therefore a transient, recoverable signal, not a hard failure, though to a client that does not handle it, it presents the same way as an error.
The 429 does not name the limit that fired. The response may carry retry-after guidance, but it does not say which counter or which subject produced the refusal. A blocked budget can also return 429, so the same status does not by itself separate a rate limit from an exhausted spend cap.
Applications are expected to back off and retry rather than fail or retry immediately:
- Retry the request after a short delay rather than treating the
429as fatal. - Use exponential back-off with jitter (increasing the delay on each successive
429, with a small random offset) so that many clients hitting the limit at once do not retry in lockstep and re-create the burst. - Respect any retry-after guidance the response carries, where the client library surfaces it.
- Cap the number of retries so that a sustained limit does not turn into an unbounded retry loop, which is itself a source of the pressure the limit exists to contain.
Where to go next
Set rate limits on projects and keys
The request ceilings, per-minute windows, and Console screen arriving in the 2026 Q4 release.
Set a budget and track spend against it
The spend cap that complements the flow control established here.
Contain a leaked key before it drains the budget
The incident response when the runaway is not legitimate traffic.
Monitor traffic and usage
Watch the effect of the limits on live traffic.