Stop runaway workloads

Enterprise Tier

A rate limit holds one consumer to a ceiling on requests or tokens per minute or per hour, so a runaway workload hits a wall at a chosen rate instead of at three weeks of spend. Ceilings are set on a project or an API key, in the Admin Console or through the management API.

---

A rate limit is flow control, not a spend cap: it protects shared capacity by bounding the volume of traffic, and it does not stop a consumer from spending a budget over the course of a month. The two are complementary, and choosing between them is covered in Choose the right cost control for each workload. Worked scenarios, including how two limits on the same key resolve, are in Set rate limits on projects and keys.

Persona: Platform operator working in the Admin Console or through the management API, with the Developer Console's usage and traffic view for confirming the effect.

Estimated time: 15 to 25 minutes for an initial pass; ongoing as workloads evolve.

When this guide applies

Rate limits are the right concern in any of these situations:

Situation	Control that fits
A consumer drives large token volumes (long contexts and verbose completions) that strain upstream capacity	A token ceiling on its key
No single key in a project should take more than its share, and keys come and go	A ceiling on the project
A new or untrusted integration should be bounded before its real traffic shape is known	A deliberately tight ceiling at the key scope
A retry loop or runaway agent is sending far more requests than a healthy client would	A request ceiling per minute bounds the damage; ending the loop means addressing the key, as in Contain a leaked key before it drains the budget
Spend over a month should be capped regardless of traffic volume	A budget, not a rate limit. See Set a budget and track spend against it

Outcomes

By the end of this guide:

• The available counters (requests or tokens, per minute or per hour) and subjects (an API key or a project) are understood.
• At least one rate limit is applied at the subject appropriate to the workload it protects.
• The caller-side experience of a hit limit, an HTTP 429 response, is understood, along with the back-off behavior applications are expected to implement.
• A way to confirm whether limits are being hit is in place.

Prerequisites

• Administrator access to the Admin Console or the management API: typically the super_admin role, or a role with permission to manage rate limit policies.
• A data plane on 0.8.0 or later serving the project, for the request ceilings and the per-minute token ceiling. Earlier data planes enforce the per-hour token ceilings only, and a limit that sets one of the newer counters is refused while one of them serves the project.
• At least one provider and model already provisioned. See Provision models and providers.
• Coordination with the developer teams that own the affected keys. Limits are enforced inline, so a misconfigured limit produces visible production impact.

Step 1: choose the counter

A rate limit counts requests or tokens over a minute or an hour. Each counter is optional, and any mix of them applies at once: a request is refused when any one is used up.

Counter	What it caps	Where it is set
Requests per minute	Call frequency: retry loops and bursts of small calls	Admin Console or API
Requests per hour	Sustained call volume	Admin Console or API
Tokens per minute	Token bursts: several large requests at once	Admin Console or API
Tokens per hour	Sustained token volume, input and output together	Admin Console or API
Input tokens per hour	Prompt volume: long contexts and large documents	API only
Output tokens per hour	Completion volume: verbose or unbounded generation	API only

A request ceiling catches a retry storm fastest, because a flood of small calls accumulates tokens slowly. Request counters are charged when the request arrives and do not overshoot. Token counters are charged once the response is finished, so requests already running can take the total past the ceiling for a moment.

Total tokens is the general-purpose token ceiling. The separate input and output counters fit workloads whose load is lopsided: a summarization pipeline that sends large prompts and returns short answers is bounded by its input volume, and a generation workload by its output. Where the runaway is a misbehaving or compromised key rather than a legitimately heavy workload, the faster tool is containment.

Step 2: choose the subject

The subject determines which traffic the ceiling is measured against.

Subject	What the limit governs	Typical use
An API key	All traffic presented with one key	Bounding a specific application or integration to its expected envelope
A project	Traffic under one project	Bounding a piece of work whose keys share a project

A project limit is each key's limit, not the project's total. A project set to 100,000 tokens an hour gives every key in that project 100,000 tokens an hour of its own. Where several limits cover one key, the smallest number per counter is the one in force. On a data plane on 0.8.0 or later, a key's counter covers every model the key calls.

Tag and system-wide scopes are available through the management API only, with the per-hour token counters only. Limits already set on a tag continue to apply to the keys they match.

Separately from operator-set policies, each API key can also carry Console-side hourly token limits (Total, Input, and Output tokens per hour, counted in fixed hourly windows aligned to the UTC clock), configured by the developer who owns the key. That key-owner control is documented in Route requests across providers under What the API Keys page shows. It does not replace an administrator rate-limit policy, and a developer cannot see or change the administrator policies that cover the key.

Step 3: set the limit

Step 3.1: in the Admin Console

1. In the Admin Console, open Projects and select the project.
2. Open the Rate limits tab.
3. Select Set project limit for the project, or Add key limit and pick the API key.
4. Enter a name and the counters to enforce. The action is Block, the only one offered.
5. Save. A refusal with FailedPrecondition names a data plane serving the project that is older than 0.8.0; upgrading it, or setting only Tokens per hour, clears it.

The tab lists the project's limit and every key limit in the project, with Edit and Remove. Emptying a counter and saving removes that one counter.

Step 3.2: through the management API

The operations are documented in the Policy API reference, starting with CreateRateLimitPolicy. Three behaviors of the API are worth knowing:

• The API accepts a period value of daily, weekly, or monthly, and it has no effect: each counter's window comes from its name.
• UpdateRateLimitPolicy removes a single counter through clear_dimensions.
• A limit left unset for a given subject is not enforced for that subject. A key with no ceiling of its own is still covered by any project ceiling (or any tag or system-wide ceiling set through the API) that applies to it.

A changed limit reaches the data plane in about 5 seconds, and in at most about 10; Configuration propagation covers configuration delivery in general.

Step 4: size and adjust limits

Sizing is the part of this work that takes the most judgment. Two failure modes are worth avoiding.

• Limits set too tight. Normal traffic hits the ceiling, healthy clients start receiving 429 responses, and the workload looks, from the consumer's side, as though Tetrate Agent Router is failing. The cost of this failure mode is immediate and visible.
• Limits set too loose. A runaway consumer is never actually constrained, and the limit becomes a number that never fires. The cost surfaces later, as overloaded upstreams or a degraded experience for other consumers sharing the same capacity.

The dependable approach is to size each limit from observed traffic. The expected peak is read from the usage surface (see Monitor traffic and usage), the highest legitimate volume over a representative window is taken as the baseline, and the limit is set comfortably above that baseline but well below any level that would constitute overload. Setting a limit at roughly two to three times the observed peak is a common starting point, adjusted for how bursty the workload is.

Each data plane counts on its own, so a project served from two data planes admits up to twice its ceiling. A target stated per day is divided by 24 for an hourly counter, or by 1,440 for a per-minute one.

For a brand-new workload with no traffic history, a deliberately tight limit is the safer starting point. Real traffic surfaces the true shape quickly, and the limit is relaxed once the baseline is known. Relaxing a limit that proved too tight is a low-risk adjustment; discovering that a generous limit allowed an overload is harder to recover from.

Some workloads are quiet most of the time and very loud occasionally: end-of-month batch runs and scheduled report generation. A limit sized for the quiet baseline fires on the burst. The cleanest answer is usually to isolate the bursty work on its own key with its own, more generous limit, which keeps the steady-state key tight and the usage attribution clean. The operational complication of issuing a second key is typically smaller than the complication of explaining why a single key's traffic shape is irregular.

Step 5: monitor whether limits are being hit

A limit that never fires and a limit that fires constantly are both worth knowing about: the first may be set too loosely to matter, and the second is likely throttling healthy traffic. Both are visible in Agent Router's usage and traffic surfaces.

The signal to watch for is the rate of 429 responses, broken down by the subject the limit is set on. A workflow that pairs well with this guide:

1. Open the usage and traffic view. See Monitor traffic and usage.
2. Apply a time range that matches the cadence of the workload under review.
3. Break the traffic down by the subject the limit is set on: by key or by project.
4. Look at the proportion of requests returning 429. A small, occasional fraction during peaks is normal; a sustained high fraction indicates the limit is too tight for the legitimate workload.

When a limit is firing more than expected, the choices are to raise it if the traffic is legitimate (see Step 4), or to address the consumer if the traffic is not, by working with the owning team or, in the case of a suspected-compromised key, following Contain a leaked key before it drains the budget.

What a caller experiences when a limit is hit

When a request would breach an active limit, the gateway rejects it with an HTTP 429 Too Many Requests response rather than forwarding it upstream. The rejection applies only to the request that crossed the threshold; once the window advances, traffic flows again. A 429 is therefore a transient, recoverable signal, not a hard failure, though to a client that does not handle it, it presents the same way as an error.

The 429 names the limit that fired. On a data plane on 0.8.0 or later, the body carries the code rate_limit_exceeded, the display name of the policy that set the limit, the window (1m or 1h), and retry_after_seconds, with a matching retry-after header. The policy name is null when two policies set the same number for the same window under different names. A blocked budget also returns 429, with the code budget_exceeded, so the body separates a rate limit from an exhausted spend cap. The full body is shown in Set rate limits on projects and keys.

Applications are expected to back off and retry rather than fail or retry immediately:

• Retry the request after a short delay rather than treating the 429 as fatal.
• Use exponential back-off with jitter (increasing the delay on each successive 429, with a small random offset) so that many clients hitting the limit at once do not retry in lockstep and re-create the burst.
• Respect the retry-after header, where the client library surfaces it.
• Cap the number of retries so that a sustained limit does not turn into an unbounded retry loop, which is itself a source of the pressure the limit exists to contain.

---

Where to go next

Set rate limits on projects and keys

Eight scenarios, from one runaway key to two limits on the same key.

Set a budget and track spend against it

The spend cap that complements the flow control established here.

Contain a leaked key before it drains the budget

The incident response when the runaway is not legitimate traffic.

Monitor traffic and usage

Watch the effect of the limits on live traffic.