Stop runaway workloads

Enterprise Tier

Rate limits keep any one consumer from taking more than its share, so a runaway hits a wall at two to three times normal traffic instead of at three weeks of spend. A single misbehaving consumer, an agent framework in a retry loop or a batch job that saturates the provider's concurrency allowance, can otherwise degrade Agent Router for everyone.

---

Tetrate Agent Router exposes three distinct rate limits, configured in the Admin Console. Tokens per minute (TPM) caps how much model work a consumer can drive; requests per minute (RPM) caps how often it can call regardless of payload size; and a maximum-parallel-requests limit caps how many calls it can have in flight at once. Each addresses a different failure mode, and they are usually combined.

A rate limit is flow control, not a spend cap. It protects shared capacity by smoothing the rate of traffic; it does not stop a consumer from spending a budget over the course of a month. The two are complementary, and choosing between them is covered in Choose the right cost control for each workload; the spend caps themselves are set in Set a budget and track spend against it.

Persona: Platform operator working in the Admin Console.

Estimated time: 15 to 25 minutes for an initial pass; ongoing as workloads evolve.

When this guide applies

Rate limits are the right concern in any of these situations:

Situation	Limit type that fits
A retry loop or runaway agent is sending far more requests than a healthy client would	Requests per minute (RPM)
A consumer drives large token volumes (long contexts and verbose completions) that strain upstream capacity	Tokens per minute (TPM)
A batch job opens many connections at once and saturates the provider's concurrency allowance	Maximum parallel requests
One team's traffic should never crowd out another team sharing the same deployment	A per-group limit across all of that group's keys
A new or untrusted integration should be bounded before its real traffic shape is known	A deliberately tight limit at the key scope
Spend over a month should be capped regardless of request rate	A budget, not a rate limit. See Set a budget and track spend against it

Outcomes

By the end of this guide:

• The three limit types (TPM, RPM, and maximum parallel requests) are understood, along with the failure mode each one addresses.
• At least one rate limit is applied at the scope appropriate to the workload it protects.
• The rule for how limits at different scopes compose, the most restrictive applicable limit wins, is understood.
• The caller-side experience of a hit limit, an HTTP 429 response, is understood, along with the back-off behavior applications are expected to implement.
• A way to confirm whether limits are being hit is in place.

Prerequisites

• Administrator access to the Admin Console: typically the super_admin role, or a role with permission to manage limits.
• At least one provider and model already provisioned. See Provision models and providers.
• For per-group limits, the groups that traffic is attributed to. Group membership is sourced from the corporate identity provider, as described in Map Entra ID groups to business functions.
• Coordination with the developer teams that own the affected keys. Limits are enforced inline, so a misconfigured limit produces visible production impact.

Step 1: choose the limit type

The three limits answer different questions about a consumer's traffic. Pick the one, or the combination, that matches the capacity being protected.

Limit	What it caps	The failure mode it addresses
Tokens per minute (TPM)	Total tokens processed per minute, input plus output	Heavy model work: long contexts and verbose completions that consume upstream throughput out of proportion to the request count
Requests per minute (RPM)	Number of requests per minute, regardless of size	High call frequency: retry storms, tight polling loops, and chatty agents that make many small calls
Maximum parallel requests	Number of requests in flight at the same moment	Concurrency exhaustion: batch jobs and fan-out patterns that open many simultaneous connections and saturate the provider's concurrency allowance

TPM and RPM are rate measures; they constrain traffic averaged over a window of time. Maximum parallel requests is a concurrency measure; it constrains an instantaneous count and is unconcerned with rate. A consumer can stay well under its RPM yet still exhaust the parallel-request limit by issuing a single large burst, and the reverse is equally possible. Because they catch different shapes of traffic, the limits are usually applied together.

A reasonable default for a new workload is a moderate RPM to absorb retry storms, a TPM sized to the expected token volume, and a parallel-request limit that reflects how much fan-out the workload legitimately needs. The values are tightened or relaxed once real traffic reveals the workload's shape, the sizing approach is covered in Step 4.

Step 2: choose the scope

The same three limit types can be applied at several scopes. The scope determines which traffic the limit is measured against.

Scope	What the limit governs	Typical use
Per model	All traffic to one model, across every consumer	Protecting a single upstream model or deployment from aggregate overload, independent of who is calling it
Per API key	All traffic presented with one key	Bounding a specific application or integration to its expected envelope
Per user	All traffic across every key a user owns	Holding an individual developer's total footprint in check, regardless of how many keys they hold
Per group	All traffic across every member of a group	Reserving a fair share of capacity for a business function, so one team does not crowd out another

Per-key limits are the most precise and the most common, because a key usually maps to a single workload. Per-user and per-group limits sit above the key scope and govern aggregate footprint: a user's group is resolved from the identity provider at request time, so traffic is attributed to the right group automatically once the mapping in Map Entra ID groups to business functions is in place. Per-model limits are orthogonal to the others; they cap total load on an upstream model regardless of which consumer is driving it, and are the right tool when a particular model or deployment has a known capacity ceiling.

Limits are set from the Admin Console surface that manages the relevant entity: the model, the key, the user, or the group. The fields presented are the three limit types from Step 1; a limit left unset at a given scope is not enforced at that scope.

Alongside the Admin Console scopes, each API key also carries Console-side hourly token limits (Total, Input, and Output tokens per rolling hour), configured by the developer who owns the key. The operator-side decision about which keys should carry which limits is what makes that mechanism useful: the operator establishes the policy ("research keys cap at 100 K tokens per hour") and the developer applies it. The developer-side flow is documented in Route requests across providers under What the API Keys page shows.

Step 3: understand how scopes compose

A single request can fall under limits at more than one scope at once. A request made with a particular key, by a particular user, who belongs to a particular group, against a particular model is subject to the limits set at all four scopes simultaneously.

The composition rule is straightforward: every applicable limit is evaluated independently, and the request is rejected if it would breach any one of them. The most restrictive applicable limit is therefore the one that takes effect. A generous per-group allowance does not loosen a tight per-key limit, and a generous per-key limit does not override a tight per-model ceiling. The limits do not add together, and a higher-scope limit does not raise a lower one.

A worked example makes the rule concrete:

• A group is allowed 10,000 RPM across all its members.
• A key belonging to a member of that group is allowed 500 RPM.
• The model that key targets is allowed 2,000 RPM across all consumers.

Traffic on that key is held to 500 RPM, because the per-key limit is the most restrictive of the three that apply. If a second key in the same group also runs near its own limit, the two together are still held under the group's 10,000 RPM; and all consumers of the model together are held under the model's 2,000 RPM. Each ceiling is enforced at its own scope, and a request must satisfy all of them to pass.

The practical consequence is that the tightest limit governs. When a limit appears not to be taking effect, the usual cause is a tighter limit at another scope firing first. The way to confirm which limit is firing is covered in Step 5.

Step 4: size and adjust limits

Sizing is the part of this work that takes the most judgment. Two failure modes are worth avoiding.

• Limits set too tight. Normal traffic hits the ceiling, healthy clients start receiving 429 responses, and the workload looks, from the consumer's side, as though Agent Router is failing. The cost of this failure mode is immediate and visible.
• Limits set too loose. A runaway consumer is never actually constrained, and the limit becomes a number that never fires. The cost surfaces later, as overloaded upstreams or a degraded experience for other consumers sharing the same capacity.

The dependable approach is to size each limit from observed traffic. The expected peak is read from the usage surface, see Monitor traffic and usage, the highest legitimate per-minute value over a representative window is taken as the baseline, and the limit is set comfortably above that baseline but well below any level that would constitute overload. Setting a limit at roughly two to three times the observed peak is a common starting point, adjusted for how bursty the workload is.

For a brand-new workload with no traffic history, a deliberately tight limit is the safer starting point. Real traffic surfaces the true shape quickly, and the limit is relaxed once the baseline is known. Relaxing a limit that proved too tight is a low-risk adjustment; discovering that a generous limit allowed an overload is harder to recover from.

Limits are adjusted from the same surface used to set them. A change takes effect on subsequent requests, with no restart and no downtime window. Because the change is felt by live traffic, an adjustment to a production key is best coordinated with the team that owns it.

Some workloads are quiet most of the time and very loud occasionally: end-of-month batch runs and scheduled report generation. A limit sized for the quiet baseline fires on the burst. The cleanest answer is usually to isolate the bursty work on its own key with its own, more generous limit, which keeps the steady-state key tight and the usage attribution clean. The operational complication of issuing a second key is typically smaller than the complication of explaining why a single key's traffic shape is irregular.

Step 5: monitor whether limits are being hit

A limit that never fires and a limit that fires constantly are both worth knowing about: the first may be set too loosely to matter, and the second is likely throttling healthy traffic. Both are visible in Agent Router's usage and traffic surfaces.

The signal to watch for is the rate of 429 responses, broken down by the scope the limit is set on. A workflow that pairs well with this guide:

1. Open the usage and traffic view. See Monitor traffic and usage.
2. Apply a time range that matches the cadence of the workload under review.
3. Break the traffic down by the scope the limit is set on: by key, by user, by group, or by model.
4. Look at the proportion of requests returning 429. A small, occasional fraction during peaks is normal; a sustained high fraction indicates the limit is too tight for the legitimate workload.
5. Correlate the 429 rate with the limit type. A spike concentrated in one limit type (a parallel-request limit firing while RPM stays clear, for instance) points to the specific control that needs adjustment.

When a limit is firing more than expected, the choices are to raise it if the traffic is legitimate, see Step 4, or to address the consumer if the traffic is not, by working with the owning team or, in the case of a suspected-compromised key, following Contain a leaked key before it drains the budget.

What a caller experiences when a limit is hit

When a request would breach an active limit, the gateway rejects it with an HTTP 429 Too Many Requests response rather than forwarding it upstream. The rejection is immediate and applies only to the request that crossed the threshold; once the window advances or in-flight requests drain, traffic flows again. A 429 is therefore a transient, recoverable signal, not a hard failure, though to a client that does not handle it, it presents the same way as an error.

Applications are expected to back off and retry rather than fail or retry immediately:

• Retry the request after a short delay rather than treating the 429 as fatal.
• Use exponential back-off with jitter (increasing the delay on each successive 429, with a small random offset) so that many clients hitting the limit at once do not retry in lockstep and re-create the burst.
• Respect any retry-after guidance the response carries, where the client library surfaces it.
• Cap the number of retries so that a sustained limit does not turn into an unbounded retry loop, which is itself a source of the RPM pressure the limit exists to contain.

Scenarios

The situations below are common in day-to-day operation, and each one resolves to a particular limit type at a particular scope. Every scenario is written in four parts: the scenario, the setup, what happens, and what to watch for.

A shared model is overloaded and no single consumer is at fault

The scenario. One model carries traffic from a dozen keys across several teams. Every key sits inside its own limit, no consumer is misbehaving, and the upstream deployment is still being pushed past what it serves comfortably. There is no key to tighten, because the problem is the sum rather than any one term in it.

The setup

1. Read the peak for that model from the usage surface across all consumers, rather than per key. See Monitor traffic and usage.
2. On the Admin Console surface that manages the model, set a requests-per-minute ceiling, a tokens-per-minute ceiling, or both, sized against the deployment's known capacity ceiling rather than against any one consumer's traffic.
3. Leave the existing per-key limits in place. They are not made redundant by the model ceiling.

What happens. Every request to that model counts against one ceiling, whichever key, user, or group it came from. A request that would cross the ceiling is rejected with 429 as it arrives rather than being forwarded upstream, and the per-key limits go on applying underneath it: a request has to satisfy every limit that covers it, so the tightest applicable limit is still the one that governs a given consumer. See Step 3.

What to watch for. Two things.

A limit left unset at a scope is not enforced at that scope. A key issued next week has no per-key limit until one is set, while a model ceiling already in place covers that key's traffic from its first request. That makes the model scope a dependable backstop against aggregate load, and a poor substitute for sizing each key.

The ceiling does not choose between consumers. It rejects the request that crosses it, not the request from the consumer driving the most traffic. Holding a share of a shared model for one particular team is a per-group limit on that team's keys, described in Step 2.

One developer holds several keys and each one looks reasonable

The scenario. A developer holds one key for a prototype, one for a test harness, and one for a scheduled evaluation run. Each key sits inside a sensible per-key limit. The combined footprint is not sensible, and it grows every time another key is issued.

The setup

1. Break the traffic down by user rather than by key, to establish what the combined footprint actually is. See Step 5.
   1. Size the ceiling to the total one individual should account for, taken from that combined figure rather than from the sum of the per-key limits already in place.
2. Leave the per-key limits as they are. A ceiling over the combined traffic governs the total; the per-key limits go on bounding each workload to its own envelope.

What happens. Where a ceiling covers every key one person owns, the traffic across those keys is measured against it as one figure. Another key competes for the same allowance rather than adding to it, because limits at different scopes do not add together and a higher-scope limit does not raise a lower one. A key already held below that ceiling by its own limit is unaffected by it.

What to watch for. A rate limit bounds a rate, not a total. Tracking what one person's traffic costs over a month, and alerting on it, is a budget, a different control that measures money over a budget period rather than traffic over a short window. See Set a budget and track spend against it.

An hourly token ceiling is wanted on a class of keys

The scenario. Research keys should be free to burst while an experiment is being run interactively, but bounded in what they consume across an hour. None of the three operator-side limits is measured over an hour: two are per minute, and the third is an instantaneous count.

The setup. The hourly control is not an operator field. Each API key also carries Console-side hourly token limits (Total, Input, and Output tokens per rolling hour), configured by the developer who owns the key.

1. Establish the policy: which classes of key carry which hourly ceiling, for example research keys capped at 100 K tokens per hour.
   1. Keep the per-minute limits that cover those keys for the burst shape, sized as in Step 4.
2. Pass the policy to the teams that own the keys. The developer-side flow is documented in Route requests across providers under What the API Keys page shows.

What happens. Two controls apply to the same key and answer different questions about it. The operator-set limits constrain how fast the key may consume; the developer-set hourly limits constrain how much it may consume within a rolling hour. Deciding the ceiling is the operator's part, and setting it is the owning team's.

What to watch for. The hourly ceiling exists only where the owning team has applied it. It is enforced once set, but it is not a control an operator applies alone, which makes it a coordinated one. Where a ceiling has to hold without the key owner's cooperation, the operator-side limits are the ones that do it, expressed per minute rather than per hour.

A limit is set, nothing is being rejected, and the upstream is still strained

The scenario. A key is held to 500 requests per minute and never approaches it. The workload sends around 40 requests a minute, each carrying a 20 K-token document and returning a long summary. The 429 rate is zero, so the diagnostic in Step 5 has nothing to break down, and one consumer is still driving more model work than the upstream absorbs comfortably.

The setup

1. Read the total token volume for that key over a representative window and divide it down to a per-minute figure, alongside its request count rather than the request count alone. See Monitor traffic and usage.
2. Bring a token-based ceiling to bear on that traffic, sized from the observed token volume rather than from the request count.
3. Leave the requests-per-minute limit where it is.

What happens. The measure that matches the failure is the one that now carries a ceiling. Requests per minute goes on covering the failure it was set for, a retry storm or a tight polling loop, which a token limit catches late because a flood of small calls accumulates tokens slowly.

What to watch for. Two things.

A limit that never fires is not evidence that traffic is within bounds. It is evidence about one measure of that traffic. Requests per minute is indifferent to the size of a call, tokens per minute is indifferent to how many calls carried them, and maximum parallel requests is indifferent to both. That is why the three are normally set together rather than chosen between.

Adjusting a limit that is not firing changes nothing. The same reasoning holds when requests are being rejected: the control to adjust is the one the 429 responses are concentrated in, and widening a ceiling that was never the binding one leaves the workload where it was while removing a protection that was doing its job.

---

Where to go next

Set a budget and track spend against it

The spend cap that complements the flow control established here.

Contain a leaked key before it drains the budget

The incident response when the runaway is not legitimate traffic.

Map Entra ID groups to business functions

Confirm group attribution is correct before relying on per-group limits.

Monitor traffic and usage

Watch the effect of the limits on live traffic.