429 Too Many Requests From an LLM API: Rate Limits, Concurrency, and Retries That Work
429 means you exceeded a quota, but there are three different quotas and they fail differently. Identify which one you hit before changing anything, then fix it in the order that works.
1. Read the error body before changing anything
A 429 body usually names the limit that was hit and often includes a retry-after hint. Skipping this step is why people fix the wrong thing — halving concurrency does nothing if you were hitting TPM with a single enormous prompt. Log the full body on every 429, not just the status code, together with the model, the prompt size and the `max_tokens` you sent. The four cases look different in the text: a rate-limit 429 talks about requests or tokens per minute, a concurrency 429 about too many simultaneous requests, and a quota 429 about billing, credit or plan. Many providers also return rate-limit headers on ordinary successful responses, showing how much allowance remains; if yours does, log them too, because they show you approaching the ceiling before you hit it. Check your provider's rate-limit guide for the exact header names and what each one counts.
2. A 429 that is not about speed: the empty balance
Some providers return 429 for exhausted credit or quota, not for request rate. OpenAI's API, for example, uses 429 with an insufficient-quota code when the account has no balance. Retrying such an error is pure waste: it will fail identically forever, and an automatic retry loop can hide the real problem for hours. Build the distinction into your client: retry rate-limit 429s with backoff, and on a quota 429 stop, alert a human, and surface the message. If you are behind a gateway, the same applies to the gateway's own balance — a request rejected for an empty balance is a billing event, not a traffic event.
3. max_tokens counts before you use it
On several providers, the tokens-per-minute figure is estimated at request time from your prompt plus the `max_tokens` you allowed, not from what the model actually generates. A request that sets a huge `max_tokens` "just in case" reserves that capacity and can trigger TPM throttling even when the real outputs are short. This is the first thing to check when you see 429 although the dashboard says you are under the limit: the dashboard counts what was used, while the limiter counted what you reserved. Set `max_tokens` to what the task needs — measure the longest real output over a day and add a margin — and watch whether 429s drop. It is one of the few fixes that costs nothing and needs no new code.
No overseas card needed: cocodot's OpenAI-compatible API — switch your base_url to call the models on sale.
Top up via Alipay / WeChat Pay (RMB), USDT / USDC · billed per token · switch your base_url to connect
4. Exponential backoff, with jitter and a ceiling
On a rate-limit 429, wait and retry with the delay doubling each attempt, capped at a ceiling, with a hard limit on attempts. The jitter matters more than people expect: without random variation, every client that got throttled at the same instant retries at the same instant, recreating the spike that caused the problem. Add a random fraction to each delay. Respect retry-after when the provider sends it — that is the provider telling you exactly how long to wait. Two more rules keep backoff from backfiring. First, if you use an official SDK, check whether it already retries internally; wrapping your own retry loop around it multiplies attempts. Second, assume rejected requests may still count against your allowance unless your provider's documentation says otherwise, which is why immediate retries rarely work.
5. Rate limit on your side, not just theirs
Backoff is reactive; a client-side limiter is preventive, and it is what keeps batch jobs from melting. Wrap calls in a semaphore sized to your real concurrency allowance, and if you are hitting RPM, add a token-bucket limiter as well. For TPM, a limiter that counts estimated tokens per request — prompt length plus `max_tokens` — works better than one that counts requests. The rewrite is small: the loop that fires everything becomes a loop that acquires a slot first. It converts a job that fails halfway into one that simply runs a bit slower, and it makes your throughput predictable enough to plan around.
6. The concurrency mistake almost everyone makes once
Concurrency limits are global, and limiters usually are not. Ten workers each holding a semaphore of ten gives you a hundred in flight while your config says ten. If you run multiple processes, containers or serverless instances, the limiter has to be shared — a central queue, a shared counter, or a limit divided across workers. Streaming makes this worse: a concurrency slot stays occupied until the response finishes, and long agent turns can pin slots for a minute or more while your request rate looks tiny. If 429s appear with low RPM and modest tokens, count in-flight requests over time rather than requests per minute. Also check whether another service, a teammate's script or a CI job shares the same key — limits are per key or per account, not per program.
7. Queue the bursts, and know your gateway's limits
If traffic arrives in bursts — a nightly job, a user action that fans out to dozens of calls — a queue between the trigger and the API turns a spike into a steady stream. This is more work than the first fixes, so do it when you have burst-shaped traffic rather than as a default; for steady traffic a shared limiter is enough. If you call through a gateway rather than the provider directly, there are two limiters in the path: the provider's upstream and the gateway's own. A gateway that pools capacity can absorb bursts an individual account could not, or can become the bottleneck itself, so ask any provider what its concurrency allowance is before you build around an assumption. cocodot does not impose an additional per-key concurrency cap beyond what upstream enforces, so what you see is the provider's own limit.
8. Checklist when you are stuck
1. Log the full 429 body and identify which of the four cases it is. 2. On a quota 429, stop retrying and fix the balance. 3. Compare `max_tokens` with your real outputs, and lower it. 4. Confirm effective concurrency by counting in-flight requests, not by reading config — and look for other programs on the same key. 5. Verify backoff has jitter, a ceiling and a cap on attempts, and is not stacked on an SDK's own retries. 6. For TPM, measure actual tokens per request; a context that grew over weeks is a common silent cause. 7. Put a queue in front of bursty jobs. 8. Only then ask for a higher limit — after the steps above, the limit you need is usually smaller than the one you were about to request.
Four kinds of 429, and how to tell them apart
| Limit | What it counts | Typical trigger | First fix |
|---|---|---|---|
| RPM | Requests per minute | Many small calls | Client-side rate limiter |
| TPM | Tokens per minute, often estimated at request time | Long prompts, big context, oversized max_tokens | Trim context, set max_tokens to what you need |
| Concurrency | Requests in flight | Parallel workers, long streaming responses | One semaphore shared across all workers |
| Balance / quota | Credit or plan allowance | Account out of credit | Top up or raise the cap — never retry |