cocodot
← Back to guides
Local card declined for overseas AI? cocodot: one card + one key
ExplainersUpdated 2026-07

429 Too Many Requests From an LLM API: Rate Limits, Concurrency, and Retries That Work

429 means you exceeded a quota, but there are three different quotas and they fail differently. Identify which one you hit before changing anything, then fix it in the order that works.

TL;DR: 429 Too Many Requests means you went over an allowance — but there are several, and the fix differs. RPM (requests per minute), TPM (tokens per minute, the one long prompts blow through), concurrency (how many requests are in flight at once), and on some providers an exhausted balance or quota, which also comes back as 429. Read the error body first: most providers name which limit you hit. Then, in this order: (1) exponential backoff with jitter — on a rate-limit 429, wait and retry; hammering harder makes the queue worse, and without jitter your retries synchronise into a new spike. (2) A client-side limiter shared by every worker — it turns fire all 1,000 at once into a controlled few at a time. (3) A queue for burst-shaped traffic. Two silent causes explain most "429 when I'm under the limit" reports: `max_tokens` set far above what you need, and ten workers each running their own limiter of ten.

1. Read the error body before changing anything

A 429 body usually names the limit that was hit and often includes a retry-after hint. Skipping this step is why people fix the wrong thing — halving concurrency does nothing if you were hitting TPM with a single enormous prompt. Log the full body on every 429, not just the status code, together with the model, the prompt size and the `max_tokens` you sent. The four cases look different in the text: a rate-limit 429 talks about requests or tokens per minute, a concurrency 429 about too many simultaneous requests, and a quota 429 about billing, credit or plan. Many providers also return rate-limit headers on ordinary successful responses, showing how much allowance remains; if yours does, log them too, because they show you approaching the ceiling before you hit it. Check your provider's rate-limit guide for the exact header names and what each one counts.

2. A 429 that is not about speed: the empty balance

Some providers return 429 for exhausted credit or quota, not for request rate. OpenAI's API, for example, uses 429 with an insufficient-quota code when the account has no balance. Retrying such an error is pure waste: it will fail identically forever, and an automatic retry loop can hide the real problem for hours. Build the distinction into your client: retry rate-limit 429s with backoff, and on a quota 429 stop, alert a human, and surface the message. If you are behind a gateway, the same applies to the gateway's own balance — a request rejected for an empty balance is a billing event, not a traffic event.

3. max_tokens counts before you use it

On several providers, the tokens-per-minute figure is estimated at request time from your prompt plus the `max_tokens` you allowed, not from what the model actually generates. A request that sets a huge `max_tokens` "just in case" reserves that capacity and can trigger TPM throttling even when the real outputs are short. This is the first thing to check when you see 429 although the dashboard says you are under the limit: the dashboard counts what was used, while the limiter counted what you reserved. Set `max_tokens` to what the task needs — measure the longest real output over a day and add a margin — and watch whether 429s drop. It is one of the few fixes that costs nothing and needs no new code.

No overseas card needed: cocodot's OpenAI-compatible API — switch your base_url to call the models on sale.

Top up via Alipay / WeChat Pay (RMB), USDT / USDC · billed per token · switch your base_url to connect

4. Exponential backoff, with jitter and a ceiling

On a rate-limit 429, wait and retry with the delay doubling each attempt, capped at a ceiling, with a hard limit on attempts. The jitter matters more than people expect: without random variation, every client that got throttled at the same instant retries at the same instant, recreating the spike that caused the problem. Add a random fraction to each delay. Respect retry-after when the provider sends it — that is the provider telling you exactly how long to wait. Two more rules keep backoff from backfiring. First, if you use an official SDK, check whether it already retries internally; wrapping your own retry loop around it multiplies attempts. Second, assume rejected requests may still count against your allowance unless your provider's documentation says otherwise, which is why immediate retries rarely work.

5. Rate limit on your side, not just theirs

Backoff is reactive; a client-side limiter is preventive, and it is what keeps batch jobs from melting. Wrap calls in a semaphore sized to your real concurrency allowance, and if you are hitting RPM, add a token-bucket limiter as well. For TPM, a limiter that counts estimated tokens per request — prompt length plus `max_tokens` — works better than one that counts requests. The rewrite is small: the loop that fires everything becomes a loop that acquires a slot first. It converts a job that fails halfway into one that simply runs a bit slower, and it makes your throughput predictable enough to plan around.

6. The concurrency mistake almost everyone makes once

Concurrency limits are global, and limiters usually are not. Ten workers each holding a semaphore of ten gives you a hundred in flight while your config says ten. If you run multiple processes, containers or serverless instances, the limiter has to be shared — a central queue, a shared counter, or a limit divided across workers. Streaming makes this worse: a concurrency slot stays occupied until the response finishes, and long agent turns can pin slots for a minute or more while your request rate looks tiny. If 429s appear with low RPM and modest tokens, count in-flight requests over time rather than requests per minute. Also check whether another service, a teammate's script or a CI job shares the same key — limits are per key or per account, not per program.

7. Queue the bursts, and know your gateway's limits

If traffic arrives in bursts — a nightly job, a user action that fans out to dozens of calls — a queue between the trigger and the API turns a spike into a steady stream. This is more work than the first fixes, so do it when you have burst-shaped traffic rather than as a default; for steady traffic a shared limiter is enough. If you call through a gateway rather than the provider directly, there are two limiters in the path: the provider's upstream and the gateway's own. A gateway that pools capacity can absorb bursts an individual account could not, or can become the bottleneck itself, so ask any provider what its concurrency allowance is before you build around an assumption. cocodot does not impose an additional per-key concurrency cap beyond what upstream enforces, so what you see is the provider's own limit.

8. Checklist when you are stuck

1. Log the full 429 body and identify which of the four cases it is. 2. On a quota 429, stop retrying and fix the balance. 3. Compare `max_tokens` with your real outputs, and lower it. 4. Confirm effective concurrency by counting in-flight requests, not by reading config — and look for other programs on the same key. 5. Verify backoff has jitter, a ceiling and a cap on attempts, and is not stacked on an SDK's own retries. 6. For TPM, measure actual tokens per request; a context that grew over weeks is a common silent cause. 7. Put a queue in front of bursty jobs. 8. Only then ask for a higher limit — after the steps above, the limit you need is usually smaller than the one you were about to request.

Four kinds of 429, and how to tell them apart

LimitWhat it countsTypical triggerFirst fix
RPMRequests per minuteMany small callsClient-side rate limiter
TPMTokens per minute, often estimated at request timeLong prompts, big context, oversized max_tokensTrim context, set max_tokens to what you need
ConcurrencyRequests in flightParallel workers, long streaming responsesOne semaphore shared across all workers
Balance / quotaCredit or plan allowanceAccount out of creditTop up or raise the cap — never retry

FAQ

Should I just retry immediately on 429?

No. Immediate retries add load to the exact resource that is already saturated and usually extend the problem. Back off, add jitter, cap the attempts — and do not retry a quota 429 at all.

Why do I get 429 when I'm under the rate limit?

Usually one of three reasons: `max_tokens` is set so high that the estimated token load hits the per-minute cap even though real usage is low; several workers or programs share the key, so effective concurrency is higher than you think; or the 429 is about balance or quota, not speed. The error body tells you which.

Does a bigger balance raise my rate limits?

Sometimes with direct provider accounts, where tiers can scale with usage history — check your provider's rate-limit page for how tiers work today. It is not a general rule, and it never fixes a client-side concurrency bug.

How do I avoid 429 Too Many Requests altogether?

You cannot promise zero, but you can make them rare: a shared client-side limiter sized to your real allowance, `max_tokens` set to what the task needs, backoff with jitter for the occasional miss, and a queue in front of bursty jobs.

What does insufficient_quota mean on a 429?

On OpenAI's API it signals a billing problem — the account has no balance or quota left — not that you are sending too fast; your provider's error-code page lists the exact codes. Slowing down will not help; top up or raise the cap, then resume. Treat it as an alert in your code, not as a retryable error.

What is the difference between 429 and 503 or 529?

429 means you went over your allowance; retrying later on your schedule is correct. 503 and similar overload codes mean the provider is short of capacity regardless of your usage — backoff still helps, but reducing your own rate will not.

About cocodot

cocodot is a payment and AI access service for developers and cross-border teams in mainland China. It provides US-BIN virtual cards issued by a licensed institution — used to pay for overseas subscriptions and ad accounts — and an OpenAI-compatible AI API gateway for calling Claude, GPT and Gemini from within mainland China. Both share one wallet, funded by Alipay / WeChat Pay (RMB), USDT / USDC and accounted in USD. Card: $9.9 to open, 3% to load, $1 per active card per month; spending: $0.60 settlement fee on purchases under $20; a corresponding fee applies when the issuer charges one.

Service scope, pricing and limits →
OpenAI-compatible API, pay in RMBGet API access