Handling AI Rate Limits
Recover from AI rate limits by respecting Retry-After and retrying with exponential backoff and jitter.
TL;DR
- Detect a
429status and read theRetry-Afterheader. - Retry with exponential
backoffplus random jitter. - Cap retries and limit
concurrencyto avoid a storm.
Detect the Limit
Check the StatusA 429 means you are being rate limited.
if (res.status === 429) await backoff();Read Retry-AfterThe header says how long to wait.
const ra = res.headers.get('Retry-After');Parse SecondsConvert the header to milliseconds.
const ms = Number(ra) * 1000;Check Other Codes503 can also mean overloaded; retry it too.
if (res.status === 503) await backoff();Back Off and Retry
Exponential DelayDouble the wait after each attempt.
const delay = base * 2 ** attempt;Add JitterRandomize so clients do not sync up.
const wait = delay + Math.random() * 250;Cap AttemptsGive up after a few tries.
if (attempt >= 5)
throw new Error('rate limited');Max Delay CapCap the delay so waits do not grow unbounded.
const wait = Math.min(delay, 30000);Limit Concurrency
Max In FlightOnly allow a few requests at once.
if (active >= 3) await queue.next();Queue ExtrasHold the rest until a slot frees.
queue.push(task);Per-User CapEnforce a limit on the server too.
// server: N requests per user / minuteRelease the SlotFree the slot in a finally block so it always runs.
finally { active--; queue.next(); }Tell the User
Show a WaitLet them know a retry is coming.
setStatus('Busy, retrying...');Disable SendBlock new requests while retrying.
<button disabled={retrying}>Send</button>Final ErrorSurface a clear message if it gives up.
setError('Too many requests, try later');Show Attempt CountLet users see progress across retries.
setStatus(`Retrying (${attempt}/5)...`);Tips
- Honor the
Retry-Afterheader when present; it tells you exactly how long to wait, which beats guessing with backoff alone. - Add random
jitterto each backoff delay so many clients do not retry in sync and hammer the API at the same instant.
Warnings
- Do not retry forever; cap attempts and surface a clear error, or a rate-limited user waits behind endless silent
retries. - Retrying a non-idempotent action can double an effect; only auto-retry safe reads, not
writeswith side effects.
In Practice
Retry 429s with exponential backoff, jitter, and an attempt cap.
- Read Retry-After, or fall back to exponential backoff with jitter.
- Return non-429 responses immediately.
- Give up after a fixed cap and throw.
async function askWithRetry(body: unknown, tries = 4) {
for (let i = 0; i < tries; i++) {
const res = await fetch('/api/chat', {
method: 'POST',
body: JSON.stringify(body),
});
if (res.status !== 429) return res;
const ra = res.headers.get('Retry-After');
const wait = Number(ra) * 1000 || 500 * 2 ** i;
await sleep(wait + Math.random() * 250);
}
throw new Error('Rate limited');
}FAQ
You have hit a rate limit: too many requests or tokens in a window. The response often includes a Retry-After header telling you how many seconds to wait. Back off, wait, and retry rather than failing immediately.
You wait longer after each failed attempt, doubling the delay, and add a small random offset. Doubling avoids hammering the API; the random jitter stops many clients retrying in lockstep and causing another spike.
Limit how many requests you send at once, cache and dedupe repeats, and debounce input. A small client-side concurrency limit plus server-side per-user limits keeps you comfortably under the provider's ceiling.