Today's concept: retries, timeouts, and fallbacks for LLM APIs
The official Anthropic and OpenAI Python SDKs quietly retry a failed request twice before your code hears about it. So "it just worked" in your test script can mean "it failed twice and you never knew."
Think of it like ordering food on a busy Friday night.
1) Retry the flaky The kitchen is slammed: "try again in 30 seconds." That is a rate limit (HTTP 429): the provider is fine, you are just asking too fast. Both SDKs retry connection errors, 408, 409, 429 and 5xx by default, with exponential backoff plus jitter: wait longer each time, offset randomly so everyone who failed at once does not return at once. If the response carries a retry-after header, they obey that instead of guessing. And it all keys off the status code, not the message.
2) Time out the silent The order vanishes and nothing comes back. A timeout is you deciding up front how long you will wait. Both SDKs default to 10 minutes, an eternity if a human is watching, and reasoning models make that wait less predictable. The fix is rarely a bigger number: it is streaming, so bytes keep flowing and nothing on the path mistakes the call for a dead connection. Anthropic's SDK even refuses a non-streaming request it expects to run past 10 minutes.
3) Fall back on the dead The restaurant is closed, so retrying is just noise. Real overload (Anthropic returns 529) needs a different destination: another provider, a smaller model, or a cached answer. That is the circuit breaker pattern from distributed systems, and routers like LiteLLM ship it as config (cooldowns, fallbacks, retry policies). The part nobody budgets for: your prompt cache does not travel with you, so the fallback path is slower and pricier just when you are already degraded.
The trap: a retry is only safe if the thing retried is safe to repeat. Re-asking for a completion is harmless. Re-running an agent's tool call that sends an email or charges a card is not. SDK retries protect the HTTP request, not your side effects, so idempotency has to live in your tool layer.
It compounds: 20 chained calls at 99% each finish clean only about 82% of the time. So: retry the flaky, time out the silent, fall back on the dead.
Quick check before you scroll: Your API call returns a 429 error whose message says you've exhausted your billing credit (for example, OpenAI's "credit_balance_exhausted") instead of the generic "rate limit reached." Should you retry it?
Full breakdown + the answer: frankduah.me/learnings/2026-08-09-retries-timeouts-and-fallbacks-for-llm-apis
New here? I post a bite-size AI / ML concept like this every day, follow me for the daily drop, and it compounds fast. Why I do it: https://lnkd.in/gK8knHDH
#AI #LLM #AIAgents #MachineLearning
The answer
No, retrying won't help. A generic rate-limit 429 means slow down and try again later; a billing/quota 429 means you're out of credit, and no amount of backoff fixes that until someone adds funds.