Rate limits, retries, and graceful degradation

Provider quotas are a fact of life. Plan for them.

Why it matters

Every LLM call can be rate-limited, timed out, or partially answered. The supervisor + checkpointer pattern makes recovery cheap.

Every LLM/provider call can be rate-limited, timed out, or partially served — at scale these are normal, not exceptional. Resilient systems plan retries and graceful degradation instead of failing the user session.

def backoff(attempt, base=0.5, cap=8.0):
    jitter = random.uniform(0, base)
    return min(cap, base * (2 ** attempt) + jitter)

Exponential backoff with jitter and a hard cap is the baseline. Exponential spacing stops you hammering a struggling provider; jitter prevents a thundering herd where every client retries in lockstep and re-triggers the limit; the cap bounds worst-case latency so a user isn't stuck for a minute.

Retrying immediately in a tight loop is the classic mistake — it amplifies the outage and often gets you rate-limited harder. So is unlimited retries, which just moves the failure from 'error' to 'hang'.

Degrade gracefully. After the final retry, fall back deliberately: a smaller/cheaper model, a retrieval-only answer with citations, or a queued 'we'll follow up'. Because the supervisor + checkpointer pattern (Level 3) persists state, recovery is cheap — the run resumes from the last checkpoint rather than restarting, and the user keeps their context.

Resilience is merging into busy traffic. You don't floor it the instant you're blocked (tight-loop retry) — you ease in, wait a beat longer each time (backoff), vary your timing so you don't lurch in sync with the car beside you (jitter), and after a few tries you take the side road (degraded fallback).

• Tight-loop retries. Symptom: amplified outage, harder rate-limiting. Fix: exponential backoff.
• No jitter. Symptom: thundering herd. Fix: randomised backoff.
• Unbounded retries. Symptom: hangs instead of errors. Fix: hard attempt cap.
• No degraded mode. Symptom: whole session fails. Fix: fall back to cheaper model / retrieval-only.