What is the safest retry strategy on a rate-limit error from an LLM provider?

Exponential backoff with jitter and a hard ceiling on attempts.

Retry immediately, in a tight loop.

Exponential backoff with jitter and a hard ceiling on attempts.

Switch to a smaller model with no notification.

Rate limits, retries, and graceful degradation — Semantic Web Academy

Overview

Provider quotas are a fact of life. Plan for them.

Why it matters

Every LLM call can be rate-limited, timed out, or partially answered. The supervisor + checkpointer pattern makes recovery cheap.

Where this sits in the stack

Rate limits, retries, and graceful degradation is one of the load-bearing decisions in a KG/RAG/agent system: choices made here propagate to retrieval quality, agent reliability, cost per query, and the on-call burden of whoever ships it. Teams that name this trade-off explicitly ship faster than teams that leave it implicit.

Why this is load-bearing

Rate limits, retries, and graceful degradation is the building-code of this layer. You can ignore building codes on a shed, but the moment you put two storeys on top of the same foundation they decide whether the structure stands or falls. In a KG/RAG/agent stack, the equivalent of 'two storeys' is the second feature you ship on top of this primitive — GraphRAG on top of chunking, supervisor agents on top of state machines, regression CI on top of metrics. The cost of cutting the wrong corner now is paid by every later layer, with interest.

Every LLM call can be rate-limited, timed out, or partially answered. The supervisor + checkpointer pattern makes recovery cheap.

Reflect — apply it

Anchor rate limits, retries, and graceful degradation to something concrete in your own work.

›Where have you seen rate limits, retries, and graceful degradation done well? Name one team or product and what they got right.
›Where have you seen it done badly? What was the first symptom that surfaced (latency, hallucination, cost, outage)?
›What is the *cheapest* version of this you could ship in your next sprint, and what single metric would tell you it's working?

Reading in progress · 0 of 3 activities done