An AI demo and an AI product use the same model. The difference is everything around it: what happens when the provider is slow, when a customer double-clicks, when a thousand jobs arrive at once, or when a bill quietly triples. That surrounding layer is made of backend primitives, small building blocks that were around long before language models.
This guide covers the 12 that matter most for people building on LLM APIs. For each one you get what it is in plain words, what the official documentation says, and the mistake it prevents. Three of them (retries, idempotency and streaming) I also ran as small experiments on my own machine, and the numbers are in the text. Everything else is cited to the vendor's or standards body's own page.
The 12 at a glance#
| # | Primitive | The problem it solves | Where the docs say it |
|---|---|---|---|
| 1 | Queues and dead-letter queues | Work arrives faster than the provider allows | Amazon SQS, OpenAI and Anthropic batch docs |
| 2 | Idempotency keys | A retry repeats a side effect | Stripe API reference, IETF draft |
| 3 | Retries with backoff and jitter | Transient failures, retry storms | AWS Builders' Library |
| 4 | Timeouts | One slow call blocks everything | Anthropic errors page |
| 5 | Circuit breakers | A failing provider drags you down with it | Martin Fowler |
| 6 | Rate limits | 429 errors, token budgets | Anthropic and OpenAI rate-limit pages |
| 7 | Streaming (SSE) | Users stare at a blank screen | WHATWG spec, Anthropic streaming docs |
| 8 | Caching | Paying again for the same prompt | Anthropic and OpenAI prompt caching |
| 9 | Structured outputs | Model returns broken JSON | OpenAI and Anthropic structured outputs |
| 10 | Webhooks | Results that arrive later | Stripe webhook docs |
| 11 | Observability | You cannot see what failed or why | Anthropic request IDs, OpenAI usage fields |
| 12 | Per-user quotas | One user burns the budget | OWASP LLM10 |
1. Queues and dead-letter queues#
A queue is a waiting line for work. Your web server drops a job on it ("summarise this document") and answers the user immediately. A separate worker picks jobs up at a pace the model provider can tolerate.
Two official features make queues especially useful for AI. First, both big providers sell a discount for waiting. Anthropic's Message Batches API charges 50 percent of standard prices, takes up to 100,000 requests or 256 MB per batch, and says most batches finish within an hour, with results kept for 29 days. OpenAI's Batch API also gives 50% lower cost, a separate pool of higher rate limits and a 24-hour turnaround, with a 200 MB upload limit. Second, a queue needs somewhere for jobs to go when they keep failing. Amazon SQS calls this a dead-letter queue: a message that has been received more than a set maxReceiveCount times without being deleted is moved aside, where you can inspect it and send it back later.
If you already run Postgres, you can build a basic queue in a table. The SELECT ... FOR UPDATE SKIP LOCKED clause, documented on the PostgreSQL SELECT page, lets several workers take different rows without waiting on each other.
The mistake it prevents: doing a 30-second model call inside a web request, then losing the work when the connection drops.
2. Idempotency keys#
An operation is idempotent if doing it twice has the same effect as doing it once. A GET is naturally idempotent. A POST that charges a card is not, and a retry after a timeout will charge twice.
The fix, described in the Stripe API reference, is for the client to generate a unique key (Stripe suggests a V4 UUID, up to 255 characters) and send it with the request. The server remembers the result for that key, and a retry with the same key returns the saved response instead of repeating the work. An IETF draft for a standard Idempotency-Key header says the key must be unique and must not be reused with a different request payload.
I tested the failure myself. A small local server takes 0.6 seconds to "charge" a customer, and my client gives up after 0.3 seconds and retries, up to three attempts.
| Setup | Customer clicks | Charges created |
|---|---|---|
| Retry without a key | 1 | 3 |
| Retry with a UUID key | 1 | 1 |
For AI work this matters as much as for payments. A retried "send the summary by email" or "create the support ticket" tool call is the same problem. Give every action an AI agent can take a key.
The mistake it prevents: duplicate emails, tickets, charges and database rows after a retry.
3. Retries with exponential backoff and jitter#
Retrying is the right response to a temporary failure, and Anthropic's official SDK does it for you: its documentation says it retries connection errors, rate limits and 5xx errors with exponential backoff, twice by default, honouring the retry-after header when present.
The danger is everyone retrying at once. The AWS Builders' Library explains it well: when failures come from overload, backing off often helps less than expected because the failed calls all back off to the same time and overload the server again. The cure is jitter, a random amount added to the delay to spread retries out.
I simulated it. A toy server handles 100 requests per second and rejects the rest with a 429. A thousand clients all call at second zero, and each retries up to 12 times.
| Retry strategy | Total requests sent | Last client succeeded at |
|---|---|---|
| Fixed 1-second delay | 5,500 | 9 s |
| Exponential, no jitter | 5,500 | 302 s |
| Exponential, full jitter | 3,011 | 38 s |
Read this fairly. Plain exponential backoff without jitter sent exactly as many requests as the fixed delay, because every client was still synchronised, and it took the longest. Full jitter sent 45% fewer requests than either. The fixed delay finished fastest only because my toy server recovers instantly every second; a real overloaded service does not recover on a clock, and a fixed delay is the setting most likely to keep hammering it.
The mistake it prevents: a retry storm that turns a short blip into an outage.
4. Timeouts#
Every network call needs a deadline. Without one, a hung request holds a worker, a connection and a user forever. The Anthropic errors page lists a 504 timeout_error and warns that long non-streaming requests can reach the server's time limit before the response starts. Its SDK refuses non-streaming requests that it expects to exceed a 10-minute timeout, and the docs recommend streaming or the batch API for requests that long.
Set a timeout for each layer: connection, first byte and total. Make the total shorter than your own user-facing limit, so you can fail, retry or fall back while the user is still there. The AWS article makes the same point from the other side: pick a timeout from the latency you actually observe, rather than a round number.
The mistake it prevents: one slow model call quietly using up every worker you have.
5. Circuit breakers#
A circuit breaker watches a dependency for failures. Martin Fowler's description is the standard one: once failures reach a threshold the breaker "trips," and further calls return an error straight away without making the protected call. After a timeout it moves to a half-open state and allows a trial call, which either resets it or restarts the timeout. Fowler also notes you will usually want an alert when it trips.
For AI apps the useful partner is a fallback: when the primary model's breaker is open, route to a cheaper model, a cached answer or a polite "try again shortly". Anthropic's error docs say a 529 overload "can occur when the API experiences high traffic across all users", meaning it is not caused by your own usage and your own backoff will not fix it.
The mistake it prevents: every request waiting out a full timeout while the provider is down.
6. Rate limits and token budgets#
Provider limits are not only about request counts. Anthropic measures requests per minute, input tokens per minute and output tokens per minute for each model class, and uses a token bucket: capacity refills continuously up to your maximum, rather than resetting at fixed intervals. OpenAI documents the same idea with x-ratelimit-* headers showing your remaining requests and tokens, and a Retry-After header on temporary 429s.
RFC 6585 defines the 429 status as "too many requests in a given amount of time." Three details from the vendor pages are worth building around:
- Wait the number of seconds in
retry-afterwhen it is present. Earlier retries fail. - Not every 429 is temporary. Anthropic's spend-limit 429 carries no
retry-afterheader and says retrying, including the SDK's automatic retries, fails until access resumes. OpenAI saysRetry-Afterdoes not mean that quota or billing errors can be fixed by retrying. - A sharp rise in traffic can itself trigger a 429. Both vendors say to increase traffic gradually.
A good client keeps its own token counter, slows down before the limit rather than after it, and puts bursty work on a queue (primitive 1).
The mistake it prevents: treating every 429 as "retry harder".
7. Streaming with server-sent events#
Streaming sends the answer in pieces as it is generated. Under the hood it is server-sent events: a normal HTTP response with the text/event-stream type, where each message is a few lines starting with data:. The WHATWG spec also gives browsers a built-in EventSource that reconnects and sends a Last-Event-ID header to resume.
I measured the difference with a local server that produces 40 tokens, 0.1 seconds apart.
| Delivery | First bytes arrive | Everything arrives |
|---|---|---|
| Whole response | 4.01 s | 4.01 s |
| Streamed | 0.10 s | 4.14 s |
Total time barely changes, but the person sees words 40 times sooner. Two things in Anthropic's streaming docs catch people out. The stream can include ping events, and it can send an error event after the HTTP 200 has already been returned, for example an overloaded_error, so a check of the status code alone is not enough. Tool-call arguments also arrive as partial JSON strings, which you must collect and parse only when the block ends.
The mistake it prevents: a spinner that makes a fast model feel broken, and a stream that dies silently halfway.
8. Caching#
AI has two caches. The first is a normal response cache, useful when identical questions repeat. The second is prompt caching, where the provider stores the processed start of a long prompt so you do not pay full price for it each time.
Anthropic's documentation gives the multipliers: a 5-minute cache write costs 1.25 times the base input price, a 1-hour write costs 2 times, and a cache read costs 0.1 times on most models (a footnote lists lower multipliers on some of the newest). The minimum cacheable prompt is 512 tokens on the newest models and 2,048 on older ones. OpenAI says caching works by routing requests to servers that recently processed the same prompt, and that you can track cached_tokens in the usage fields.
A worked example from Anthropic's multipliers, my arithmetic rather than a measurement: ten calls that share a 10,000-token prefix.
| Approach | Input cost, in "10,000-token units" |
|---|---|
| No cache | 10 × 1 = 10 |
| 5-minute cache | 1 × 1.25 + 9 × 0.1 = 2.15 |
That is roughly 78% less on the repeated part. The catch is that the prefix must be identical and the calls must land inside the cache lifetime. Put stable content (instructions, tools, documents) first and the changing question last, and check the cache-hit fields in the response instead of assuming it works.
The mistake it prevents: paying full price, again and again, to re-read the same 10,000 words.
9. Structured outputs and validation#
If your code expects JSON, a model that sometimes returns almost-JSON is a bug factory. Both vendors now offer constrained output. OpenAI's Structured Outputs promise "reliable type-safety" and explicit, programmatically detectable refusals. Anthropic's structured outputs use constrained decoding and say you get valid JSON that matches your schema, so you do not need retries for schema violations.
Still validate. A schema guarantees the shape, not the truth: a perfectly formed {"refund": 9999} is still wrong. Treat the model's output like any other untrusted input: check ranges, check that IDs exist, and handle the refusal case.
The mistake it prevents: a crash on a stray trailing comma, and a confidently wrong value that passes every type check.
10. Webhooks and the async result pattern#
Some AI jobs take minutes: video generation, long document analysis, large batches. The standard shape is to accept the job, return an ID straight away, and tell the caller later by webhook (or let them poll).
Stripe's webhook documentation is the best short guide to doing it safely. Return a 2xx status quickly, before any heavy work. Verify the Stripe-Signature header against the raw request body, because any change to the body breaks verification. And expect duplicates, which is exactly why your handler needs the idempotency key idea from primitive 2.
The same pattern covers your own endpoints: a 202 Accepted with a job ID, a status URL and a signed callback. Our REST API guide explains the status codes involved.
The mistake it prevents: a handler that does 40 seconds of work before answering, gets retried by the sender, and does the work twice.
11. Observability#
You cannot fix what you cannot see, and AI failures are quiet. Every Anthropic response carries a unique request-id header, repeated as request_id in error bodies, and the docs ask you to include it when contacting support. OpenAI's prompt caching guide tells you to track cached_tokens, cache_write_tokens, latency and realised cost, and to calculate your hit rate.
At minimum, log one line per model call with: your own request ID, the provider's request ID, model name, input and output tokens, cached tokens, latency, status, retry count and the user or tenant it was for. That single table answers "why was Tuesday so expensive?" and "which customer hits 429s?" in seconds.
The mistake it prevents: a bill that doubles with no way to say which feature, model or user caused it.
12. Per-user quotas and spend guards#
The last primitive protects your wallet. OWASP's 2025 list of LLM risks includes Unbounded Consumption, where excessive and uncontrolled inference leads to service disruption and financial loss, including a "Denial of Wallet" scenario in which an attacker exploits cost-per-use pricing. Its mitigations include rate limiting and user quotas so no single source can make unlimited requests.
Provider caps are a backstop only. OpenAI notes that a hard spend limit makes affected requests return a 429, and Anthropic's spend-limit 429 has no retry-after header, so a cap stops everyone, including your honest users. Build your own limits per user, per day and per feature, cap max_tokens on every call, and decide in advance what the app does when a user hits their limit.
The mistake it prevents: one script, or one curious user, spending your month's budget in an afternoon.
A one-week learning order#
- Day 1-2: retries with jitter, then idempotency keys. Break them on purpose with a local server like the ones above.
- Day 3: timeouts and a circuit breaker around one model call.
- Day 4: read your provider's rate-limit page and log the headers you get back.
- Day 5: add streaming to one screen and handle the mid-stream error event.
- Day 6: turn on prompt caching and measure your hit rate.
- Day 7: move one slow job onto a queue with a dead-letter queue, and add a per-user limit.
For a real third-party API to practise against, any free endpoint works. Our weather API on APIHiver is one, and our API design principles guide covers the habits behind a clean one.
What I left out#
- Vector storage. Retrieval is a big part of AI apps, but it is a database choice rather than a reliability primitive. pgvector's README notes it does exact nearest-neighbour search by default and approximate search (HNSW or IVFFlat) if you add an index, trading some accuracy for speed.
- Authentication and secrets handling. Essential, and covered by every backend guide.
- Agent frameworks and orchestration. They are built from the primitives above, which is why the primitives come first. See live web data for AI agents for one example.
What I did not test#
- The three experiments ran on one laptop against local toy servers, not against OpenAI, Anthropic or any real provider, and not under real network conditions. They show the shape of each effect, not the size you will see in production.
- The retry result comes from a simulation where the server resets every second. Real servers under overload behave less neatly.
- The caching cost example is arithmetic from Anthropic's published multipliers. I did not run billed API calls, and I did not measure real cache-hit rates.
- Provider limits, prices and minimum token counts change often. All vendor figures were read from the vendors' own documentation on 9 November 2026.
Sources#
Anthropic's rate limits, errors, streaming, prompt caching, batch processing and structured outputs documentation; OpenAI's prompt caching, batch, rate limits and structured outputs guides; the Stripe API reference on idempotent requests and Stripe's webhook documentation; the IETF Idempotency-Key header draft (draft-08); the AWS Builders' Library article on timeouts, retries and backoff with jitter; Amazon SQS dead-letter queue documentation; the PostgreSQL SELECT reference; Martin Fowler's circuit breaker article; RFC 6585; the WHATWG server-sent events specification; OWASP's LLM10:2025 Unbounded Consumption page; the pgvector README; Appwrite's "What is an AI backend" and Mastering Backend's "The 6 layers every AI backend needs" for how others frame the same problem. Experiments were written and run locally in Python 3.10.