Rate Limits, Retries, and Backoff: Integrating With APIs You Don't Control

The moment your software depends on an API you do not control, you have taken on a partner who will not return your calls, changes the rules without warning, and occasionally vanishes for an afternoon. That is not a complaint about any particular provider — it is the nature of integrating with systems someone else runs. Their limits, their outages, and their bad days become yours, and you cannot fix any of it from the outside.

What you can control is how your side behaves when theirs misbehaves. The difference between an integration that quietly falls apart under real conditions and one that stays calm and correct is almost entirely in how it handles the API's rate limits, its transient failures, and its occasional disappearances. These are not edge cases to bolt on later; they are the core of the job.

Rate limits are a contract, not an insult

Nearly every serious API caps how many requests you may make in a window. People treat hitting a limit as an error to work around, but it is better read as a contract: this is the pace at which you may ask for things. Respecting that pace is part of being a well-behaved client.

The naive integration fires requests as fast as it has work and slams into the limit the first time there is a burst. A well-built one paces itself to stay within the allowance. The clean way to do that is a token bucket: you hold a bucket of tokens that refills at the permitted rate, and every request spends one. When the bucket is empty, work waits. This decouples how fast work arrives from how fast you send it — the single most useful property in a high-volume integration.

class TokenBucket {
  private tokens: number;
  private last = Date.now();

  constructor(private capacity: number, private refillPerSec: number) {
    this.tokens = capacity;
  }

  async take(): Promise<void> {
    for (;;) {
      const now = Date.now();
      this.tokens = Math.min(
        this.capacity,
        this.tokens + ((now - this.last) / 1000) * this.refillPerSec,
      );
      this.last = now;

      if (this.tokens >= 1) {
        this.tokens -= 1;
        return;
      }
      // Not enough tokens — wait for roughly one to accrue, then re-check.
      const waitMs = ((1 - this.tokens) / this.refillPerSec) * 1000;
      await new Promise((r) => setTimeout(r, waitMs));
    }
  }
}

The other half is reading what the API tells you. Good APIs return your remaining allowance in headers, and when you do hit the limit they tell you how long to wait via Retry-After. Honouring that signal — rather than blindly retrying — is the difference between backing off gracefully and being throttled harder for hammering a door that is temporarily closed.

if (res.status === 429) {
  // Respect the server's own instruction over any guess of ours.
  const retryAfter = Number(res.headers.get("retry-after")) || 1;
  await new Promise((r) => setTimeout(r, retryAfter * 1000));
  return send(); // then retry
}

Not every failure deserves a retry

Retrying is essential; retrying indiscriminately does real harm. The first discipline is telling apart the failures worth retrying from the ones that never will be.

• Transient failures a retry might fix — a timeout, a temporary server error, a rate limit, a brief network blip. Worth retrying, because the same request may well succeed a moment later.
• Permanent failures a retry will never fix — a malformed request, an authentication failure, a request for something that does not exist. Retrying just repeats a doomed call and delays the moment a human realises something is genuinely wrong.

Retrying a permanent failure is worse than pointless: it hides a real problem behind a flurry of identical attempts, so nobody notices the thing that needs fixing.

That timeout row carries a warning: retrying is only safe if the operation is idempotent, because a timeout means you genuinely do not know whether it took effect. Retry a non-idempotent create and you may have just made two of something.

Backoff: retry like you're trying to help

Once you know a failure is worth retrying, how you retry matters enormously. The instinct — try again immediately, and keep trying — is precisely wrong. If the API is struggling, a flood of instant retries is the last thing it needs, and you can turn a brief wobble into a sustained outage by piling on at the worst moment.

The established approach is exponential backoff: wait a short moment, then longer, then longer still, giving the struggling system progressively more room. Cap the wait so it does not grow absurd, and cap the attempts so a dead operation eventually gives up rather than retrying forever.

The addition that matters more than it looks is jitter — a little randomness in each wait. Without it, many operations that failed at the same instant (say, during a brief outage) all wait the identical interval and retry in perfect unison, hitting the recovering API with a synchronised wave that knocks it over again. Scattering the retries across time smooths that wave into something the other side can absorb.

async function withRetry<T>(fn: () => Promise<T>, max = 5): Promise<T> {
  for (let attempt = 0; ; attempt++) {
    try {
      return await fn();
    } catch (err) {
      if (!isRetryable(err) || attempt >= max) throw err;

      const cap = Math.min(1000 * 2 ** attempt, 30_000); // exponential, capped at 30s
      const delay = Math.random() * cap;                 // "full jitter" — 0..cap
      await new Promise((r) => setTimeout(r, delay));
    }
  }
}

   attempt   0        1          2              3
   window   [0–1s]   [0–2s]     [0–4s]         [0–8s]
            · ·      ·   ·  ·    ·    ·   ·     ·     ·   ·
            random within a growing cap → no synchronised stampede

The silent failure is the one that hurts

Ask an experienced integration engineer which failure they fear most, and it is rarely the loud crash. It is the silent stall — the integration that stops working without complaint. Nothing errors visibly, no alarm sounds, data simply stops flowing. Nobody notices until a customer asks where their order went or the month-end numbers refuse to reconcile, and by then there is a day or more of divergence to untangle.

A trustworthy integration is deliberately loud about trouble. When something goes wrong that a person must act on — retries exhausted, auth failing repeatedly, a queue backing up — someone is told promptly, with enough context to respond. Crucially, that includes detecting the absence of activity. Ordinary error alerts only fire when something actively fails; they say nothing when an integration quietly stops doing anything at all. A dead-man's switch — a check that expects a certain flow of activity and raises the alarm when it goes quiet — is what catches the silent stall the error alerts miss.

// Runs on a schedule. Fires when EXPECTED work has gone missing — the case
// ordinary error alerts are blind to, because nothing threw.
async function deadMansSwitch() {
  const lastSync = await getLastSuccessfulSyncAt();
  const stalledFor = Date.now() - lastSync.getTime();
  if (stalledFor > 15 * 60_000) {
    await alert(`No successful sync for ${Math.round(stalledFor / 60_000)}m`);
  }
}

Build a place for failures to land

Even with sensible retries, some operations will ultimately fail and stay failed. The wrong answers are dropping them silently — which loses data — or retrying forever, which clogs the system. The right answer is to set them aside deliberately, in a dead-letter queue: a holding area where operations that have exhausted their retries wait for a human.

                    ┌───────────┐  success   ┌──────────┐
   work ─────────▶  │  process  │ ─────────▶ │ provider │
                    └───────────┘            └──────────┘
                          │ retries exhausted
                          ▼
                    ┌───────────┐   alert    ┌──────────┐
                    │  dead-    │ ─────────▶ │  humans  │
                    │  letter   │            └──────────┘
                    └───────────┘
                     inspect · fix cause · replay — never lost

This does two valuable things. It stops a few permanently-failing operations from blocking everything behind them, so one poisoned request does not stall the whole pipeline. And it preserves the failures instead of losing them, so someone can investigate, fix the root cause, and replay them once resolved. A failure you have captured is a problem you can solve on your own schedule. A failure you dropped is data gone for good, usually noticed too late to recover.

Make the whole thing observable

When an integration misbehaves — and over enough time it will — you need to see what happened without guesswork: what came in, what you sent, what came back, what was retried and how many times, and where it finally ended up. Tie it together with a correlation ID stamped on every log line and outbound request.

The boring machinery is the whole value

Step back and notice what all of this shares. None of it is the interesting part — the actual API calls, the data mapping, the feature the integration exists to deliver. It is all the unglamorous machinery around those calls: pacing, retry judgement, backoff with jitter, loud failure, a place for failures to land, and enough logging to see what happened.

That machinery is exactly why integrations get underestimated so reliably. The interesting part comes together quickly and demos beautifully, which is precisely what lulls people into thinking the work is nearly done. The reliability — the boring part — is where the real engineering lives, and it decides whether the integration is dependable infrastructure or a liability that fails quietly at the worst possible time. When you depend on an API you do not control, that boring machinery is the only thing standing between you and the other side's bad day. It is worth building well.