The Difference Between an Integration That Works and One You Can Trust

I get called in to fix integrations more often than I get called in to build them, and the failures rhyme. Someone wired two systems together, it demoed beautifully, it shipped. Then a few weeks later an order silently failed to sync, nobody noticed for a day, and now the storefront and the accounting system disagree about what actually happened — and reconciling them by hand is somebody's whole afternoon.

The gap between the demo and that afternoon is the entire discipline of building integrations you can trust. It has almost nothing to do with the happy path — the part you show people — and almost everything to do with what happens when things go wrong. With systems you don't control, things go wrong constantly.

Real APIs are hostile environments

The demo works because you controlled the conditions. Production is not controlled. In production, the API you depend on will, sooner or later, do all of this:

• Rate-limit you at the worst possible moment — usually mid-batch.
• Return a 200 OK with incomplete or subtly wrong data.
• Time out and leave you genuinely unsure whether your request took effect.
• Change its behaviour after an update you were never told about.
• Go down entirely for a while.

None of this is exotic. This is the normal weather of integrating with someone else's system. An integration written for clear skies fails the first time it rains, which is to say almost immediately.

An integration isn't the code that runs when everything works. It's the code that runs when everything doesn't.

Idempotency is the decision everything else rests on

If you take one idea from this, take this one. An operation is idempotent if performing it twice has the same effect as performing it once.

Here is why it matters so much. In a distributed system you will frequently not know whether an operation succeeded. You send a request to create an order; the connection drops before the response comes back. Did the order get created? You cannot tell from where you're standing. Your only safe move is to try again — and if that operation isn't idempotent, the retry creates a second order.

The fix is to attach a stable, unique key to each logical operation, and have the receiving system recognise and ignore duplicates. Most mature APIs support this directly through an idempotency key header:

async function createOrder(order: Order) {
  // A stable key derived from *our* domain, not a random per-attempt value —
  // so every retry of the same logical order carries the same key.
  const idempotencyKey = `order:${order.id}`;

  return fetchWithRetry("https://api.vendor.com/v1/orders", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "Idempotency-Key": idempotencyKey,
    },
    body: JSON.stringify(order),
  });
}

The subtle part is where the key comes from. It has to be derived from your own data — the order's ID — so that a retry of the same logical order always produces the same key. Generate a fresh random key per attempt and you've built an elaborate way to create duplicates. Almost every serious integration bug I've untangled traces back, in the end, to a write that wasn't idempotent and a retry that therefore did damage.

Retries, but with judgement

Once operations are safe to retry, you need a retry strategy. "Try again immediately, forever" is not one — it hammers a system that's already struggling and can turn a brief blip into a full outage you caused.

The standard approach is exponential backoff with jitter: wait a little and retry, then wait longer, then longer still, up to a cap — with a bit of randomness so that a thousand failed operations don't all retry in perfect lockstep and re-create the stampede.

async function fetchWithRetry(url: string, init: RequestInit, attempt = 0) {
  const res = await fetch(url, init);
  if (res.ok) return res;

  // Only retry what can actually improve on a retry.
  const retryable = res.status === 429 || res.status >= 500;
  if (!retryable || attempt >= 5) return res;

  const base = Math.min(1000 * 2 ** attempt, 30_000); // cap at 30s
  const jitter = Math.random() * base * 0.3;
  await new Promise((r) => setTimeout(r, base + jitter));

  return fetchWithRetry(url, init, attempt + 1);
}

The other half of judgement is knowing what not to retry. A 429 or a 503 will likely succeed later. A 400 (malformed request) or 403 (permission) will fail identically every time — retrying just wastes effort and, worse, delays the moment a human realises there's a real problem to fix.

Fail loudly

Here's the failure mode that quietly does the most damage: the integration that breaks silently. Data stops flowing. Nothing crashes, no alarm sounds, so nobody notices — until a customer complains or the month-end numbers don't reconcile. By then you have a day of divergence to reverse-engineer.

A trustworthy integration is loud about failure. When something goes wrong that a human needs to act on, a human is told — quickly, and with enough context to act. That means monitoring the thing that actually matters: not "is the service up" but "is data flowing and are operations succeeding." The most valuable alert I deploy is a dead-man's switch — one that fires when an expected event hasn't happened in a while. It catches the silent stall that ordinary error alerts, which only fire when something actively throws, will always miss.

And when an operation exhausts its retries, it shouldn't vanish. It should land in a dead-letter queue — a holding area for failures — so it can be inspected and replayed rather than lost:

                    ┌─────────────┐   success   ┌─────────────┐
   order.created ─▶ │  integrate  │ ──────────▶ │   vendor    │
                    └─────────────┘             └─────────────┘
                          │  exhausted retries
                          ▼
                    ┌─────────────┐   alert     ┌─────────────┐
                    │ dead-letter │ ──────────▶ │   humans    │
                    └─────────────┘             └─────────────┘
                     replayable, never lost

Make the flow observable

When an integration misbehaves — and eventually it will — you need to see what happened without archaeology. That means logging each operation's journey in a way you can actually query: what arrived, what you sent, what came back, what was retried, and what ultimately happened, all tied together by a correlation ID.

Without this, debugging production is guesswork dressed up as investigation. With it, an integration becomes operable — someone who isn't you can answer "why didn't this sync?" and fix it. Observability isn't a nice-to-have you bolt on later; it's part of what makes the system runnable at all.

The boring parts are the point

Notice that none of this is the clever part. The API calls themselves are usually a few lines. The engineering value is almost entirely in the unglamorous machinery around them: idempotency, backoff, dead-lettering, monitoring, correlation logging.

This is exactly why integrations get underestimated. The interesting part — the demo — really does take an afternoon. The reliability is where the actual work lives, and it's the part that decides whether you've built an asset or a liability with a nice first impression.

What trust actually buys you

An integration you can trust changes how the business runs. People stop double-checking it. Manual reconciliation disappears. The systems stay in agreement without anyone watching them. And when something does go wrong — because eventually it will — you find out immediately, you can see exactly what happened, and you fix it before it becomes a mess anyone else notices.

That's the whole difference between an integration that works and one you can trust. The first is a demo. The second is infrastructure. Only one of them is worth paying for.