Skip to content

Retries, not circuit breakers: making a Pulumi provider survive a flaky API

Published: at 06:00 PM

The community Terraform providers I used in my last App Store post mirror Apple’s API one object per locale. So I wrote a native Pulumi provider for the App Store Connect API, for store listings, translations, markets, prices, in-app purchases and the release flow. While I built it, the API returned a lot of HTTP 500s. My first instinct was to give the provider the full distributed-systems kit: retries, exponential backoff and a circuit breaker.

Before writing the circuit breaker, I checked what other Terraform and Pulumi providers do. None of the ones I looked at use one. And the closest thing that did ship, in the AWS provider, ended up being switched off by default. This post covers what I built instead, what Apple’s API taught me on the way, and how I test it.

A provider is not a service

Circuit breakers come from long-running services. Many callers hit one dependency, and when that dependency fails you stop sending traffic so it can recover and your threads don’t pile up.

A provider is a different kind of process:

A breaker inside that process doesn’t protect the API in any meaningful way. What it does is couple resources together: resource B fails because resource A had bad luck a moment earlier. That makes failures nondeterministic, and it replaces the API’s real error message with “circuit open”.

What the ecosystem actually does

The pattern across all of them is bounded retries per request, a list of retryable errors, and backoff. Nothing is shared across requests that can make one resource fail because of another.

The checklist

1. Classify every response

ResponseAction
500, 502, 503, 504, network errors, per-request timeoutsRetry with backoff, but only requests that are safe to repeat (see 3)
429Retry. Honour Retry-After up to five minutes, and slow down when the rate-limit header says 2% of the budget or less is left
401Mint a new token once and retry once (expiry or clock skew), then fail with a message that names the key, issuer and clock
404 on a read right after a createRetry for up to 10 seconds. Apple’s reads lag its writes
Other 4xxNever retry. These are validation and state errors, so show the API’s own message

Apple sends its budget on every response, as X-Rate-Limit: user-hour-lim:3600;user-hour-rem:3590;. Read it; it is cheaper than finding the limit with a 429.

The 404 row surprised me. A create returns 201 with the new object, and a GET of that object a moment later can still answer 404. Every read that directly follows a create in the same operation gets a short 404 window. No other read does, because a 404 there means the object is really gone.

2. Bounded backoff that respects the deadline

Exponential backoff with full jitter: base 1 s, factor 2, cap 30 s, at most 6 attempts. Pulumi’s customTimeouts arrive in a Go provider as a context deadline, so the client never sleeps past it:

// backoff is exponential with full jitter. attempt counts from 0.
func backoff(attempt int, base, maxDelay time.Duration, rnd func() float64) time.Duration {
	ceil := base
	for i := 0; i < attempt && ceil < maxDelay; i++ {
		ceil *= 2
	}
	ceil = min(ceil, maxDelay)
	return time.Duration(rnd() * float64(ceil))
}

The loop instead of a bit shift is deliberate: base << attempt overflows for a large attempt and turns into a negative duration.

There’s one trap with Retry-After. Apple’s rate limit is counted over a rolling hour. If the API tells you to come back in 3,600 seconds, do not sleep for an hour inside pulumi up. Fail with a clear message instead.

3. Writes: reconcile before you retry

This is the part that matters most, and the part a generic retry library gets wrong. A POST that returned 500 may still have been applied. App Store Connect has no idempotency keys, so retrying blindly creates duplicates.

Check what your HTTP library does by default. go-retryablehttp’s default policy (v0.7.8) retries connection errors, 429 and 5xx without looking at the method, so it re-sends POSTs too.

“Reconcile” turned out to mean four different things, picked per object type:

The read-back after a failed PATCH is a Pulumi thing. Terraform refreshes before every plan, so the Terraform AWS provider can return the error and let the next plan see what really happened. Pulumi doesn’t refresh by default. If the provider returns only the error, state keeps the old values and the next preview is built on a guess. Pulumi’s azure-native provider reads back after an ambiguous failure, and so does mine.

Also check the other way round. Some creates are idempotent on their own: a second POST of a capability that a bundle ID already has answers 201 with the same id. That one can be retried freely, but I only know because I tried.

4. Stop early and record what happened

When one request runs out of retries, the resource stops. It doesn’t keep going through the other 39 locales. It returns the state it actually applied; in pulumi-go-provider that’s infer.ResourceInitFailedError for a create, or partial state on an update. The next pulumi up picks up from there.

This is what limits an outage, and it does the job people want a circuit breaker for. During a full outage, each request gives up after about 30 seconds of backoff, each resource stops at its first failed request, and up fails within minutes with an honest error. Nothing is shared across resources, so nothing fails because of something unrelated.

5. A concurrency cap, not a retry quota

One semaphore per provider process (default 4) smooths out the spikes when the engine runs many resources in parallel. It limits how many requests are in flight. It never makes a request fail.

A global cap wasn’t enough for one part of Apple’s API, though. I sent eight capability creates for one bundle ID at once. All eight answered 201. Afterwards one or two were on the bundle ID. Eight deletes sent at once answered 404 for capabilities that were there. And a rename of the bundle ID, sent while a capability create was in flight, answered 200 next to the create’s 201, and the capability was gone. Apple rewrites the bundle ID’s whole capability set on every write, its name included.

So writes to one bundle ID now go through a lock per bundle ID and run one at a time, and each one is checked against the list afterwards. Everything else still runs in parallel. If an API stores children as one set on the parent, expect the same, and test it: every single response said success.

6. Log the retries, never the secrets

7. Test the failure modes on purpose

The provider’s tests run against a fake API with a fault switch. It has eight kinds of fault: 500, 502, 503, 504, 429, a connection reset, a hang, and the nasty one, apply the write, then return 500. The fault choice is seeded, so a failing run repeats exactly with the same seed. On top, single faults can be aimed at one method and path for the next request.

The tests that matter:

The real API then added faults the fake didn’t have. Deleting a localization answered 500 UNEXPECTED_ERROR in two of four runs and 204 on a retry three seconds later. A plain GET timed out after 60 seconds. And the fake was wrong in at least one place: it answered 204 to racing capability deletes where Apple answered 404. A fake encodes your assumptions about the API, so it drifts. A nightly run of the same programs against the real API, on a test app, is what keeps it honest.

When a circuit breaker does make sense

Long-running processes that call the same dependency continuously: controllers, operators, reconcilers, backend services. There the breaker protects your own resources and gives the dependency room to recover. A CLI-driven provider process is none of those things.

So for a provider: bounded retries per request, reconcile before re-sending writes, stop at the first hard failure, and let the engine and the next pulumi up do the rest.


Next Post
270 GB of images my k3s node never cleaned up