The community Terraform providers I used in my last App Store post mirror Apple’s API one object per locale. So I wrote a native Pulumi provider for the App Store Connect API, for store listings, translations, markets, prices, in-app purchases and the release flow. While I built it, the API returned a lot of HTTP 500s. My first instinct was to give the provider the full distributed-systems kit: retries, exponential backoff and a circuit breaker.
Before writing the circuit breaker, I checked what other Terraform and Pulumi providers do. None of the ones I looked at use one. And the closest thing that did ship, in the AWS provider, ended up being switched off by default. This post covers what I built instead, what Apple’s API taught me on the way, and how I test it.
A provider is not a service
Circuit breakers come from long-running services. Many callers hit one dependency, and when that dependency fails you stop sending traffic so it can recover and your threads don’t pile up.
A provider is a different kind of process:
- It lives for one
pulumi up, for a few seconds or minutes. - It works for one user, on a bounded set of resources.
- Its load on the API is tiny compared with the API’s other traffic.
- The Pulumi engine already decides what runs, in what order, and what to do when something fails.
A breaker inside that process doesn’t protect the API in any meaningful way. What it does is couple resources together: resource B fails because resource A had bad luck a moment earlier. That makes failures nondeterministic, and it replaces the API’s real error message with “circuit open”.
What the ecosystem actually does
- The GitHub Terraform provider has
max_retries,retry_delay_msandretryable_errors, which defaults to 500, 502, 503 and 504. Someone suggested a circuit breaker in its issue tracker, and it was never built (integrations/terraform-provider-github#210). - The Confluent provider has
max_retrieswith an environment-variable fallback (confluentinc/terraform-provider-confluent#290). - The AWS provider is the cautionary tale. The AWS SDK for Go v2 has a retry
token bucket shared by every request a client makes, which works much like a
circuit breaker. After the Terraform AWS provider moved to SDK v2, users hit
failed to get rate limit token, retry quota exceededon large applies, even though nothing was wrong with any individual request. Since v5.42.0 (March 2024) the provider turns that quota off by default and lets users opt back in withtoken_bucket_rate_limiter_capacity(hashicorp/terraform-provider-aws#36024).
The pattern across all of them is bounded retries per request, a list of retryable errors, and backoff. Nothing is shared across requests that can make one resource fail because of another.
The checklist
1. Classify every response
| Response | Action |
|---|---|
| 500, 502, 503, 504, network errors, per-request timeouts | Retry with backoff, but only requests that are safe to repeat (see 3) |
| 429 | Retry. Honour Retry-After up to five minutes, and slow down when the rate-limit header says 2% of the budget or less is left |
| 401 | Mint a new token once and retry once (expiry or clock skew), then fail with a message that names the key, issuer and clock |
| 404 on a read right after a create | Retry for up to 10 seconds. Apple’s reads lag its writes |
| Other 4xx | Never retry. These are validation and state errors, so show the API’s own message |
Apple sends its budget on every response, as
X-Rate-Limit: user-hour-lim:3600;user-hour-rem:3590;. Read it; it is cheaper
than finding the limit with a 429.
The 404 row surprised me. A create returns 201 with the new object, and a GET of that object a moment later can still answer 404. Every read that directly follows a create in the same operation gets a short 404 window. No other read does, because a 404 there means the object is really gone.
2. Bounded backoff that respects the deadline
Exponential backoff with full jitter: base 1 s, factor 2, cap 30 s, at most 6
attempts. Pulumi’s customTimeouts arrive in a Go provider as a context
deadline, so the client never sleeps past it:
// backoff is exponential with full jitter. attempt counts from 0.
func backoff(attempt int, base, maxDelay time.Duration, rnd func() float64) time.Duration {
ceil := base
for i := 0; i < attempt && ceil < maxDelay; i++ {
ceil *= 2
}
ceil = min(ceil, maxDelay)
return time.Duration(rnd() * float64(ceil))
}
The loop instead of a bit shift is deliberate: base << attempt overflows for a
large attempt and turns into a negative duration.
There’s one trap with Retry-After. Apple’s rate limit is counted over a
rolling hour. If the API tells you to come back in 3,600 seconds, do not sleep
for an hour inside pulumi up. Fail with a clear message instead.
3. Writes: reconcile before you retry
This is the part that matters most, and the part a generic retry library gets wrong. A POST that returned 500 may still have been applied. App Store Connect has no idempotency keys, so retrying blindly creates duplicates.
Check what your HTTP library does by default. go-retryablehttp’s default
policy (v0.7.8) retries connection errors, 429 and 5xx without looking at the
method, so it re-sends POSTs too.
“Reconcile” turned out to mean four different things, picked per object type:
- PATCH with absolute values, and DELETE: safe to retry. Treat a 404 on a DELETE retry as success. When a PATCH runs out of retries on a 5xx or a timeout, read the object back before reporting the error. If it already shows the new values, the update succeeded. If not, return the error together with the state Apple actually has.
- Creates whose key the API keeps unique (a product ID, a subscription group name): after an ambiguous failure, wait out the backoff, look the object up, and adopt it if it’s there. Otherwise POST again. One twist: the second POST can come back 409 “already used” while the lookup still shows nothing. The first POST landed and Apple’s list hadn’t caught up. Look up once more and adopt. Without that, a successful create reports a name clash.
- Creates of things the API keeps any number of: send once, never again. Apple accepted two review submissions for the same platform, both 201, and then refused to cancel the empty one. Uploading the same file twice gives two separate assets. After an ambiguous failure, poll the lookup with backoff for a few seconds and adopt what shows up. If nothing does, fail; the next run’s lookup adopts what landed late.
- Irreversible actions (submit for review, release to the store): never retry blindly. Re-read the state, and if the action already happened, it counts as success.
The read-back after a failed PATCH is a Pulumi thing. Terraform refreshes before every plan, so the Terraform AWS provider can return the error and let the next plan see what really happened. Pulumi doesn’t refresh by default. If the provider returns only the error, state keeps the old values and the next preview is built on a guess. Pulumi’s azure-native provider reads back after an ambiguous failure, and so does mine.
Also check the other way round. Some creates are idempotent on their own: a second POST of a capability that a bundle ID already has answers 201 with the same id. That one can be retried freely, but I only know because I tried.
4. Stop early and record what happened
When one request runs out of retries, the resource stops. It doesn’t keep going
through the other 39 locales. It returns the state it actually applied; in
pulumi-go-provider that’s infer.ResourceInitFailedError for a create, or
partial state on an update. The next pulumi up picks up from there.
This is what limits an outage, and it does the job people want a circuit
breaker for. During a full outage, each request gives up after about 30 seconds
of backoff, each resource stops at its first failed request, and up fails
within minutes with an honest error. Nothing is shared across resources, so
nothing fails because of something unrelated.
5. A concurrency cap, not a retry quota
One semaphore per provider process (default 4) smooths out the spikes when the engine runs many resources in parallel. It limits how many requests are in flight. It never makes a request fail.
A global cap wasn’t enough for one part of Apple’s API, though. I sent eight capability creates for one bundle ID at once. All eight answered 201. Afterwards one or two were on the bundle ID. Eight deletes sent at once answered 404 for capabilities that were there. And a rename of the bundle ID, sent while a capability create was in flight, answered 200 next to the create’s 201, and the capability was gone. Apple rewrites the bundle ID’s whole capability set on every write, its name included.
So writes to one bundle ID now go through a lock per bundle ID and run one at a time, and each one is checked against the list afterwards. Everything else still runs in parallel. If an API stores children as one set on the parent, expect the same, and test it: every single response said success.
6. Log the retries, never the secrets
- Log each retry at Info with the method, path, status, the API’s error code,
the request ID and the attempt number:
retrying DELETE /v1/appStoreVersionLocalizations/... after HTTP 500 (Apple code UNEXPECTED_ERROR, request ...), attempt 2/6, waiting 1.4s. Users then see that the API is degraded even when the run succeeds, and the request ID ties the failure to a request on Apple’s side. - After a run that needed retries, print one Warning summary per resource: “3 requests needed retries (HTTP 500)”.
- When retries run out on 5xx or network errors, say the API looks degraded and link its status page.
- Never log the
Authorizationheader, the token or the private key. Write a test that checks this.
7. Test the failure modes on purpose
The provider’s tests run against a fake API with a fault switch. It has eight kinds of fault: 500, 502, 503, 504, 429, a connection reset, a hang, and the nasty one, apply the write, then return 500. The fault choice is seeded, so a failing run repeats exactly with the same seed. On top, single faults can be aimed at one method and path for the next request.
The tests that matter:
- At a 30% fault rate, the full lifecycle of every resource family (create, update, delete for releases, products, media and bundle IDs) converges to the right final state, with no duplicates and no double submissions.
- A total outage fails fast, with the “API looks degraded” message.
- An outage in the middle of an update records the partial state, and the next run finishes the job.
- A bad key fails after exactly one new token.
- No error message and no log line contains the token or the key.
The real API then added faults the fake didn’t have. Deleting a localization
answered 500 UNEXPECTED_ERROR in two of four runs and 204 on a retry three
seconds later. A plain GET timed out after 60 seconds. And the fake was wrong in at
least one place: it answered 204 to racing capability deletes where Apple
answered 404. A fake encodes your assumptions about the API, so it drifts. A
nightly run of the same programs against the real API, on a test app, is what
keeps it honest.
When a circuit breaker does make sense
Long-running processes that call the same dependency continuously: controllers, operators, reconcilers, backend services. There the breaker protects your own resources and gives the dependency room to recover. A CLI-driven provider process is none of those things.
So for a provider: bounded retries per request, reconcile before re-sending
writes, stop at the first hard failure, and let the engine and the next
pulumi up do the rest.