AWS Credentials for k3s Pods, Without a Single Stored Key

Published: at 06:00 PM

The usual way to give a pod access to AWS is an access key in a Kubernetes secret. It never expires, it ends up in backups and CI logs, and rotating it is a chore you postpone. On EKS you would use IRSA instead: the pod proves who it is with its service-account token and gets short-lived credentials.

That is not an EKS feature. It is plain OIDC, and a single k3s node can do it too. My k3s server now sends mail through Amazon SES this way, and there is no AWS key anywhere on it. It took a handful of Pulumi resources, one k3s config file and no admission webhook. It also took three gotchas, which are the actual reason for this post.

How it works

  1. Kubernetes mounts a service-account token into the pod, signed by the API server, with the audience sts.amazonaws.com.
  2. The AWS SDK in the pod sends that token to STS (AssumeRoleWithWebIdentity) together with a role ARN.
  3. STS looks up the token’s issuer, fetches the issuer’s public keys over HTTPS and checks the signature.
  4. If the role trusts exactly this service account, STS returns credentials valid for an hour. The SDK refreshes them on its own.

So AWS needs three things: a public HTTPS address for your issuer, the issuer registered as an IAM OIDC provider, and a role that trusts one specific service account.

1. Put the issuer documents in S3, not in the cluster

STS fetches two small JSON files from the issuer URL:

Neither is secret. k3s serves both, but only inside the cluster, under the issuer https://kubernetes.default.svc.cluster.local, which AWS cannot reach.

You could expose them through your ingress. I did not: then every AWS call from every pod depends on your own ingress being up and reachable from AWS. A public S3 bucket is more boring and more reliable. The issuer becomes the bucket URL, and the bucket policy makes exactly two objects readable:

discovery = {
    "issuer": ISSUER_URL,  # https://<bucket>.s3.<region>.amazonaws.com
    "jwks_uri": f"{ISSUER_URL}/openid/v1/jwks",
    "response_types_supported": ["id_token"],
    "subject_types_supported": ["public"],
    "id_token_signing_alg_values_supported": ["RS256"],
}

The JWKS file is a copy of what the cluster publishes:

kubectl get --raw /openid/v1/jwks > jwks.json

Then register the issuer with IAM. No thumbprint needed:

provider = aws.iam.OpenIdConnectProvider(
    "k3s",
    url=ISSUER_URL,
    client_id_lists=["sts.amazonaws.com"],
    opts=pulumi.ResourceOptions(depends_on=[policy, discovery_doc, jwks_doc]),
)

Gotcha 1: that depends_on is not decoration. IAM fetches the discovery document at creation time. Pulumi created the provider in parallel with the bucket objects, and the first run failed with:

InvalidInput: Please check .well-known/openid-configuration of provider:
https://<bucket>.s3.eu-central-1.amazonaws.com is valid.

The documents have to be public before the provider exists.

2. Tell k3s to sign with the new issuer

The API server has to put the S3 URL into the iss claim of new tokens. k3s takes API server flags from a config file, so this is a drop-in:

# /etc/rancher/k3s/config.yaml.d/oidc.yaml
kube-apiserver-arg+:
  - service-account-issuer=https://<bucket>.s3.eu-central-1.amazonaws.com
  - service-account-issuer=https://kubernetes.default.svc.cluster.local
  - service-account-jwks-uri=https://<bucket>.s3.eu-central-1.amazonaws.com/openid/v1/jwks

Two issuers is the trick that makes this safe on a running cluster. The first one signs new tokens; every one in the list is still accepted. Tokens already mounted in pods carry the old issuer and keep working, and the kubelet replaces them with new ones on its normal refresh. The switch costs one k3s restart: a few seconds without an API server, running pods are not touched.

Check it:

kubectl create token default --audience sts.amazonaws.com \
  | cut -d. -f2 | base64 -d 2>/dev/null | jq -r .iss
# https://<bucket>.s3.eu-central-1.amazonaws.com

Does this change every token in the cluster? The issuer, yes: every new token now names the S3 URL. The audience, no. A normal pod still gets a token for https://kubernetes.default.svc.cluster.local and k3s; only tokens requested with audience: sts.amazonaws.com (step 4) carry the AWS audience. STS rejects everything else, so a leaked token from some other pod is worthless at AWS.

Gotcha 2: note the + in kube-apiserver-arg+. In k3s config files, “the last value found for a given key will be used”. A drop-in with plain kube-apiserver-arg: silently replaces any API server flags you already set in config.yaml. The + appends. Mine used the plain key at first. It only worked because config.yaml set no API server flags; it uses + now.

Gotcha 3: if a tool manages that file, check what happens when the content changes. I write it with a Pulumi remote.Command whose delete removes the file. A content change replaces the command, and Pulumi’s default is create before delete: write the new file, restart, then run the old resource’s delete, which removes the same path and restarts again. End state: no drop-in, tokens signed with the cluster-internal issuer again, STS rejects all of them, and every AWS call fails. One option fixes it:

opts=pulumi.ResourceOptions(delete_before_replace=True)

On a single node, also keep a way back that does not need the API server. Mine is a script that removes the drop-in and restarts k3s over SSH.

3. A role that trusts one service account

"Principal": {"Federated": oidc_provider_arn},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {"StringEquals": {
    f"{issuer_host}:sub": "system:serviceaccount:my-app:my-app",
    f"{issuer_host}:aud": "sts.amazonaws.com",
}},

Pin both sub and aud, with StringEquals. Without the sub condition, every service account in the cluster can assume the role. A StringLike with a wildcard is nearly as bad: any pod in that namespace qualifies.

4. The pod: no webhook needed

On EKS, a mutating webhook injects the token volume and environment variables. On k3s you can just write them into the pod spec. It is three lines of environment and one projected volume:

spec:
  serviceAccountName: my-app
  automountServiceAccountToken: false
  containers:
    - name: app
      env:
        - { name: AWS_REGION, value: eu-central-1 }
        - { name: AWS_ROLE_ARN, value: arn:aws:iam::<account>:role/my-app-ses }
        - {
            name: AWS_WEB_IDENTITY_TOKEN_FILE,
            value: /var/run/secrets/aws/token,
          }
      volumeMounts:
        - { name: aws-token, mountPath: /var/run/secrets/aws, readOnly: true }
  volumes:
    - name: aws-token
      projected:
        sources:
          - serviceAccountToken:
              audience: sts.amazonaws.com
              expirationSeconds: 3600
              path: token

Every official AWS SDK picks up AWS_ROLE_ARN plus AWS_WEB_IDENTITY_TOKEN_FILE in its default credential chain. The application code does not change. automountServiceAccountToken: false keeps the regular Kubernetes API token out of the pod; it only gets the one with the AWS audience.

In my case the consumer is ses-relay, a small sidecar that takes SMTP from Listmonk and sends it through the SES API. It has never seen an SMTP password or an access key.

What can still break

The signing key. The JWKS in S3 is a copy. If the cluster’s service-account key changes, for example after rebuilding the server, STS rejects every new token until you upload the new JWKS. Pods do not notice right away: the kubelet refreshes a one-hour token at 80% of its lifetime, and the AWS credentials from the last exchange stay valid for up to an hour. That is your window. A check that compares kubectl get --raw /openid/v1/jwks with the copy in S3 and alerts on a difference closes it.

Cost. STS reads the two documents when it validates a token. S3 charges about $0.0004 per 1,000 GET requests in eu-central-1, and both files together are under 1 KB. Even if STS fetched them on every exchange, one pod refreshing hourly is around 730 requests a month: a fraction of a cent.

The bucket is public, so anyone can request those files too, and successful requests are billed to you. That is the realistic abuse case: one million requests cost about $0.43, plus about 1 GB of transfer that the free tier covers. Requests for anything else in the bucket are denied, and S3 no longer bills you for unauthorized requests from outside your account. A budget alert in AWS Billing is enough to notice if someone tries.

What an attacker would need

Everything in this post is public by design: the issuer URL, the public key, the role’s trust policy. Knowing it does not help, because a token is only accepted if it is signed with the cluster’s private key. That key never leaves the node.

So the questions are who could get a token signed by a key AWS trusts:

What you get. No AWS key on the server, in the cluster, in CI or in a backup. The role can only be assumed by one service account in one namespace, and the credentials are useless an hour later.

Next Post
Listmonk with Amazon SES: Three Ways It Fails Quietly