Kubernetes Security: Identity, Configuration and Runtime

Most Kubernetes compromises do not start inside Kubernetes. They start with a credential, a pod that can reach something it should not, or a workload allowed to run with more power than it needs. The cluster then does what it was built to do: it trusts whoever holds a valid identity.

That is why cluster security reads best in three layers. Identity decides who and what can act. Configuration decides how much damage a valid identity can do. Runtime decides whether anyone notices. One weak layer is survivable. Two usually are not.

This article is about the cluster itself: its control plane, its permissions and its behavior. Image scanning and registry controls sit upstream and are covered in the related piece on container security.

Identity: three callers, three trust paths

A cluster has three kinds of caller, and each reaches the API server by a different route. Teams usually secure the first and forget the other two.

Service account tokens deserve a closer look. Since version 1.24, Kubernetes no longer creates long-lived token Secrets automatically. Tokens are issued through the TokenRequest API instead: they expire, they are bound to an audience, and they are tied to the pod that received them. Legacy Secret-based tokens still exist if someone creates one explicitly. They never expire. A legacy token that leaks once stays useful until a person deletes it.

Then there is RBAC. Role names tell you little. Judge permissions by what they allow:

• Create pods in a namespace. The holder can run a pod under any service account in that namespace and mount its secrets, unless admission policy stops it.

• Get or list Secrets, cluster-wide. That is a read on every credential the cluster stores.

• Escalate, bind or impersonate. These verbs let an identity grant itself more access.

• Wildcards in verbs or resources. They usually mean someone gave up on working out what was needed.

A periodic review that lists every subject holding these permissions is more useful than one that checks role names against a policy document.

Configuration: what the cluster allows by default

Pod admission

The Pod Security Standards define three levels. Privileged is unrestricted. Baseline blocks the known escalations: privileged containers, host namespaces and hostPath volumes among them. Restricted adds the hardening most application workloads can live with: run as non-root, no privilege escalation, all Linux capabilities dropped, and a seccomp profile set. Pod Security Admission enforces a level per namespace through labels.

The practical sequence is to run in warn and audit mode first, read what would have been blocked, fix the workloads, then enforce. Baseline across the board is a realistic first target. Restricted fits most application namespaces. System namespaces need explicit, documented exceptions rather than a silent one.

Network reach

By default, any pod can talk to any other pod. A default-deny NetworkPolicy per namespace changes that, with explicit allows for what each workload needs. One caveat: policies only work if the network plugin enforces them. Confirm that before trusting them.

Secrets at rest

Kubernetes Secrets are base64-encoded. That is encoding, not encryption. Encryption at rest has to be configured on the API server, ideally with a KMS provider so the key lives outside the cluster. Managed services handle this differently, so check what your provider encrypts and who controls the key. Even with encryption on, anyone with RBAC permission to read a Secret gets the plaintext. Encryption protects the storage layer. Permissions protect the secret.

A case worth learning from: the exposed admission controller

The ingress-nginx vulnerabilities, named IngressNightmare by the Wiz researchers who found them, show how identity and configuration failures combine. The controller ran an admission webhook that was reachable from any pod on the network without authentication. Crafted requests could inject configuration and run code inside the controller pod. That pod commonly held permission to read Secrets across namespaces, so code execution there could lead to control of the whole cluster. Wiz scored the issue 9.8 and reported more than 6,500 clusters exposing the webhook to the internet. Fixed releases were 1.12.1 and 1.11.5.

The Kubernetes project has since announced the retirement of ingress-nginx: best-effort maintenance ending, with no further releases and no fixes for flaws found later. Existing deployments keep running, which is the risk. The project points users to Gateway API or another supported ingress controller.

The lesson is wider than one product. For every controller in the cluster, ask who can reach it, what it can read and who patches it. Then find out whether you are running ingress-nginx at all. A label query for app.kubernetes.io/name=ingress-nginx answers that in a minute.

Runtime: seeing what the cluster is doing

The Kubernetes audit log records who called the API and what they asked for. On managed services it is often not collected by default. Amazon EKS, for example, does not send control plane logs to CloudWatch until each log type is enabled. If the audit log is off, there is nothing to investigate with later.

Alert on a short list of API events that rarely have a benign explanation outside of change windows:

• Exec or attach into a running pod.

• Creation of a privileged pod, or one that mounts a host path.

• A new binding to cluster-admin or another high-privilege role.

• Secrets listed across namespaces by an identity that is not a system component.

• Pods created in system namespaces by a human user.

Audit logs have a limit. They show API activity, not what an attacker does after reaching a shell in a container. For that you need telemetry from the node: process, file and network behavior. Open-source tools such as Falco, and commercial sensors built on eBPF, serve this purpose. Expect noise at first. A small set of high-signal rules, tuned against your own workloads, beats a large default rule set nobody reads.

Where to start

Hardening guides list hundreds of settings. The CIS Kubernetes Benchmark, the NSA and CISA Kubernetes Hardening Guidance and NIST SP 800-190 are good checklists. None can tell you which three matter in your cluster. The attack path does. Work in this order:

1. List every identity that can reach cluster-admin, human and workload. Remove what no one can justify.

2. Turn off service account automounting by default and delete legacy token Secrets.

3. Stop pods from reaching node metadata and from reaching each other without cause.

4. Put Pod Security Admission in warn mode on every namespace, then enforce Baseline.

5. Collect the audit log and alert on the five events above.

6. Inventory every controller and webhook: reachability, permissions, patch owner.

For leadership, the decision is ownership. Platform teams usually own the cluster and application teams own the workloads. Permissions, admission policy and runtime alerts fall between them. Name one owner for each, or each will assume the other has it.

Use this to evaluate cloud control and configuration risk: take your largest cluster and test it against the six steps above. The gaps you find first are the ones to fund first.

Sources

• Wiz Research, IngressNightmare: critical unauthenticated remote code execution vulnerabilities in Ingress NGINX. https://wiz.io/blog/ingress-nginx-kubernetes-vulnerabilities

• Kubernetes Blog, Ingress NGINX Retirement: What You Need to Know. Kubernetes project. kubernetes.io

• Kubernetes Documentation, Service Accounts. https://kubernetes.io/docs/concepts/security/service-accounts/

• Kubernetes Documentation, Pod Security Standards. https://kubernetes.io/docs/concepts/security/pod-security-standards/

• Kubernetes Documentation, Encrypting Confidential Data at Rest. https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/

• Amazon EKS User Guide, Amazon EKS control plane logging. https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html

• NSA and CISA, Kubernetes Hardening Guidance. media.defense.gov

• NIST, Special Publication 800-190, Application Container Security Guide. csrc.nist.gov

• Center for Internet Security, CIS Kubernetes Benchmark. cisecurity.org