Kubernetes security

A cluster is an API and a set of controllers that make reality match what the API says. Everything goes through that API: creating workloads, reading secrets, changing permissions. Securing a cluster is therefore mostly about who can call that API and what they may ask it for.

Four layers carry the risk. The control plane, meaning the API server, the scheduler, the controllers and the datastore that holds every object including secrets. Permissions, expressed as role-based access control, which is fine-grained and easy to get wrong in the direction of too much. Workload isolation, which is the boundary between a container and the node it runs on, and which is far weaker than most people assume. And the supply chain of images, since a cluster runs whatever it was told to pull.

The property that makes clusters different from other platforms is that several ordinary-looking permissions are equivalent to full control. The right to create a pod means the right to run a container on a node, mount a host path, and read anything on that node including the credentials of other workloads. The right to read secrets in a namespace where a privileged component runs means the right to become it. Neither of these is called administrator in any user interface.

Identity for workloads comes from service accounts, whose tokens are mounted into pods. Where a cluster is federated with a cloud provider, that identity extends outwards, and a compromised pod becomes a cloud principal.

The finding we write most often is a permission that reads as harmless. A continuous integration service account with the right to create workloads in a namespace, so that anyone who can influence a pipeline can run a privileged container. A developer role that includes the ability to execute into pods, which is remote code execution in every workload in the namespace. A role that can create role bindings, which is the right to grant itself anything. None of these look like administrator, all of them are.

The second is workload configuration that gives the container the host. Privileged containers, host path mounts, host networking, and capabilities that were added to make something work and never removed. A container with the host filesystem mounted is a process on the node, and container escape in that case is not an exploit, it is a file copy.

The third is secrets. In a default cluster, secret objects are base64-encoded in the datastore rather than encrypted, so read access to that datastore, or a backup of it, is read access to every secret in the cluster. And secrets mounted into pods are readable by anyone who can execute in the pod, which brings us back to the permission model.

The fourth is the network, where the default is that every pod can reach every other pod. Without policies, one compromised workload reaches the entire cluster, including the internal endpoints of components that assumed nobody could call them. On engagements this is what makes a single vulnerable application into a cluster compromise.

The fifth is the join with the cloud. A pod with a service account federated to a cloud role means an application flaw becomes a cloud credential. That is the same chain as SSRF reaching a metadata endpoint, and it ends in the same place.

Kubernetes security discussions collapse three different problems into one word. They have different owners and different tooling.

The reason to separate them is that most organisations buy tooling for the third, do some of the second through defaults, and never look at the first, which is where the routes to cluster administrator are.

Granting wildcard permissions in a role. Every verb on every resource is administrator with extra steps.

Overlooking the dangerous verbs. Creating pods, executing in pods, reading secrets, creating role bindings and impersonating are each equivalent to substantially more than they appear.

Treating a namespace as a security boundary. It is a naming and policy scope. Without network policy, admission control and separate node pools, it is not isolation.

Leaving default-allow networking. Every cluster starts flat. Somebody has to make it not flat.

Assuming a container is a sandbox. It is a set of kernel namespaces and cgroups. Configured loosely, it is a process on the host with a different view of the filesystem.

Start with permissions, because that is where the shortest paths are. Compute who can reach what through role bindings, including service accounts, and specifically enumerate every principal that can create pods, execute in pods, read secrets or modify bindings. That list should be short and it is usually not. This is Kubernetes role-based access control reviewed as a graph rather than as a set of individual manifests.

Enforce workload standards through admission rather than through review. A policy that rejects privileged pods, host path mounts and unnecessary capabilities at admission time is enforced; guidance in a wiki is not. Use the built-in pod security standards as the baseline and an admission controller for anything beyond them.

Apply default-deny network policy per namespace and open what is needed, which is the cluster version of microsegmentation and is the single change that most limits a compromised workload.

Encrypt secrets in the datastore, restrict access to the datastore and its backups, and prefer an external secrets manager with short-lived credentials over long-lived secret objects. Disable automatic mounting of service account tokens where a workload does not call the API, which is most of them.

For detection, the audit log is the highest-value source and is frequently not collected. Alert on pod creation with privileged settings, on execution into pods outside a change window, on role binding changes, and on secret reads by principals that do not normally read them.

A cluster assessment is written as reachability from a starting position rather than as a list of settings: from a compromised application pod, what can be reached, and how far does it go. That framing is what makes the report actionable, because the client can see which single policy breaks the chain.

We record the permission or configuration that permitted each step, quoted from the cluster’s own objects so the platform team can verify it, along with the audit log entries our activity produced, which frequently reveals that the log was not being collected.

Severity follows the escape and the join. A misconfigured workload with no route out of its namespace is a moderate finding. The same workload on a node that also runs privileged components, in a cluster federated to a cloud account, is critical because the path continues past the cluster.

Clusters are assessed as part of testing a cloud environment and what runs in it, where the interesting findings usually cross the boundary between the two.

Is a container a security boundary? Not by default. It is a set of kernel namespaces and control groups sharing the host kernel. Configured strictly it is a reasonable boundary; configured with privileges, host mounts or extra capabilities it is not a boundary at all.

What is the most dangerous Kubernetes permission? Creating pods is the usual answer, because it allows running a container that mounts the host or uses the node’s identity. Executing in pods, reading secrets and creating role bindings are close behind, and none of them are labelled as administrative.

Are Kubernetes secrets encrypted? In a default cluster they are base64-encoded in the datastore, which is not encryption. Encryption at rest for secrets has to be configured explicitly, and access to the datastore and its backups has to be restricted accordingly.

Does a service mesh secure the cluster? It secures traffic between services, which is valuable, and it addresses one layer. It does nothing about over-broad permissions, privileged workloads or the image supply chain.