Kubernetes Security Monitoring

Security is one of the harder challenges of running Kubernetes. The reason it is difficult is that there are multiple moving layers in a cloud native stack, so operators may not focus on security early on. Hence, detecting and addressing threats is important but also challenging at the same time.

Another contributor is that some distributions of Kubernetes may not be secure by default, contrary to what can be assumed.

As Kubernetes continues to grow in adoption, it is important to know how to secure it.

Information Security process has multiple strategies and activities, but we can group them all into three distinct phases — prevention, detection, and response.

The first reflex when thinking about securing a Kubernetes cluster is to think about preventing the compromise by follow security best practices and other configuration-based measures. However, it is very important to setup good detection mechanisms to be able to act quickly to limit the blast radius in case of compromise.

This article touches on the detection part and how to setup Kubernetes security monitoring to enable quick identification of any potential misuse of the cluster improving its resiliency.

K8s monitoring vs K8s security monitoring

Kubernetes monitoring is the use of tools and processes to keep track of Kubernetes workloads and clusters. Through robust monitoring, administrators can identify patterns that could grow into larger system issues later on. Important areas to monitor include resource metrics like CPU utilization, and API metrics like request rates. Since misconfigurations are a leading cause of cloud security breaches, it’s also important to monitor configurations to identify any security issues before workloads are deployed.

On the other hand:

Kubernetes security monitoring focus on security aspects to detect anomalous container activities, unauthorized access attempts, supply chain vulnerabilities, or identity‑based threats to ensure a unified and proactive defense against evolving threat.

Components of a K8s security monitoring system

Each new container increases your application’s attack surface, or the number of potential entry points for unauthorized access. Without complete visibility into every managed container and application request, you can easily overlook gaps in your application’s security as well as malicious activity.

The cloud threat landscape has evolved significantly over the last years, requiring organizations to adopt cloud detection and response with a multi‑layered approach as part of cloud native application protection platform.

K8s monitoring sources (defense in depth)

Here are the most important components that can form such approach:

K8s audit policies and logs

Kubernetes audit logs provide a complete record of activity (e.g., the who, where, when, and how) in a Kubernetes control plane. Monitoring audit logs can be invaluable in helping to detect and mitigate misconfigurations or abuse of Kubernetes resources before confidential data is compromised.

When a request comes in to the Kubernetes API server, it can create one of several different audit events such as creating a new pod or service account. The server filters these events through an audit policy. An audit policy is a set of rules that specifies which audit events should be recorded and where they should be sent, for example to either a JSON log file or an external API backend for storage.

The scope of cluster activity that Kubernetes will capture with audit logs depends on the audit policy’s configuration and the levels set for each of your resources. It’s important that the policy collects the data required for monitoring Kubernetes security. Otherwise, it may not be possible to easily spot legitimate threats to the applications.

Audit logs record requests to the Kubernetes API because the API server processes all changes to Kubernetes state, it is an ideal point for capturing all changes that occur within a cluster. The API server processes requests from various internal and external components, including:

• The control plane (built-in controllers, the scheduler)
• Node daemons (the kubelet, kube-proxy, and others)
• Cluster services (e.g., the cluster autoscaler, kube-state-metrics, CoreDNS, etc.)
• Users making kubectl requests
• Applications, controllers, and operators that send requests through a kube client
• Even the API server itself

Logs from the Cloud Host/VM level

Multi‑layered detection mechanisms, such as host‑based runtime security, log analysis, and vulnerability host scanning, can identify and block suspicious activities, detect anomalous behavior, and provide real‑time protection against attacks targeting hosts in the cloud. That’s why it’s important to have deep runtime detection capabilities at the host level.

According to a study by RedHat, over 60% of Kubernetes security incidents in 2020 were related to misconfigured or vulnerable hosts.

Logs from Containers

Multi‑layered detection mechanisms for container security, including image scanning, runtime protection, and behavioral analysis, help identify and block malicious activities, detect container‑based attacks, and enforce security policies to mitigate risks associated with workload attacks on containers.

Kubernetes Runtime Security

Kubernetes runtime security is the protection of containers (or pods) against active threats once the containers are running.

It’s impossible to guarantee that attackers won’t sneak malware into a container image by compromising your source code repository or build tools, for example, or that your Kubernetes RBAC policies and security contexts guarantee perfect isolation between workloads.

Runtime security helps protect workloads against a variety of threats that could emerge after containers have been deployed, such as:

• Privilege escalation attacks in which a container exploits security bugs in the container runtime, Kubernetes, or the host OS to access resources it should not be able to access (like storage volumes or external binaries).
• The deployment of unauthorized containers by an attacker who exploits a gap in an access control policy or a bug in Kubernetes.
• Unauthorized access to secrets or other sensitive information that a running container should not be able to read, but which it manages to access due to improper access control configurations or a security vulnerability that enables privilege escalation.
• The activation of malware that is hidden inside a container image.

That’s why runtime security is so important. It’s the final layer of defense against threats that may creep into a Kubernetes environment. While you can, and should, take all reasonable measures to mitigate security risks within the pipeline and cluster architecture, you also need runtime security to help alert you to and control threats that sneak past your other defenses.

Kubernetes itself offers relatively little in the way of runtime security tooling. The only real resource it provides is auditing, which help generate logs that track resource requests to the Kubernetes API.

While Kubernetes can record this information, Kubernetes doesn’t do anything on its own to analyze audit logs or alert you to activity that may be suspicious.

External Runtime Security tools

So, instead of turning to Kubernetes itself to secure your environment against runtime threats, you’ll need to lean on external runtime security tools. For Kubernetes, these tools fall into two main categories: enforcement tools and auditing tools.

Enforcement tools: allow you to define policies that restrict the access rights and permissions of resources within a runtime environment. While the tools can’t prevent a threat from materializing, they can minimize its impact by ensuring that, for example, malware that appears within one container can’t access resources external to the container.

• Seccomp: Seccomp is a Linux kernel-level tool that you can use to force processes to run in a secure state. When you use seccomp to place a process in a secure state, the kernel will prevent the process from making any system calls other than exit, sigreturn, and reading and writing to files that are already open.
• SELinux: SELinux is a kernel module that lets you define a broad array of access controls that the kernel enforces. You can use SELinux to set granular rules over which resources a container can access and which types of actions it can perform.
• AppArmor: AppArmor is also a kernel module that enables the definition of a broad set of access control rules. It’s very similar in most ways to SELinux; the debate about whether to use SELinux or AppArmor is like the one about vi vs. emacs: some folks just prefer one tool or the other, but you can achieve the same general results with either solution.

To an extent, you can think of Kubernetes RBAC policies and security contexts as also providing a certain level of enforcement control because they can be used to do things like prevent a container from running in privileged mode or disallowing access to kernel-level resources. However, the difference between RBAC policies and security contexts on the one hand, and tools like SELinux and AppArmor on the other, is that the latter enforce access controls at the kernel level. That means that, even if Kubernetes itself is compromised in some way or has a security vulnerability, kernel-level enforcement tools can mitigate the impact of a security breach by preventing compromised containers from accessing external resources.

SELinux, AppArmor, and seccomp are also handy because they can be used to control access for any type of Linux workload. They are not specific to Kubernetes. If you are managing an environment that includes a mix of containers and VMs, then, you may find kernel-level access control frameworks useful because you can use the same tooling to manage all of your workloads.

Auditing Tools for Runtime Security

Again, Kubernetes provides the tooling for generating audit logs, but it doesn’t actually analyze the logs for you. For that, you’ll want to use a tool like Falco, an open source threat detection engine. Falco lets you define rules that will trigger alerts if the Falco engine detects the presence of certain conditions based on data like Kubernetes audit logs.

The Falco engine is designed to detect host‑based threats while services and applications are running. In doing so, Falco detects anomalous, suspicious behavior in real time.

Best Practices for Kubernetes Runtime Security

While deploying enforcement and auditing tools are among the steps required to address Kubernetes runtime security threats, you’ll want to go further than just setting up these tools. You should also ensure that you:

• Perform continuous image scanning: Although image scanning isn’t a security runtime operation per se (it falls within the scope of pipeline security), image scanning is important for preventing malware from reaching a runtime environment.
• Scan Kubernetes policy files: You can also help prevent runtime security issues by scanning Kubernetes RBAC files, deployments, and other resources to detect misconfigurations that could enable or exacerbate a runtime breach.
• Scan external policy files: You should also scan policy files or profiles that you create for SELinux, AppArmor, or other frameworks. Oversights within these files could weaken your runtime defenses.
• Don’t forget dev/test: When applying runtime security defenses, you’re likely to focus first and foremost on your production environment. After all, that’s where threats tend to have the greatest impact. But don’t forget to secure dev/test as well. Catching runtime threats in dev/test is a great way to prevent them from reaching production. Plus, even in a dev/test environment, a runtime security issue could have major consequences if, for example, an attacker is able to access secret data from within dev/test resources.
• Make a plan for incident remediation: Don’t wait until a breach occurs to start planning your response. Develop playbooks that will guide your team in handling different types of runtime security events. What will you do in the case of a privilege escalation attack, for example? What if one node is compromised but other nodes are not? Thinking through these issues beforehand will help you respond more quickly to live threats.

Runtime security is the only thing standing between your cluster and threats that have managed to elude the other defenses you erected within your application development pipeline and your Kubernetes architecture.

But by continuously auditing your runtime environments, as well as using enforcement tools to segment runtime resources from each other, you can minimize the potential impact of runtime threats.