Securing Kubernetes: Don’t Underestimate the Risk Posed by Misconfigurations
Deployed by more than 60% of organizations worldwide, Kubernetes (K8s) is the most widely adopted container-orchestration system in cloud computing. K8s clusters have emerged as the preferred solution for practitioners looking to orchestrate containerized applications effectively, so these clusters often contain various software, services, and resources, enabling users to deploy and scale applications with relative ease.
To support a typical K8s environment operation, a cluster is often granted access to other environments such as artifact repositories, CI/CD environments, databases etc. Thus, K8s clusters can store customer data, financial records, intellectual property, access credentials, secrets, configurations, container images, infrastructure credentials, encryption keys, certificates, and network or service information. With so many clusters containing potentially valuable and lucrative data exposed to the internet, K8s provides a tempting target for threat actors. This risk escalates with the number of organizations that have misconfigurations that leave K8s clusters exposed and vulnerable to attacks.
In recent research, Aqua Nautilus discovered that in a period of three months, more than 350 reputable organizations — some in the Fortune 500 — and open-source projects were completely exposed to the world. This exposure was for a period of several days to several months. Exploited by a threat actor, these misconfigurations could have resulted in a severe security breach.
In the cases of exposed clusters of open source projects, if exploited by attackers they may result in a supply chain infection vector with implications for millions of users. Aqua Nautilus researchers found that it could take just one Shodan search for an organization’s misconfigured cluster to be identified.
What’s at Risk in an Exposed Cluster?
Over a three-month period, Aqua Nautilus conducted a series of separate searches using Shodan. From these searches, the team pinpointed just over 350 distinct IP addresses connected to at-risk K8s API servers. At least 60% of them were breached and had an active campaign that deployed malware and backdoors.
K8s clusters often contain secrets and unauthenticated access to the API server may enable access to these secrets, so open access to it enables an attacker to take full control over the cluster. Even worse, K8s clusters usually don’t only store their own secrets. In many instances, the K8s cluster is a part of the organization's software development life cycle (SDLC), so it grants access to source code management (SCM), continuous integration/continuous deployment (CI/CD), registries, and the cloud service provider (CSP).
The secrets often contain information about internal or external registries. In many cases, developers constructed a configuration file of a registry (such as “.dockerconfig”), containing links to other environments and secrets or credentials (including Docker Hub, Cloud Service Provider, and internally managed ones). Threat actors can use these credentials to expand their reach. They can even poison the registry (if the key allows that) to run malicious code on further systems in the network.
Real-World Exposure Examples
The team discovered live examples of unsecured K8s API servers containing a wide range of additional secrets associated with various environments. These include SCM environments like GitHub, CI platforms like Jenkins, various registries such as Docker Hub, external database services like Redis or PostgreSQL, and many others. SCM access tokens allow an attacker to access an organization’s code. In some cases, attackers can even modify it to damage the organization (if the key allows that).
Some of the misconfigured clusters identified were only accessible for a few hours, but Aqua Nautilus’ data collection tools managed to identify and record the exposed information. This highlights a sobering truth about such misconfigurations; even if promptly detected and corrected, a well-prepared attacker with automation capabilities can still gain access to the K8s cluster at a later stage or infiltrate various other elements of the SDLC. In security, automation is very much a two-way street. With just one, limited instance of secret exposure, threat actors may be able to gain authenticated access to your cluster at will.
What Misconfigurations Should Organizations be Aware of?
Aqua Nautilus research identified two common misconfigurations widely found in organizations and actively exploited in the wild.
1. Anonymous User With High Privileges
When creating a new cluster, there are four things you need to take into consideration: 1) if the API server is exposed to the internet; 2) who is authorized to communicate with the cluster; 3) which privileges they have; and 4) if there are any further access controls. In many cases (both in native K8s environments and even on some cloud providers’ managed clusters), the cluster is set to be open to the internet by default, and there are many practical reasons to do that. But, this also means that if the IP address on which the API server is hosted is scanned, it will reveal that this is a K8s cluster and anyone can try connecting to the cluster.
Additionally, in many cases, unauthenticated requests to the K8s cluster are enabled by default. This means that the cluster will receive requests from anyone. However, requests by anonymous users have no privileges, by default. Thus they will result in 403 replies, which forbid access. Lastly, admission controls need someone to proactively create them.
The team has seen cases in which practitioners bind the anonymous user role with other roles, often with admin roles, which puts their clusters in danger. A mixture of these misconfigurations can allow attackers to gain unauthorized access to the Kubernetes cluster, potentially compromising all applications running on it as well as other environments.
As a result, your organization could be only one YAML away from disaster. A simple misconfiguration, an easy mistake can lead to an exposed cluster. Here’s a real-world example of the mistake:
2. The 'Kubectl Proxy' Command
Another misconfiguration that Nautilus observed wasn’t previously seen or published about: the ‘kubectl proxy’ command. When using the ‘kubectl proxy’ command with specific flags. Some publications encourage practitioners to use the ‘kubectl proxy’ command for a number of purposes. For instance, there are tutorials about Kubernetes Dashboard installation that encourage users to run the proxy command and do not explicitly warn about possible implications.
When running ‘kubectl proxy’, you're forwarding authorized and authenticated requests to the API server. When running the command ‘kubectl proxy’ with the following flags ‘--address=`0.0.0.0` --accept-hosts `.*`’, the proxy on the workstation will now listen and forward authorized and authenticated requests to the API server from any host that has HTTP access to the workstation, as demonstrated in the image below. Note: the privileges are the same as those of the user who ran the ‘kubectl proxy’ command.
The Threat Is Real
Aqua Nautilus recorded many attacks in the wild against their honeypots underscoring the threat to exposed K8s clusters. In fact, 60% of the clusters were actively under attack by cryptominers. Three primary examples are the Lchaia/xmrig, the SSWW attack and the Dero campaign. Researchers also discovered the RBAC Buster campaign, which exploits the RBAC to create a very hidden backdoor. Finally, the team reported a novel highly aggressive campaign by TeamTNT (part 1, part 2). In this campaign, TeamTNT is searching for and collecting cloud service providers tokens (AWS, Azure, GCP etc).
Tips for Mitigation
To avoid the risk of the number of active threat campaigns targeting these misconfigurations, securing cloud resources and protecting your clusters, Nautilus recommends that organizations follow these five steps:
- Train Employees: Organizations must invest in training their staff about the potential risks, best practices, and correct configurations. This will minimize human errors leading to such misconfigurations.
- Secure the `kubectl proxy`: Ensure that the `kubectl proxy` is not exposed to the internet. It should be set up within a secure network environment and accessible only by authenticated and authorized users.
- Use Role-Based Access Control (RBAC): RBAC is a native Kubernetes feature that can limit who can access the Kubernetes API and what permissions they have. Avoid assigning the admin role to an anonymous user. Make sure to assign appropriate permissions to each user and strictly adhere to the principle of least privilege.
- Implement Admission Control policies: Kubernetes Admission Controllers can intercept requests to the Kubernetes API server before persistence, enabling you to define and enforce policies that bolster security. The team strongly recommends admission controls that prevent binding any role with the anonymous role, which will harden the security posture of your Kubernetes clusters.
- Audit Regularly: Implement regular auditing of your Kubernetes clusters. This allows you to track each action performed in the cluster, helping in identifying anomalies and taking quick remedial actions.
By employing these mitigation strategies, organizations can significantly enhance their Kubernetes security, ensuring that their clusters are safe from common attacks.
As Kubernetes grows and is used by more businesses, it is imperative that practitioners become familiar with the risks it could bring to organizations. It is critical to remember that security in K8s is not a static, set-it-and-forget-it situation. There are constant actions and events occurring in the cluster, and just because your cluster was secure yesterday does not mean it remains secure today. Because of this, manually checking your clusters for known misconfigurations is not enough. If your organization uses K8s there is always a risk if you aren’t actively scanning.
