Introduction
Over the past year, during our Active Cloud Security Penetration Testing engagements, we have consistently identified a pattern of recurring misconfigurations in our clients' environments, particularly in their Argo Workflows instances. These misconfigurations have created exploitable conditions, allowing us to compromise clusters, escalate privileges, and conduct lateral movements - ultimately gaining Kubernetes Cluster-Admin access.
In this blog post, we’ll explore the root cause of these misconfigurations, the potential impact and how to deal with them.
What is Argo Workflows?
Argo Workflows is an open-source workflow orchestration tool for Kubernetes and is widely adopted for automating workflows in cloud environments. A workflow is a series of steps organized in a specific sequence to achieve a particular goal, such as data processing, machine learning model training, or CI/CD pipelines. Workflows are commonly used in modern cloud-native environments to streamline complex processes, improve efficiency, and reduce manual effort.
Why is Argo Workflows an attractive attack surface?
Argo Workflows is a powerful automation tool for orchestrating CI/CD pipelines, but its default configurations and common deployment practices can introduce serious security risks.
Argo Workflows servers, by default, are deployed without authentication (when using the QuickStart K8s manifest - quick-start-minimal.yaml ), allowing anyone with network access to interact with them without credentials. This misconfiguration poses a significant risk, enabling unauthorized users to execute workflows and access Kubernetes resources.
During the research, we discovered approximately 3,000 misconfigured Argo Workflows instances that were left publicly available (Based on an internet-wide scanning API search conducted in November 2024).
The following are key reasons why Argo Workflows is considered an appealing target for attackers:
- Extensive Permissions and Access: Argo Workflows deployments often have broad permissions to Kubernetes and cloud resources, allowing attackers to execute code within the Kubernetes environment under privileged roles or groups.
- Exposure of Secrets and Keys: Secrets and keys are commonly embedded within workflow steps, making them accessible to attackers if the workflows are compromised.
- Unauthenticated Access: In some cases, Argo Workflows instances are deployed without authentication, enabling attackers to gain unrestricted access without proper credentials.
- Potential for Lateral Movement: Workflows can execute in environments connected to different networks, giving attackers an opportunity to pivot between internal organizational networks.
How to Exploit an Argo Workflows Instance?
#1: Identifying and Validating Argo Workflows Instances
As part of our assessment process, the E.V.A Cloud Research Team begins by identifying and validating open Argo Workflows instances. We use techniques like network scanning to detect exposed services and endpoints, and leverage internet-scanning tools such as Shodan to locate publicly accessible instances. By analyzing service responses and recognizing the default web interface, we confirm the presence of these instances and the authentication method in use.
To verify whether the instances are unauthenticated, we can query the /api/v1/userinfo endpoint (or any other authenticated API available). If the server responds with a valid JSON object that includes information about the instance’s serviceAccountName, it is usually an indication that the instance is using an insecure, unauthenticated default method.
Misconfigured instances can be identified by reviewing their arguments and validating the configured authentication method specified by the --auth-mode argument
Once unauthenticated access to the instance is confirmed, all actions are authorized by default using the argo-server Service Account from the argo namespace.
This Service Account is bound by default to a privileged ClusterRoleBinding, granting it a ClusterRole with elevated permissions. These permissions enable the creation and execution of workflows across the entire cluster, including all namespaces.
This default configuration significantly increases the attack surface, as an unauthenticated attacker can exploit these privileges to execute arbitrary workflows and potentially compromise the entire Kubernetes environment.
The argo-server ClusterRole is configured to create workflows enabling it to create pods in any namespace across the cluster.
#2: Creating a Malicious Workflow for Code Execution
With access to an unauthenticated Argo Workflows instance, the next step involves crafting a malicious workflow to achieve code execution within the Kubernetes environment. Argo Workflows simplifies this process by allowing users to define workflows as YAML files, which can include commands or scripts to be executed.
We crafted a new workflow designed to deploy a controlled container image that executes a reverse shell command. To escalate privileges, the workflow was configured to be executed with an admin-level ServiceAccount, granting it full permissions to perform unrestricted actions on any resource within the cluster.
The image below highlights several key specifications:
- Service Account: admin — a privileged ServiceAccount with elevated permissions.
- Namespace: argo — the namespace where this ServiceAccount was created.
- Image: Python — the latest Python Docker image, chosen for its versatility in executing commands.
- Command and Arguments: Configured to establish a reverse shell.
#3: Kubernetes Cluster Takeover
After executing the above workflow, a reverse shell was created in our controlled server:
By specifying the admin Service Account within the workflow, we could leverage the Service Account's permissions to perform any action associated with Kubernetes Cluster-Admin privileges.
The following demonstrates the process of obtaining the Service Account token mounted to our container.
Using the admin Service Account token, we can authenticate against the Kubernetes API and validate permissions to perform any action on any resource in the cluster.
How Can Organizations Defend Themselves?
1. Configure Argo Workflows Authentication
To secure Argo Workflows, it is crucial to implement proper authentication methods. According to Argo's documentation, access to the instance can be configured using the following authentication methods:
- Server Mode: Use the server's Service Account for hosted instances or your local kubeconfig for local instances.
- Client Mode: Authenticate with the Kubernetes bearer token of clients.
- Single Sign-On (SSO): Use SSO to authenticate users. By default, SSO uses the server's Service Account for RBAC unless SSO RBAC is explicitly enabled.
We strongly recommend organizations adopt either Client Mode or SSO authentication. These methods require users to provide valid credentials before accessing the instance or creating workflows, effectively reducing the risk of unauthorized access.
2. Scope Permissions to Specific Namespace
By default, the argo-server-cluster-role ClusterRole in Argo Workflows is not scoped to any specific namespace, granting it broad permissions to create resources across all namespaces within the cluster. This lack of namespace restriction increases the risk of lateral movement and privilege escalation within the cluster.
To mitigate this, we recommend scoping the permissions of the argo-server-cluster-role to specific namespaces.
By restricting permissions to only the necessary namespaces, organizations can limit the potential attack surface and reduce the impact of a compromised Argo Workflows instance.
Bottom Line
Argo is a critical component of modern CI/CD workflows and a high-value target for attackers. Exploiting vulnerabilities in Argo can enable attackers to achieve the following:
- Malicious Artifact Deployment: Injecting malicious artifacts or compromised code into the CI/CD pipeline to gain control over production systems. This can lead to supply chain attacks, compromising dependencies consumed by downstream applications.
- Privilege Escalation: Exploiting misconfigurations such as overly permissive service accounts or weak RBAC policies to perform unauthorized actions, including:some text
- Modifying deployment manifests or creating malicious workflows.
- Accessing sensitive secrets stored in the system.
- Deploying unauthorized workloads to target environments.
- Escalating privileges to administrative levels, potentially gaining full control over the ArgoCD environment.
