Elastic
Start free trial
Loading

Kubernetes Suspicious Assignment of Controller Service Account

Elastic Stack Serverless Security

This rule detects a request to attach a controller service account to an existing or new pod running in the kube-system namespace. By default, controllers running as part of the API Server utilize admin-equivalent service accounts hosted in the kube-system namespace. Controller service accounts aren’t normally assigned to running pods and could indicate adversary behavior within the cluster. An attacker that can create or modify pods or pod controllers in the kube-system namespace, can assign one of these admin-equivalent service accounts to a pod and abuse their powerful token to escalate privileges and gain complete cluster control.

Rule type: query

Rule indices:

  • logs-kubernetes.*

Severity: medium

Risk score: 47

Runs every: 5m

Searches indices from: None (https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html#date-math[Date Math format], see also Additional look-back time)

Maximum alerts per execution: 100

References:

Tags:

  • Data Source: Kubernetes
  • Tactic: Execution
  • Tactic: Privilege Escalation
  • Resources: Investigation Guide

Version: 7

Rule authors:

  • Elastic

Rule license: Elastic License v2

Investigation guide

Triage and analysis

[TBC: QUOTE]
Investigating Kubernetes Suspicious Assignment of Controller Service Account

Kubernetes uses service accounts to manage pod permissions, with controller service accounts in the kube-system namespace having elevated privileges. Adversaries may exploit this by assigning these accounts to pods, gaining admin-level access. The detection rule identifies such suspicious assignments by monitoring audit logs for pod creation events in the kube-system namespace with controller service accounts, flagging potential privilege escalation attempts.

Possible investigation steps

  • Review the audit logs to confirm the presence of a "create" event for a pod in the "kube-system" namespace with a service account name containing "controller".
  • Identify the source of the request by examining the user or service account that initiated the pod creation event in the audit logs.
  • Check the history of the involved service account to determine if it has been used in any other suspicious activities or unauthorized access attempts.
  • Investigate the pod’s configuration and associated resources to understand its purpose and whether it aligns with expected operations within the cluster.
  • Assess the potential impact by evaluating the permissions and roles associated with the controller service account assigned to the pod.
  • Review recent changes or deployments in the "kube-system" namespace to identify any unauthorized modifications or anomalies.

False positive analysis

  • Routine maintenance tasks in the kube-system namespace may involve creating or modifying pods with elevated service accounts. Review the context of such actions to determine if they are part of scheduled maintenance or updates.
  • Automated deployment tools might temporarily assign controller service accounts to pods for configuration purposes. Verify if these actions align with known deployment processes and consider excluding these specific tools from triggering alerts.
  • Legitimate testing or debugging activities by cluster administrators could involve using controller service accounts. Ensure these activities are documented and consider creating exceptions for known testing environments.
  • Some monitoring or logging solutions might require elevated permissions and could inadvertently trigger this rule. Validate the necessity of these permissions and whitelist these solutions if they are deemed non-threatening.
  • Regularly review and update the list of known benign service account assignments to ensure that only unexpected or unauthorized assignments are flagged.

Response and remediation

  • Immediately isolate the affected pod by cordoning the node it is running on to prevent further scheduling of pods and drain the node if necessary to stop the pod from executing.
  • Revoke the service account token associated with the suspicious pod to prevent further unauthorized access or actions using the compromised credentials.
  • Conduct a thorough review of recent changes in the kube-system namespace to identify unauthorized modifications or deployments, focusing on the creation and modification of pods and service accounts.
  • Reset credentials and rotate keys for any service accounts that may have been compromised to ensure that any stolen credentials are rendered useless.
  • Implement network policies to restrict pod-to-pod communication within the kube-system namespace, limiting the potential lateral movement of an attacker.
  • Escalate the incident to the security operations team for further investigation and to determine if additional clusters or systems have been affected.
  • Enhance monitoring and alerting for similar activities by ensuring audit logs are comprehensive and that alerts are configured to detect unauthorized service account assignments promptly.

Setup

The Kubernetes Fleet integration with Audit Logs enabled or similarly structured data is required to be compatible with this rule.

Rule query

event.dataset : "kubernetes.audit_logs"
  and kubernetes.audit.annotations.authorization_k8s_io/decision:"allow"
  and kubernetes.audit.verb : "create"
  and kubernetes.audit.objectRef.resource : "pods"
  and kubernetes.audit.objectRef.namespace : "kube-system"
  and kubernetes.audit.requestObject.spec.serviceAccountName:*controller

Framework: MITRE ATT&CKTM