Kubernetes is the industry standard for container orchestration.
According to the Cloud Native Computing Foundation’s 2022 CNCF annual survey, only 30% of survey respondents “have adopted cloud native approaches across nearly all development and deployment activities. Still, 62% of organizations that do not regularly use cloud native techniques have containers for pilot projects or limited production use cases, indicating there is room for growth”. Forty percent of respondent organizations stated that security and lack of training are their main challenges when deploying containers in production.
In this context, Policy as Code tools can help avoid misconfigurations and other security issues faced by cloud native adopters. This article presents the Kyverno tool and how to use it to overcome misconfigurations and security issues in cloud native deployments. For a better understanding of the Policy as Code technique, read Introduction to Policy as Code.
About Kyverno
Kyverno is an Open-Source CNCF incubating Policy as Code tool that manages policies as Kubernetes resources using the YAML format. It provides a Command Line Interface (CLI) that can be used in a CI/CD pipeline, and a Kubernetes Admission Controller.
Kyverno can be explored in a test environment using k3s. According to the official installation guide, it can be installed in standalone mode using a Helm chart as follows:
helm install kyverno kyverno/kyverno -n kyverno --create-namespace
In addition to the Kubernetes Admission Controller, it is best to install the Kyverno CLI because policies are often developed with a Test-Driven approach.
The Kyverno CLI can be installed via krew, with your OS package manager, or by downloading the corresponding binary from the releases page.
Policies with Kyverno
This section discusses policies with Kyverno and provides examples to illustrate concepts and demonstrate uses.
Detecting Kubernetes Workloads Misconfigurations
As discussed in the Introduction to Policy as Code article, Kubernetes workload misconfigurations pose one of the main challenges encountered by cloud native services. The use of Policy as Code can help mitigate such misconfigurations earlier in a CI/CD pipeline or during the admission phase with Kyverno Admission Controller. Some of these misconfigurations can be mitigated using Kyverno policies.
A deployment with a single replica can cause the application to be unavailable during the replacement phase in the event of a node crash [2023 Kubernetes Benchmark Report]. The policy shown in Figure 1 addresses this situation by checking if the replicas attribute is equal to or greater than 2.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: minimum-replicas
annotations:
policies.kyverio.io/title: Check replicas
policies.kyverio.io/category: Best Pratices
policies.kyverio.io/severity: low
policies.kyverio.io/subject: Deployment replica
policies.kyverio.io/description: >-
Check the number of the minimum replicas
spec:
validationFailureAction: audit
rules:
- name: validate-pod-controller
match:
any:
- resources:
kinds:
- Deployment
- StatefulSet
validate:
message: >-
The minimum replicas should be >= '2'
pattern:
spec:
replicas: ">=2"
Listing 1: Minimum Replicas Policy
Note that a Kyverno policy comprises one or more policy rules and additional settings, such as the failure action (validationFailureAction), all of which are coded as Kubernetes resources using YAML files. These policies can be configured as either cluster-wide (using the ClusterPolicy resource) or namespaced (using the Policy resource).
In the example in Figure 1, the match statement under the rules definition is responsible for filtering the input by the kinds attribute, thus only evaluating the policy for Deployment and StatefulSet resources. The validationFailureAction attribute defines how Kyverno behaves during a policy violation. A value of audit does not block the deployments, though it will generate a policy violation report, while a value of enforce blocks deployments, and only pass evaluations will be reported.
Policy rules also define a pattern that will validate resource data during the evaluation phases. In the example policy from Figure 1, it checks if the input manifest spec.replicas has two or more replicas.
Policies in Kyverno can have Validate, Mutate, Cleanup, Generate, or ImageValidate rules. The most common policies use Validate rules, such as the minimum-replica policy. Mutate rules are used when there is a need to modify the resource, which occurs before validation, and can be used to add an Istio sidecar injection label to a namespace as shown in Figure 2.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: add-sidecar-injection-namespace
annotations:
policies.kyverno.io/title: Add Istio Sidecar Injection
policies.kyverno.io/category: Istio
policies.kyverno.io/severity: medium
kyverno.io/kyverno-version: 1.8.0
policies.kyverno.io/minversion: 1.6.0
kyverno.io/kubernetes-version: "1.24"
policies.kyverno.io/subject: Namespace
policies.kyverno.io/description: >-
In order for Istio to inject sidecars to workloads deployed into Namespaces, the label
`istio-injection` must be set to `enabled`. As an alternative to rejecting Namespace definitions which don't already contain this label, it can be added automatically. This policy adds the label `istio-inject` set to `enabled` for all new Namespaces.
spec:
rules:
- name: add-istio-injection-enabled
match:
any:
- resources:
kinds:
- Namespace
mutate:
patchStrategicMerge:
metadata:
labels:
istio-injection: enabled
Figure 2: Mutate Policy
Resource mutation is done via the patchStrategicMerge attribute or JSON Patch. The patchesJson6902 method can be used when a mutation cannot be performed using patchStrategicMerge.
Detecting the Use of Kubernetes’ Unnecessary Capabilities
Workloads using capabilities beyond the application requirements is another common misconfiguration case. Some capabilities are enabled by default in Kubernetes, and according to the 2023 Kubernetes Benchmark Report, many organizations do not turn them off.
Figure 3 shows a policy excerpt that demonstrates how those capabilities can be dropped, along with other Kyverno features.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: drop-all-capabilities
spec:
validationFailureAction: audit
background: true
rules:
- name: require-drop-all
match:
any:
- resources:
kinds:
- Pod
preconditions:
all:
- key: "{{ request.operation || 'BACKGROUND' }}"
operator: NotEquals
value: DELETE
validate:
message: >-
Containers must drop `ALL` capabilities.
foreach:
- list: request.object.spec.[ephemeralContainers, initContainers, containers][]
deny:
conditions:
all:
- key: ALL
operator: AnyNotIn
value: "{{ element.securityContext.capabilities.drop[].to_upper(@) || `[]` }}"
Figure 3: Drop Capability Policy
This policy also demonstrates the use of the foreach declaration to process lists. In this case, the items are ephemeralContainers, initContainer and containers. A Deny rule verifies that all capabilities are dropped; otherwise, the service deployment will fail.
Preconditions can be used to control when a given rule should run. In Figure 3, the statement all indicates that all conditions must be evaluated as TRUE for a successful validation (it acts as a logical AND).
In cases where the precondition should act as a logical OR, the statement any can be used, as shown in Figure 4.
rules:
- name: any-all-rule
match:
any:
- resources:
kinds:
- Deployment
preconditions:
any:
- key: "{{ request.object.metadata.labels.color || '' }}"
operator: Equals
value: blue
- key: "{{ request.object.metadata.labels.app || '' }}"
operator: Equals
value: busybox
Figure 4: Preconditions with Any Statement
Test-driven Policy Development
During policy development, it is crucial to follow a test-driven approach. This ensures any error can be caught before the policies are deployed onto a Kubernetes cluster. The Kyverno CLI can be used in those cases in a CI/CD pipeline, as well as the developer workstation.
Figure 5 shows the sections present in a Kyverno policy test case, where name is the test case name; policies contain the policy file names; resources contain the Kubernetes resource files to be validated; and results define the expected validation result for each resource under test.
apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata:
name: require-pod-probes
policies:
- policy.yaml
resources:
- require-probes.yaml
results:
- kind: Pod
policy: require-pod-probes
resources:
- badpod01
- badpod02
result: fail
rule: validate-probes
- kind: Pod
policy: require-pod-probes
resources:
- goodpod01
- goodpod02
- goodpod03
- goodpod04
result: pass
rule: validate-probes
Figure 5: Kyverno Policy Test
To run policy tests the Kyverno test command is used, and it accepts some flags as listed in Table 1.
Kyverno Test Flags |
Description |
-f |
Test file (default: kyverno-test.yaml) |
-v |
Log level |
-t <test case> |
Run specified test cases |
-b |
Test git repository branch |
Table 1 – Kyverno Test Flags
Some policies need to interact with the Kubernetes API, and in those cases, we need to provide a values file in YAML format to test cases via variables attribute, as shown in Figure 6. This is necessary because the policies that run via CLI are not able to call the Kubernetes API; thus, the expected response is not available.
apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata:
name: example-policy
policies:
- example-policy.yaml
resources:
- ./resource.yaml
results:
- policy: example-policy
rule: example-policy-rule
resource: example-resource
kind: Pod
result: pass
variables: values.yaml
Figure 6: Test Case Variables with YAML File
Kyverno does not enforce any folder structure, but it is a good practice to place kyverno-test.yaml under the tests folder, application manifests into the resources folder and have all policies organized by category as shown in Figure 7.
Figure 7: Example Folder Structure for Policies
Defining policies as Kubernetes resources lowers the entry barrier for teams aiming to adopt policy as code. It offers an extensive policy library that can be deployed to a Kubernetes cluster using GitOps principles with Flux or ArgoCD and help mitigate the common misconfiguration issues presented herein. This makes Kyverno an excellent option to have in your DevOps toolchain.
Key Takeaways
- Kyverno makes it easy to develop policies as Kubernetes resources
- Policies can run in a CI/CD pipeline or in Kubernetes
- An extensive policy library lowers the adoption barrier
- It integrates well with GitOps tools
Acknowledgements
This piece was written by Marcelo da Silva Pires – Senior DevOps Engineer, and João Longo – Systems Architect and Innovation Leader at Encora’s Engineering Technology Practices group. Thanks to João Caleffi and André Scandaroli for reviews and insights.