Policy as Code with Kyverno

Kubernetes is the industry standard for container orchestration.

According to the Cloud Native Computing Foundation’s 2022 CNCF annual survey, only 30% of survey respondents “have adopted cloud native approaches across nearly all development and deployment activities. Still, 62% of organizations that do not regularly use cloud native techniques have containers for pilot projects or limited production use cases, indicating there is room for growth”. Forty percent of respondent organizations stated that security and lack of training are their main challenges when deploying containers in production.

In this context, Policy as Code tools can help avoid misconfigurations and other security issues faced by cloud native adopters. This article presents the Kyverno tool and how to use it to overcome misconfigurations and security issues in cloud native deployments. For a better understanding of the Policy as Code technique, read Introduction to Policy as Code.

About Kyverno

Kyverno is an Open-Source CNCF incubating Policy as Code tool that manages policies as Kubernetes resources using the YAML format. It provides a Command Line Interface (CLI) that can be used in a CI/CD pipeline, and a Kubernetes Admission Controller.

Kyverno can be explored in a test environment using k3s. According to the official installation guide, it can be installed in standalone mode using a Helm chart as follows:

helm install kyverno kyverno/kyverno -n kyverno --create-namespace

In addition to the Kubernetes Admission Controller, it is best to install the Kyverno CLI because policies are often developed with a Test-Driven approach.

The Kyverno CLI can be installed via krew, with your OS package manager, or by downloading the corresponding binary from the releases page.

Policies with Kyverno

This section discusses policies with Kyverno and provides examples to illustrate concepts and demonstrate uses.

Detecting Kubernetes Workloads Misconfigurations

As discussed in the Introduction to Policy as Code article, Kubernetes workload misconfigurations pose one of the main challenges encountered by cloud native services. The use of Policy as Code can help mitigate such misconfigurations earlier in a CI/CD pipeline or during the admission phase with Kyverno Admission Controller. Some of these misconfigurations can be mitigated using Kyverno policies.

A deployment with a single replica can cause the application to be unavailable during the replacement phase in the event of a node crash [2023 Kubernetes Benchmark Report]. The policy shown in Figure 1 addresses this situation by checking if the replicas attribute is equal to or greater than 2.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: minimum-replicas
  annotations:
    policies.kyverio.io/title: Check replicas
    policies.kyverio.io/category: Best Pratices
    policies.kyverio.io/severity: low
    policies.kyverio.io/subject: Deployment replica
    policies.kyverio.io/description: >-
      Check the number of the minimum replicas
    spec:
      validationFailureAction: audit
      rules:
        - name: validate-pod-controller
          match:
            any:
            - resources:
                kinds:
                  - Deployment
                  - StatefulSet
          validate:
            message: >-
              The minimum replicas should be >= '2'
            pattern:
              spec:
                replicas: ">=2"

Listing 1: Minimum Replicas Policy

Note that a Kyverno policy comprises one or more policy rules and additional settings, such as the failure action (validationFailureAction), all of which are coded as Kubernetes resources using YAML files. These policies can be configured as either cluster-wide (using the ClusterPolicy resource) or namespaced (using the Policy resource).

In the example in Figure 1, the match statement under the rules definition is responsible for filtering the input by the kinds attribute, thus only evaluating the policy for Deployment and StatefulSet resources. The validationFailureAction attribute defines how Kyverno behaves during a policy violation. A value of audit does not block the deployments, though it will generate a policy violation report, while a value of enforce blocks deployments, and only pass evaluations will be reported.

Policy rules also define a pattern that will validate resource data during the evaluation phases. In the example policy from Figure 1, it checks if the input manifest spec.replicas has two or more replicas.

Policies in Kyverno can have Validate, Mutate, Cleanup, Generate, or ImageValidate rules. The most common policies use Validate rules, such as the minimum-replica policy. Mutate rules are used when there is a need to modify the resource, which occurs before validation, and can be used to add an Istio sidecar injection label to a namespace as shown in Figure 2.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-sidecar-injection-namespace
  annotations:
    policies.kyverno.io/title: Add Istio Sidecar Injection
    policies.kyverno.io/category: Istio
    policies.kyverno.io/severity: medium
    kyverno.io/kyverno-version: 1.8.0
    policies.kyverno.io/minversion: 1.6.0
    kyverno.io/kubernetes-version: "1.24"
    policies.kyverno.io/subject: Namespace
    policies.kyverno.io/description: >-
      In order for Istio to inject sidecars to workloads deployed into Namespaces, the label
      `istio-injection` must be set to `enabled`. As an alternative to rejecting Namespace definitions which don't already contain this label, it can be added automatically. This policy adds the label `istio-inject` set to `enabled` for all new Namespaces.      
spec:
  rules:
  - name: add-istio-injection-enabled
    match:
      any:
      - resources:
          kinds:
          - Namespace
    mutate:
      patchStrategicMerge:
        metadata:
          labels:
            istio-injection: enabled

Figure 2: Mutate Policy

Resource mutation is done via the patchStrategicMerge attribute or JSON Patch. The patchesJson6902 method can be used when a mutation cannot be performed using patchStrategicMerge.

Detecting the Use of Kubernetes’ Unnecessary Capabilities

Workloads using capabilities beyond the application requirements is another common misconfiguration case. Some capabilities are enabled by default in Kubernetes, and according to the 2023 Kubernetes Benchmark Report, many organizations do not turn them off.

Figure 3 shows a policy excerpt that demonstrates how those capabilities can be dropped, along with other Kyverno features.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: drop-all-capabilities
spec:
  validationFailureAction: audit
  background: true
  rules:
    - name: require-drop-all
      match:
        any:
        - resources:
            kinds:
              - Pod
      preconditions:
        all:
        - key: "{{ request.operation || 'BACKGROUND' }}"
          operator: NotEquals
          value: DELETE
      validate:
        message: >-
          Containers must drop `ALL` capabilities.
        foreach:
          - list: request.object.spec.[ephemeralContainers, initContainers, containers][]
            deny:
              conditions:
                all:
                - key: ALL
                  operator: AnyNotIn
                  value: "{{ element.securityContext.capabilities.drop[].to_upper(@) || `[]` }}"

Figure 3: Drop Capability Policy

This policy also demonstrates the use of the foreach declaration to process lists. In this case, the items are ephemeralContainers, initContainer and containers. A Deny rule verifies that all capabilities are dropped; otherwise, the service deployment will fail.

Preconditions can be used to control when a given rule should run. In Figure 3, the statement all indicates that all conditions must be evaluated as TRUE for a successful validation (it acts as a logical AND).

In cases where the precondition should act as a logical OR, the statement any can be used, as shown in Figure 4.

rules:
- name: any-all-rule
  match:
    any:
    - resources:
        kinds:
          - Deployment
  preconditions:
   any:
   - key: "{{ request.object.metadata.labels.color || '' }}"
     operator: Equals
     value: blue
   - key: "{{ request.object.metadata.labels.app || '' }}"
     operator: Equals
     value: busybox

Figure 4: Preconditions with Any Statement

Test-driven Policy Development

During policy development, it is crucial to follow a test-driven approach. This ensures any error can be caught before the policies are deployed onto a Kubernetes cluster. The Kyverno CLI can be used in those cases in a CI/CD pipeline, as well as the developer workstation.

Figure 5 shows the sections present in a Kyverno policy test case, where name is the test case name; policies contain the policy file names; resources contain the Kubernetes resource files to be validated; and results define the expected validation result for each resource under test.

apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata:
  name: require-pod-probes
policies:
- policy.yaml
resources:
- require-probes.yaml
results:
- kind: Pod
  policy: require-pod-probes
  resources:
  - badpod01
  - badpod02
  result: fail
  rule: validate-probes
- kind: Pod
  policy: require-pod-probes
  resources:
  - goodpod01
  - goodpod02
  - goodpod03
  - goodpod04
  result: pass
  rule: validate-probes

Figure 5: Kyverno Policy Test

To run policy tests the Kyverno test command is used, and it accepts some flags as listed in Table 1.

Table 1 – Kyverno Test Flags

Some policies need to interact with the Kubernetes API, and in those cases, we need to provide a values file in YAML format to test cases via variables attribute, as shown in Figure 6. This is necessary because the policies that run via CLI are not able to call the Kubernetes API; thus, the expected response is not available.

apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata:
  name: example-policy
policies:
  - example-policy.yaml
resources:
  - ./resource.yaml
results:
  - policy: example-policy
    rule: example-policy-rule
    resource: example-resource
    kind: Pod
    result: pass
variables: values.yaml

Figure 6: Test Case Variables with YAML File

Kyverno does not enforce any folder structure, but it is a good practice to place kyverno-test.yaml under the tests folder, application manifests into the resources folder and have all policies organized by category as shown in Figure 7.

Figure 7: Example Folder Structure for Policies

Defining policies as Kubernetes resources lowers the entry barrier for teams aiming to adopt policy as code. It offers an extensive policy library that can be deployed to a Kubernetes cluster using GitOps principles with Flux or ArgoCD and help mitigate the common misconfiguration issues presented herein. This makes Kyverno an excellent option to have in your DevOps toolchain.

Key Takeaways

• Kyverno makes it easy to develop policies as Kubernetes resources
• Policies can run in a CI/CD pipeline or in Kubernetes
• An extensive policy library lowers the adoption barrier
• It integrates well with GitOps tools

Acknowledgements

This piece was written by Marcelo da Silva Pires – Senior DevOps Engineer, and João Longo – Systems Architect and Innovation Leader at Encora’s Engineering Technology Practices group. Thanks to João Caleffi and André Scandaroli for reviews and insights.