Security Policies in Kubernetes with Kyverno

Stop suffering with REGO, go with YAML

Creating Security policies in Kubernetes is an unceasing pursuit by the Security area to prevent incidents from exposing businesses to risks, such as data leakage. Until then, we know that the deal is to apply security policies in Kubernetes clusters to prevent any exposure to risk as much as possible, and, on the market, there are several alternatives of Security Policies for this: OPA, Gatekeeper, Syra, AWS Opa, Azure Policy etc. However, the challenge for us, Kubernauts on duty, lies in implementing and maintaining these policies in the language in which they are written, REGO. Because we are used to working with YAML, tools that use a new language, in this case REGO, only make our lives harder! It is like taking a step backward in what we are doing to learn a new language and then apply it to the security policies of the K8s cluster. Tired of suffering with this, I went looking for an alternative and found Kyverno. Kyverno is a Kubernetes controller capable of creating its own security policy with YAML. Moreover, if you change your cloud provider, your rules will already be part of your CI/CD or a velero backup or, in the worst-case scenario, in the YAMLS files of your hidden directory on your drive. In summary, it is easy to write, easy to understand, they stay in a single place and can migrate from cloud provider or even to on-premise, avoiding lock-in with cloud providers. You can achieve almost anything with Kyverno, check it out:

• Block resource deletion, especially Kyverno itself;

• Track what is running that might infringe on the security of the nodes;

• Automate and police label creation to charge $$ from teams when dealing with a multi-tenant K8s;

• Automate the creation of Network Policies and who can modify them;

• Block the use of specific resources, for example, I only want Ingress to be spun up using my ingress-nginx or my kong, and traefik is only for team X;

• Automagically know that any deployment must have a minimum of: . cpu/memory resources . labels . replicas

• Distribute secrets and configmaps to any created namespace;

• Restrict the use of LoadBalancer and/or NodePort services;

• Make it impossible for people to spin up things in the Default namespace;

• Force every resource to use tags on images and always from the same registry;

• Apply NetworkPolicy when a new namespace is created;

• Create a namespace, generate all policies and generate a report of what is applied.

Additionally, you can generate violation, inspection, and enforcement reports — all in YAML — for you to extract with that “-o json” and play around with converting to HTML. There is a CLI for those who like it too.Installing Kyverno

Want to install Kyverno, play with some initial policies, and see how it works?

The installation is non-intrusive! Nothing will stop, no error will be generated by the installation, and there is no default rule that forces any policy. That is, you can apply Kyverno to your production cluster, as it will generate at most policy reports indicating where there is a problem to be addressed, based on default rules that do not interfere, only assess. Want to install Kyverno, play with some initial policies, and see how it works? Just be mindful of the fact that the more workloads you have in your cluster, the more CPU and memory resources your Kyverno will need to analyze, generate, and maintain policies and reports. If that is your case, increase these values in the Kyverno deployment.Let's get to the point! You will need:

• * kubectl

• * helm => 3.23

• * 1 k8s cluster

• * git

* kyverno cli, https://kyverno.io/docs/kyverno-cli/ (optional, only if you really want to, to validate policies, etc)$ helm repo add kyverno https://kyverno.github.io/kyverno/

$ helm repo update

$ helm install kyverno -n kyverno kyverno/kyverno --create-namespace

Verify if the Kyverno pod is installed:

$ kubectl get pod -n kyverno
NAME                           READY   STATUS    RESTARTS   AGE
pod/kyverno-6868bc56fb-gfcqn   1/1     Running   0          114s

$ kubectl get cpol
NAME                             BACKGROUND   ACTION
disallow-add-capabilities        true         audit
disallow-host-namespaces         true         audit
disallow-host-path               true         audit
disallow-host-ports              true         audit
Disallow-privileged-containers   true         audit
disallow-selinux                 true         audit
require-default-proc-mount       true         audit
restrict-apparmor-profiles       true         audit
restrict-sysctls                 true         audit

$ kubectl get polr -AO “get pod” 
#aguarde até o pod estar “running”

From this moment on, “get cpol” will return the policies applied to the cluster, at the cluster level. The “get polr -A” will bring the reports by namespaces, if you have namespaces created and with running workloads. See that all policies created by default are in audit in the action column, meaning they only audit the workloads. From the moment you want them to be applied and block some execution or configuration, just change the rule validationFailureAction: enforce and this will prevent the next pod from continuing to “break the policy”.

$ kubectl edit cpol disallow-add-capabilities
(...)
spec:
  validationFailureAction: audit    #ou enforce para forçar a regra
(...)Primeiro exemplo:Bloqueio de workloads sem label específica:$ cat require-pod-ns-svc-ing-label-required.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-certain-labels
spec:
  validationFailureAction: audit
  rules:
  - name: validate-name-labels
    exclude:
      resources:
        namespaces:
        - kube-system
        - app1
        - workloads-x
    match:
      resources:
        kinds:
        - Pod
        - Namespace
        - Service
        - Ingress
    validate:
      message: "The label `app.kubernetes.io/name` and is required."
      pattern:
        metadata:
          labels:
            app.kubernetes.io/name: "?*"
  - name: validate-component-labels
    exclude:
      resources:
        namespaces:
        - kube-system
        - app1
        - workloads-x
    match:
      resources:
        kinds:
        - Pod
        - Namespace
        - Service
        - Ingress
    validate:
      message: "The label `app.kubernetes.io/component` is required."
      pattern:
        metadata:
          labels:
            app.kubernetes.io/component: "?*"

Apply this policy:

$ kubectl apply -f require-pod-ns-svc-ing-label-required.yaml

Verify in the Kyverno pod logs and ClusterPolicy objects:

$ kubectl get cpol
NAME                             BACKGROUND   ACTION
disallow-add-capabilities        true         audit
disallow-host-namespaces         true         audit
disallow-host-path               true         audit
disallow-host-ports              true         audit
disallow-privileged-containers   true         audit
disallow-selinux                 true         audit
require-default-proc-mount       true         audit
restrict-apparmor-profiles       true         audit
restrict-sysctls                 true         audit
require-certain-labels           true         audit

Here, we can already see that our rule requiring the informed labels to be applied to any workload in any namespace, with the exception of namespaces app1, kube-system, and workloads-x.

Create a test deployment:

$ cat web.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: web
    app.kubernetes.io/name: web
  name: web
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  strategy: {}
  template:
    metadata:
      labels:
        app: web
        app.kubernetes.io/name: web
    spec:
      containers:
      - image: nginx
        name: nginx
        resources: {}
status: {}
$ kubectl apply -f web.yaml
deployment.apps/web created

Notice here that I only created a label app.kubernetes.io/name: web, which does not entirely satisfy my newly created policy.

To view the FAIL audits of the reports, run:

$ kubectl get polr -n default -o yaml | grep “status: fail” -B14
---
scored: true
status: pass
- message: 'validation error: The label`app.kubernetes.io/component`  
  is required. Rule validate-certain-labels[0] failed at path 
  /metadata/labels/app.kubernetes.io/component/.
  Rule validate-certain-labels[1] failed at path /metadata/labels  
  /app.kubernetes.io/component/.'
policy: require-certain-labels
resources:
- apiVersion: apps/v1
  kind: Deployment
  name: web
  namespace: default
  uid: cbe8ecc1-71f8-4b2c-819d-59417039fc77
rule: validate-ccomponent-labels
scored: true
status: fail

(...) The policyReport will show the kubernetes service and the default namespace with the same missing labels we indicated in the policy. If this policy is on enforce, then adding these labels to these resources will be mandatory, and the pod will not be created until this issue is resolved.

There are dozens of other policies that can be applied in terms of a cluster, which does not prevent you from applying them only to a specific namespace, using kind: ClusterPolicy or for namespace, using Kind: Policy. They are available at https://github.com/kyverno/policies. One that I really like is the one that limits resource usage and forces every pod to obey a predefined rule of resource limits/requests:

$ cat require-and-set-requests-limits.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-requests-limits
  annotations:
    policies.kyverno.io/title: Require Limits and Requests 
    policies.kyverno.io/category: Multi-Tenancy
spec:
  validationFailureAction: audit
  rules:
  - name: validate-resources
    exclude:
      resources:
        namespaces:
        - linkerd
        - kube-system
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "CPU and memory resource requests and limits are required."
      pattern:
        spec:
          containers:
          - resources:
              requests:
                memory: "<=1000Mi"
                cpu: "<=500m"
              limits:
                memory: "<=2000Mi"
                cpu: "<=1000m"
$
$ kubectl apply -f require-and-set-requests-limits.yaml

With this, any pod, in any namespace, with the exception of Kube-system, will have to declare the utilization of resources and, still, respect the limits for requests and limits. As my brother would say: “there are a thousand ways to prepare Neston, invent yours”.

Based on the models and examples in the repository and the documentation, it is easy to create the policies we need to apply, without needing to learn a new language or dive into various K8s resources for troubleshooting or evolution.

That is it, folks! I highly recommend you run to get updated. Kyverno is wonderful and the learning curve is just a few hours.

Any questions, suggestions, or criticism, you can reach out to me: adonai@getup.io.