Seliga #08 - Graceful shutdown

During the lifecycle of an application, there are several reasons why pods can be terminated, whether because the user typed the "kubectl delete" command or due to a version upgrade, among others. In these cases, Kubernetes allows container execution in the pod to be terminated "gracefully" with some configurations. But first, let's understand how the delete/termination operation happens.

As soon as the user types the "kubectl delete" command, it is passed to the API server, and from there, the endpoints are removed from the endpoints object. Endpoints are important for providing any service. They are removed directly from the control plane, which triggers events for kube-proxy, the ingress controller, DNS, and other components.

With this, all these components update their references and stop serving traffic to the POD's IP address. This operation can be fast. However, it is possible that the components are busy performing other operations, which can cause some delay in updating the references.

At the same time, the pod status in etcd is changed to "Terminating", and the Kubelet is notified, initiating the processes of unmounting any container volumes (CSI), detaching the container from the network and releasing the IP address (CNI), and destroying the container for the Container Runtime (CRI).

The pod deletion process can be problematic because while in pod creation Kubernetes waits for the Kubelet update to report IP details and update endpoints, when the pod is deleted, the endpoint is removed and the Kubelet is updated simultaneously. This can be an issue because sometimes components take time to update endpoints, and if the pod is deleted before these updates are propagated, we will face downtime. Traffic will still be routed to the removed pod as there was no update to the ingress or any high-level services.

To avoid application downtime scenarios, it is important to ensure that the pod is not deleted before the endpoint is updated. One way to do this is to use the terminationGracePeriodSeconds.

When a delete command is given to the pod, it receives the SIGTERM signal, and by default, Kubernetes waits 30 seconds before forcing the process to stop. During this time, the container can process traffic, close backend connections, and terminate the process. However, if the application needs more time to terminate, the value of terminationGracePeriodSeconds can be included or changed in the pod definition. Additionally, it is possible to include a script to wait for some time and then terminate. Kubernetes exposes a 'preStop hook' in the pod before triggering SIGTERM, which can be used for this purpose.

To avoid potential application downtime, it is important to ensure that the pod is not deleted before the endpoint is updated. One solution for this is to use the terminationGracePeriodSeconds option in the pod definition.

Upon sending the SIGTERM signal to a pod, Kubernetes waits 30 seconds before stopping the process. During this period, the container can process traffic, close backend connections, and finish the process. If the application requires more time to terminate, it is possible to include or change the value of terminationGracePeriodSeconds.

Additionally, using a "preStop hook" in the pod allows inducing a script to wait for some time before terminating. In this way, it is possible to ensure that the pod is not deleted before the endpoint is updated.

apiVersion: v1

kind: Pod

metadata:

name: test-pod

spec:

containers:

- name: nginx

image: nginx

ports:

- name: nginx

containerPort: 80

lifecycle:

preStop:

exec:

command: ["sleep", "10"]

terminationGracePeriodSeconds: 45

This configuration can help the application process all requests and close connections, avoiding forced termination.

Additionally, it is possible to change the default wait time, which is 30 seconds, when manually deleting a resource using the kubectl delete command, by adding the --grace-period=SECONDS parameter.

# kubectl delete deployment test --grace-period=60

In conclusion, removing endpoints and updating references can take some time, which can result in downtime if the pod is deleted before these updates are propagated.

By customizing the termination time and adopting appropriate strategies, it is possible to minimize downtime and ensure a smoother, more controlled pod deletion process in Kubernetes. This contributes to the continuous availability of services and user satisfaction.