Kubernetes Pod Stuck in Terminating State - How to Fix | DevOps Duoo

The Problem

Kubernetes pods can become stuck in the Terminating state for a variety of reasons, including:

• Resource constraints on the node
• Network connectivity issues
• Container crashes during shutdown
• Misconfigured pod settings or finalizers

When a pod is stuck in the Terminating state, it can cause resource leaks, performance degradation, and even security vulnerabilities. In this article, we will explore the common causes of this issue and provide step-by-step instructions on how to troubleshoot and resolve it.

Identifying the Issue

To identify a pod stuck in the Terminating state, run the following command:

kubectl get pods -o wide

This command displays a list of all pods in the current namespace, including their status. Look for pods with a status of Terminating and note their names and namespaces.

Troubleshooting

To troubleshoot the issue, retrieve the pod's logs from the previous container instance:

kubectl logs <pod_name> -n <namespace> --previous

Replace and with the actual values. The --previous flag lets you view the logs of the previous container instance, which often reveals the cause of the termination failure.

You should also describe the pod to check for events and conditions:

kubectl describe pod <pod_name> -n <namespace>

Look for events at the bottom of the output — they often indicate what is preventing the pod from being deleted.

Forcing Pod Termination

If the pod is stuck and you cannot identify the root cause, you can force-delete it using the following command:

kubectl delete pod <pod_name> -n <namespace> --force --grace-period=0

This command immediately deletes the pod and releases its resources. Use this option with caution, as it bypasses the graceful shutdown process and can potentially cause data loss or leave external resources in an inconsistent state.

Removing Finalizers

A common reason pods get stuck in Terminating is the presence of finalizers that are waiting for an external controller to act. If the controller is unavailable, the finalizers will never be removed. You can manually patch them out:

kubectl patch pod <pod_name> -n <namespace> -p '{"metadata":{"finalizers":null}}'

This command removes all finalizers from the pod, allowing Kubernetes to proceed with deletion.

Configuring Graceful Shutdown

To prevent pods from getting stuck in the Terminating state, configure a terminationGracePeriodSeconds that gives your application enough time to shut down cleanly:

kubectl patch deployment <deployment_name> -n <namespace> \
  -p '{"spec":{"template":{"spec":{"terminationGracePeriodSeconds":30}}}}'

This sets the termination grace period to 30 seconds, giving the pod time to handle in-flight requests and clean up resources before Kubernetes force-kills it.

Your application should also handle the SIGTERM signal gracefully. When Kubernetes sends SIGTERM, your process should stop accepting new requests, finish processing existing ones, and then exit cleanly within the grace period.

Common Mistakes

When troubleshooting a pod stuck in the Terminating state, avoid these common mistakes:

• Not checking the pod's logs or events for error messages before resorting to force-deletion
• Not verifying the pod's configuration and settings, particularly finalizers
• Using the --force option without understanding its implications for data integrity
• Not configuring a terminationGracePeriodSeconds that matches your application's actual shutdown time

Key Takeaways

• Use kubectl describe and kubectl logs --previous to diagnose why a pod is stuck before taking action.
• Use kubectl delete --force --grace-period=0 with caution — it can cause data loss.
• Remove stuck finalizers with kubectl patch if an external controller is unavailable.
• Configure terminationGracePeriodSeconds and proper SIGTERM handling to prevent this issue in the first place.
• Verify pod configuration and review events to identify the root cause of the issue.