Kubernetes Tips & Tricks

Removing Stuck Finalizers from namespaces

Sometimes, when deleting a resource in kubernetes, you might encounter issues where the resource gets stuck in a "Terminating" state due to finalizers. Finalizers are used to ensure that certain cleanup tasks are completed before the resource is fully deleted. However, if these tasks fail or take too long, the resource can remain stuck.

In newer versions of kubernetes, the finalizers have been moved to the metadata section of the resource. This allows you to easily remove them if needed. Unfortunately, the Namespace resource is a bit special and requires a different approach to remove the finalizers. Due to the need to provide full backwards compatibility, the Namespace resource still has the finalizers in the spec section. Thus requiring a different approach to remove them.

bash

export namespace="stuck-ns"
kubectl get namespace "$namespace" -o json | jq '.spec.finalizers=[]' | kubectl replace --raw "/api/v1/namespaces/$namespace/finalize" -f -````

powershell

$namespace = "stuck-ns"
$ns = kubectl get namespace $namespace -o json | ConvertFrom-Json
$ns.spec.finalizers = @()
$nsJson = $ns | ConvertTo-Json -Depth 10
$nsJson | kubectl replace --raw "/api/v1/namespaces/$namespace/finalize" -f -

Debug Pod

When debugging kubernetes issues, the first thing I do is to create a debug pod in the same namespace as the application I'm trying to debug. This allows me to have a clean environment with all the necessary tools to troubleshoot the issue.

apiVersion: v1
kind: Pod
metadata:
  name: netshoot
  namespace: default
spec:
  containers:
  - name: netshoot
    image: nicolaka/netshoot
    command: ["/bin/bash"]
    args: ["-c", "while true; do ping localhost; sleep 60;done"]

Privileged Debug Pod

If you need to perform more advanced node debugging, and you do not have easy access to the underlying host operating system (e.g. in a managed kubernetes environment), you can create a privileged debug pod.

In this example I'll be showcasing a privileged debug DeamonSet that has full access to the host.

WARNING

Running a privileged pod can pose significant security risks. Ensure you understand the implications and restrict access to trusted users only. Remove the privileged pod as soon as you are done debugging.

apiVersion: v1
kind: Namespace
metadata:
  name: troubleshoot
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: host-debug
  namespace: troubleshoot
labels:
  app: host-debug
spec:
  selector:
    matchLabels:
      app: host-debug
  template:
    metadata:
      labels:
        app: host-debug
    spec:
      # share host namespaces so tools see the node's view
      hostNetwork: true
      hostPID: true
      dnsPolicy: ClusterFirstWithHostNet

      tolerations:
        - operator: "Exists"          # land on tainted nodes too
      containers:
        - name: netshoot
          image: nicolaka/netshoot:latest
          imagePullPolicy: IfNotPresent
          command: ["/bin/bash", "-lc"]
          args:
            - |
              echo "Host debug pod ready. Use: kubectl -n troubleshoot exec -it <pod> -- bash";
              sleep infinity
          securityContext:
            privileged: true
            allowPrivilegeEscalation: true
            capabilities:
              add: ["NET_ADMIN","NET_RAW","SYS_ADMIN","SYS_MODULE","SYS_PTRACE","SYS_CHROOT","MKNOD"]
          volumeMounts:
            # mount host views so /proc/sys and /sys are the node's
            - name: host-proc
              mountPath: /host/proc
              readOnly: true
            - name: host-sys
              mountPath: /host/sys
              readOnly: true
            - name: host-lib-modules
              mountPath: /lib/modules
              readOnly: true
            - name: host-run
              mountPath: /host/run
            - name: host-etc
              mountPath: /host/etc
              readOnly: true
      volumes:
        - name: host-proc
          hostPath:
            path: /proc
        - name: host-sys
          hostPath:
            path: /sys
        - name: host-lib-modules
          hostPath:
            path: /lib/modules
        - name: host-run
          hostPath:
            path: /run
        - name: host-etc
          hostPath:
            path: /etc

Common Aliases

Alias	Command	Description
kga	kubectl get pods -o wide -A	List all pods in all namespaces (wide view)
kgall	kubectl get all	Get all resources in the current namespace
kdp	kubectl describe pod	Describe a pod
kdelp	kubectl delete pod	Delete a pod
kl	kubectl logs	View logs for a pod
klf	kubectl logs -f	View logs with continuous output
klfa	kubectl logs -f --all-containers	View logs of all containers in a pod
kexec	kubectl exec -it	Execute a command in a pod (for troubleshooting)
kap	kubectl apply -f	Apply a manifest file
kdel	kubectl delete -f	Delete resources from a manifest file
kc	kubectl config current-context	Get current context
kctx	kubectl config get-contexts	List all contexts
ksctx	kubectl config use-context	Switch context
knodes	kubectl get nodes -o wide	Get nodes with wide output
kpvc	kubectl get pvc -A	Get persistent volume claims in all namespaces
ksvc	kubectl get svc	Get services in the current namespace
kapd	kubectl apply -f .	Apply all YAML files in current directory

Advanced Aliases

Alias	Command	Description
knr	kubectl get pods --field-selector=status.phase!=Running	Get all pods not in "Running" state
kres	kubectl rollout restart deployment	Restart all pods in a deployment
kpf	kubectl port-forward svc/	Port forward to a specific service
kevents	kubectl get events -A --sort-by=.metadata.creationTimestamp	Get events in all namespaces (sorted by time)
kns	kubectl config view --minify --output "jsonpath={..namespace}"	Get current namespace
ktopn	kubectl top nodes	View resource usage (CPU/memory) of nodes
ktop	kubectl top pods	View resource usage (CPU/memory) of pods
klc	kubectl logs -f -c	Tail logs of a specific container in a pod

Quick Setup

I keep an update list of these aliases in a PDS function so it can be easily configured on any linux instance.

# install pds if not already installed
set_aliases

to add these aliases to your shell, simply copy and paste them into your terminal or add them to your shell configuration file (e.g., ~/.bashrc or ~/.zshrc).

alias kga='kubectl get pods -o wide -A'
alias kgall='kubectl get all'
alias kdp='kubectl describe pod'
alias kdelp='kubectl delete pod'
alias kl='kubectl logs'
alias klf='kubectl logs -f'
alias klfa='kubectl logs -f --all-containers'
alias kexec='kubectl exec -it'
alias kap='kubectl apply -f'
alias kdel='kubectl delete -f'
alias kc='kubectl config current-context'
alias kctx='kubectl config get-contexts'
alias ksctx='kubectl config use-context'
alias knodes='kubectl get nodes -o wide'
alias kpvc='kubectl get pvc -A'
alias ksvc='kubectl get svc'
alias kapd='kubectl apply -f .'
alias knr='kubectl get pods --field-selector=status.phase!=Running'
alias kres='kubectl rollout restart deployment'
alias kpf='kubectl port-forward svc/'
alias kevents='kubectl get events -A --sort-by=.metadata.creationTimestamp'
alias kns='kubectl config view --minify --output "jsonpath={..namespace}"'
alias ktopn='kubectl top nodes'
alias ktop='kubectl top pods'
alias klc='kubectl logs -f -c'