Kubernetes Tips & Tricks

Removing Stuck Finalizers from namespaces

Sometimes, when deleting a resource in kubernetes, you might encounter issues where the resource gets stuck in a "Terminating" state due to finalizers. Finalizers are used to ensure that certain cleanup tasks are completed before the resource is fully deleted. However, if these tasks fail or take too long, the resource can remain stuck.

In newer versions of Kubernetes, finalizers have been moved to the metadata.finalizers section of the resource. This allows you to easily remove them if needed.

Unfortunately, the Namespace resource is a bit special and requires a different approach to remove finalizers. Due to the need to provide full backwards compatibility, the built-in kubernetes finalizer for namespaces is still located in the spec.finalizers section. Thus requiring a different approach to remove them.

NOTE

You may also see finalizers in metadata.finalizers on namespaces, these are custom finalizers added by operators or controllers. The built-in kubernetes finalizer remains in spec.finalizers.

bashpowershell

bash

export namespace="stuck-ns"
kubectl get namespace "$namespace" -o json | jq '.spec.finalizers=[]' | kubectl replace --raw "/api/v1/namespaces/$namespace/finalize" -f -````

powershell

$namespace = "stuck-ns"
$ns = kubectl get namespace $namespace -o json | ConvertFrom-Json
$ns.spec.finalizers = @()
$nsJson = $ns | ConvertTo-Json -Depth 10
$nsJson | kubectl replace --raw "/api/v1/namespaces/$namespace/finalize" -f -

$namespace='crossplane-system';(kubectl get ns $namespace -o json | ConvertFrom-Json | % { $_.spec.finalizers=@(); $_ } | ConvertTo-Json -Depth 10) | kubectl replace --raw "/api/v1/namespaces/$namespace/finalize" -f -

for normal resources you can just remove it from the metadata field

bashpowershell

bash

# namespaced
resource="deploy/my-app"; ns="app"
kubectl patch "$resource" -n "$ns" --type=merge -p '{"metadata":{"finalizers":[]}}'

# cluster-scoped
resource="clusterRoleBinding"
kubectl patch "$resource" --type=merge -p '{"metadata":{"finalizers":[]}}'

powershell

# namespaced
$resource='deploy/my-app'; $namespace='app'; kubectl patch $resource -n $namespace --type=merge -p '{\"metadata\":{\"finalizers\":[]}}

# cluster-scoped
$resource='clusterRoleBinding'; kubectl patch $resource --type=merge -p '{\"metadata\":{\"finalizers\":[]}}

Client-Side vs Server-Side Apply

Kubernetes provides two different mechanisms for applying manifests: client-side apply and server-side apply. Understanding the differences between them is crucial for managing resources effectively.

Client-Side Apply (Default)

Client-side apply is the traditional method used by kubectl apply. The client (kubectl) calculates the diff between the current state and the desired state, then sends a patch to the API server.

Key Characteristics:

Uses the last-applied-configuration annotation to track changes
The annotation stores the entire previous configuration in JSON format
Can lead to large annotations on resources with extensive configurations
May cause conflicts when multiple tools manage the same resource
Default behavior when using kubectl apply -f

Example:

bash

kubectl apply -f deployment.yaml

Server-Side Apply

Server-side apply (SSA) moves the responsibility of calculating diffs to the API server. It uses field management to track ownership of fields and allows multiple managers to collaborate on the same resource.

Key Characteristics:

Introduced in Kubernetes 1.16 (GA in 1.22)
Uses field management instead of annotations
More efficient for large resources (no last-applied-configuration annotation)
Better support for multiple managers (GitOps tools, operators, manual changes)
Resolves conflicts based on field ownership
Required for certain CRDs and operators

Example:

bash

kubectl apply -f deployment.yaml --server-side

When to Use Server-Side Apply

Consider using server-side apply when:

Working with large manifests (reduces annotation overhead)
Multiple tools or controllers manage the same resource
Using GitOps tools like ArgoCD or Flux (often recommended or required)
Working with CRDs that require field-level ownership tracking
You need atomic operations across multiple fields

Field Management and Conflicts

With server-side apply, each field has a manager. If you try to modify a field owned by another manager, you'll get a conflict error. You can force the change using --force-conflicts:

bash

kubectl apply -f deployment.yaml --server-side --force-conflicts

WARNING

Using --force-conflicts will take ownership of conflicting fields from other managers. Use with caution in environments with multiple automation tools.

Viewing Field Managers

You can see which manager owns which fields:

bash

kubectl get deployment my-app -o yaml --show-managed-fields

Debug Pod

When debugging kubernetes issues, the first thing I do is to create a debug pod in the same namespace as the application I'm trying to debug. This allows me to have a clean environment with all the necessary tools to troubleshoot the issue.

yaml

apiVersion: v1
kind: Pod
metadata:
  name: netshoot
  namespace: default
spec:
  containers:
  - name: netshoot
    image: nicolaka/netshoot
    command: ["/bin/bash"]
    args: ["-c", "while true; do ping localhost; sleep 60;done"]

Privileged Debug Pod

If you need to perform more advanced node debugging, and you do not have easy access to the underlying host operating system (e.g. in a managed kubernetes environment), you can create a privileged debug pod.

In this example I'll be showcasing a privileged debug DeamonSet that has full access to the host.

WARNING

Running a privileged pod can pose significant security risks. Ensure you understand the implications and restrict access to trusted users only. Remove the privileged pod as soon as you are done debugging.

yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: host-debug
  namespace: default
  labels:
    app: host-debug
spec:
  selector:
    matchLabels:
      app: host-debug
  template:
    metadata:
      labels:
        app: host-debug
    spec:
      # share host namespaces so tools see the node's view
      hostNetwork: true
      hostPID: true
      dnsPolicy: ClusterFirstWithHostNet
      tolerations:
        - operator: "Exists"          # land on tainted nodes too
      containers:
        - name: netshoot
          image: nicolaka/netshoot:latest
          imagePullPolicy: IfNotPresent
          command: ["/bin/bash", "-lc"]
          args:
            - |
              echo "Host debug pod ready. Use: kubectl -n troubleshoot exec -it <pod> -- bash";
              sleep infinity
          securityContext:
            privileged: true
            allowPrivilegeEscalation: true
            capabilities:
              add: ["NET_ADMIN","NET_RAW","SYS_ADMIN","SYS_MODULE","SYS_PTRACE","SYS_CHROOT","MKNOD"]
          volumeMounts:
            # mount host views so /proc/sys and /sys are the node's
            - name: host-proc
              mountPath: /host/proc
              readOnly: true
            - name: host-sys
              mountPath: /host/sys
              readOnly: true
            - name: host-lib-modules
              mountPath: /lib/modules
              readOnly: true
            - name: host-run
              mountPath: /host/run
            - name: host-etc
              mountPath: /host/etc
              readOnly: true
      volumes:
        - name: host-proc
          hostPath:
            path: /proc
        - name: host-sys
          hostPath:
            path: /sys
        - name: host-lib-modules
          hostPath:
            path: /lib/modules
        - name: host-run
          hostPath:
            path: /run
        - name: host-etc
          hostPath:
            path: /etc

Common Aliases

Alias	Command	Description
`kga`	`kubectl get pods -o wide -A`	List all pods in all namespaces (wide view)
`kgall`	`kubectl get all`	Get all resources in the current namespace
`kdp`	`kubectl describe pod`	Describe a pod
`kdelp`	`kubectl delete pod`	Delete a pod
`kl`	`kubectl logs`	View logs for a pod
`klf`	`kubectl logs -f`	View logs with continuous output
`klfa`	`kubectl logs -f --all-containers`	View logs of all containers in a pod
`kexec`	`kubectl exec -it`	Execute a command in a pod (for troubleshooting)
`kap`	`kubectl apply -f`	Apply a manifest file
`kdel`	`kubectl delete -f`	Delete resources from a manifest file
`kc`	`kubectl config current-context`	Get current context
`kctx`	`kubectl config get-contexts`	List all contexts
`ksctx`	`kubectl config use-context`	Switch context
`knodes`	`kubectl get nodes -o wide`	Get nodes with wide output
`kpvc`	`kubectl get pvc -A`	Get persistent volume claims in all namespaces
`ksvc`	`kubectl get svc`	Get services in the current namespace
`kapd`	`kubectl apply -f .`	Apply all YAML files in current directory

Advanced Aliases

Alias	Command	Description
`knr`	`kubectl get pods --field-selector=status.phase!=Running`	Get all pods not in "Running" state
`kres`	`kubectl rollout restart deployment`	Restart all pods in a deployment
`kpf`	`kubectl port-forward svc/`	Port forward to a specific service
`kevents`	`kubectl get events -A --sort-by=.metadata.creationTimestamp`	Get events in all namespaces (sorted by time)
`kns`	`kubectl config view --minify --output "jsonpath={..namespace}"`	Get current namespace
`ktopn`	`kubectl top nodes`	View resource usage (CPU/memory) of nodes
`ktop`	`kubectl top pods`	View resource usage (CPU/memory) of pods
`klc`	`kubectl logs -f -c`	Tail logs of a specific container in a pod

Quick Setup

I keep an update list of these aliases in a PDS function so it can be easily configured on any linux instance.

bash

# install pds if not already installed
set_aliases

to add these aliases to your shell, simply copy and paste them into your terminal or add them to your shell configuration file (e.g., ~/.bashrc or ~/.zshrc).

bash

alias kga='kubectl get pods -o wide -A'
alias kgall='kubectl get all'
alias kdp='kubectl describe pod'
alias kdelp='kubectl delete pod'
alias kl='kubectl logs'
alias klf='kubectl logs -f'
alias klfa='kubectl logs -f --all-containers'
alias kexec='kubectl exec -it'
alias kap='kubectl apply -f'
alias kdel='kubectl delete -f'
alias kc='kubectl config current-context'
alias kctx='kubectl config get-contexts'
alias ksctx='kubectl config use-context'
alias knodes='kubectl get nodes -o wide'
alias kpvc='kubectl get pvc -A'
alias ksvc='kubectl get svc'
alias kapd='kubectl apply -f .'
alias knr='kubectl get pods --field-selector=status.phase!=Running'
alias kres='kubectl rollout restart deployment'
alias kpf='kubectl port-forward svc/'
alias kevents='kubectl get events -A --sort-by=.metadata.creationTimestamp'
alias kns='kubectl config view --minify --output "jsonpath={..namespace}"'
alias ktopn='kubectl top nodes'
alias ktop='kubectl top pods'
alias klc='kubectl logs -f -c'

github actions suite

aws

platform

aks

aro

eks

auto mode

rke2

handbook

Reference Architectures

devops

kubernetes

vault

edge

armbian

Build custom armbian images

single board compute

Kubernetes Tips & Tricks

Removing Stuck Finalizers from namespaces

Client-Side vs Server-Side Apply

Client-Side Apply (Default)

Server-Side Apply

When to Use Server-Side Apply

Field Management and Conflicts

Viewing Field Managers

Debug Pod

Privileged Debug Pod

Common Aliases

Advanced Aliases

aro

auto mode

rke2

handbook

Reference Architectures

vault

armbian

Build custom armbian images

single board compute

Kubernetes Tips & Tricks ​

Removing Stuck Finalizers from namespaces ​

Client-Side vs Server-Side Apply ​

Client-Side Apply (Default) ​

Server-Side Apply ​

When to Use Server-Side Apply ​

Field Management and Conflicts ​

Viewing Field Managers ​

Debug Pod ​

Privileged Debug Pod ​

Common Aliases ​

Advanced Aliases ​

Kubernetes Tips & Tricks

Removing Stuck Finalizers from namespaces

Client-Side vs Server-Side Apply

Client-Side Apply (Default)

Server-Side Apply

When to Use Server-Side Apply

Field Management and Conflicts

Viewing Field Managers

Debug Pod

Privileged Debug Pod

Common Aliases

Advanced Aliases