Advanced Debugging of Pods and Containers on Kubernetes

Advanced debugging of pods and containers on Kubernetes represents a critical skill for any Cloud operations engineer. According to the Cloud Native Now report, IT teams spend an average of 34 working days per year resolving Kubernetes problems. Mastering advanced diagnostic techniques drastically reduces this wasted time.

TL;DR: This guide covers kubectl debug, log and event analysis, network troubleshooting, and advanced diagnostic tools to efficiently resolve pod problems on Kubernetes.

This skill is at the heart of the LFD459 Kubernetes for Application Developers training.

Why is Advanced Pod and Container Debugging Essential?

Production Kubernetes environments present complexity that goes beyond simple log checks. With 82% of container users running Kubernetes in production according to the CNCF Annual Survey 2025, debugging scenarios multiply.

Common symptoms requiring advanced debugging:

Key takeaway: A pod in CrashLoopBackOff restarts in a loop. Check the previous container logs with --previous to capture the error message before the crash.

For more on deployment errors, see the guide Resolve common Kubernetes deployment errors.

How to Use kubectl debug Kubernetes to Diagnose a Pod?

The kubectl debug command introduced in Kubernetes 1.18 allows injecting a debug container into an existing pod or creating a modified copy. This approach avoids modifying production manifests.

Create an Ephemeral Debug Container

# Inject debug container into running pod
kubectl debug -it my-pod --image=busybox --target=my-container

# Create pod copy with debug image
kubectl debug my-pod -it --copy-to=my-pod-debug --container=debug --image=nicolaka/netshoot

The nicolaka/netshoot image contains all necessary network tools: tcpdump, netstat, curl, dig, nslookup. It weighs about 300 MB compared to 3 MB for Alpine (Medium Docker Optimization).

Debug a Pod That Won't Start

When a pod refuses to start, create a copy with modified entry point:

kubectl debug my-pod -it --copy-to=my-pod-debug --container=app --image=my-image -- /bin/sh

This technique gives you a shell in the container without running the failing application.

Key takeaway: Always use --target to share the process namespace with the target container. Without this option, you won't see the original pod's processes.

How to Analyze Kubernetes Logs and Events for Troubleshooting?

Log and event analysis is the first step in any diagnosis. Kubernetes generates events for every significant state change.

Retrieve Container Logs

# Main container logs
kubectl logs my-pod

# Specific container logs in multi-container pod
kubectl logs my-pod -c my-sidecar

# Previous container logs (after crash)
kubectl logs my-pod --previous

# Real-time log streaming
kubectl logs -f my-pod --since=5m

# Logs from all pods in a deployment
kubectl logs deployment/my-deployment --all-containers=true

Leverage Kubernetes Events

Events reveal what the scheduler, kubelet, and controllers are trying to do:

# Current namespace events, sorted by date
kubectl get events --sort-by='.lastTimestamp'

# Events for a specific pod
kubectl get events --field-selector involvedObject.name=my-pod

# Warning type events only
kubectl get events --field-selector type=Warning

For complete observability, discover best practices in the guide Observability and monitoring of applications on Kubernetes.

Interpreting Critical Event Messages

What Advanced Network Debugging Techniques for Kubernetes?

Network problems represent a complex category requiring specialized tools.

Verify DNS Connectivity

# Test DNS resolution from a pod
kubectl run -it --rm debug --image=busybox -- nslookup kubernetes.default

# Check CoreDNS service
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns

Diagnose Service Problems

# Check service endpoints
kubectl get endpoints my-service

# Test connectivity to a service
kubectl run -it --rm debug --image=nicolaka/netshoot -- curl my-service:8080

# Capture network traffic
kubectl debug node/my-node -it --image=nicolaka/netshoot -- tcpdump -i any port 8080

Key takeaway: A Service without Endpoints indicates the selector matches no pod. Check labels with kubectl get pods --show-labels.

For advanced network configurations, see the hub Kubernetes Cluster Administration.

How to Debug Resource and Scheduling Problems?

Resource constraints frequently cause unexpected behaviors. A pod in Pending without visible events often indicates a cluster-level resource problem.

Analyze Resource Consumption

# Pod CPU/memory consumption
kubectl top pods

# Consumption by node
kubectl top nodes

# Detail of allocated resources on a node
kubectl describe node my-node | grep -A 10 "Allocated resources"

Identify QoS and Eviction Problems

Kubernetes defines three Quality of Service classes:

# Check pod's QoS class
kubectl get pod my-pod -o jsonpath='{.status.qosClass}'

For configuration best practices, refer to the guide ConfigMaps and Secrets Kubernetes: best practices.

What Advanced Tools for Kubernetes Production Debugging?

Beyond kubectl, several tools facilitate diagnosis in complex environments.

k9s: Interactive Terminal Interface

# Installation
brew install k9s  # macOS
snap install k9s  # Linux

# Launch
k9s

k9s offers fast navigation between resources, real-time logs, and contextual actions accessible via keyboard shortcuts.

stern: Multi-Pod Log Aggregation

# Logs from all pods matching a pattern
stern "my-app-.*" --since 15m

# Logs with coloring by pod
stern deployment/my-deployment --output raw

kubectx and kubens: Fast Navigation

# Change context
kubectx production

# Change namespace
kubens monitoring

These tools integrate into a CI/CD Pipeline for Kubernetes applications workflow.

How to Debug Shell-less Containers?

Distroless and scratch images contain no shell, complicating debugging. Two approaches exist.

Use kubectl debug with Namespace Sharing

kubectl debug -it my-pod-distroless --image=busybox --target=app --share-processes

The --share-processes flag allows seeing and interacting with target container processes from the debug container.

Copy Necessary Binaries

# From debug container, copy strace into target container namespace
kubectl debug -it my-pod --image=nicolaka/netshoot -- sh -c "cp /usr/bin/strace /proc/1/root/tmp/"

Quick Debugging Checklist

Step 1: Pod State

kubectl get pod my-pod -o wide
kubectl describe pod my-pod

Step 2: Logs and Events

kubectl logs my-pod --previous
kubectl get events --field-selector involvedObject.name=my-pod

Step 3: Resources

kubectl top pod my-pod
kubectl describe node $(kubectl get pod my-pod -o jsonpath='{.spec.nodeName}')

Step 4: Network

kubectl get svc,endpoints
kubectl run debug --rm -it --image=busybox -- wget -qO- my-service:8080

Key takeaway: Document each debugging session. A shared runbook reduces resolution time for recurring incidents.

The hub Kubernetes Application Development centralizes all resources for developers.

Take Action: Get Kubernetes Debugging Training

Advanced debugging requires supervised practice on realistic environments. Official Linux Foundation trainings offer this structured learning.

Recommended trainings:

• LFD459 Kubernetes for Application Developers: 3 days of intensive practice covering application troubleshooting, CKAD preparation
• LFS458 Kubernetes Administration: 4 days to master administration and cluster-level debugging, CKA preparation
• Kubernetes Fundamentals: 1 day to discover basics before going deeper

Mastering Kubernetes debugging positions you as an indispensable expert in your organization.