☸️ kubectl Day-to-Day

kubectl Day-to-Day: reading, debugging, changing and staying safe
kubectl get pods -A; kubectl describe pod NAME; kubectl logs NAME

The Kubernetes sheet on this site lists the objects and the basic commands. This guide is about daily practice with kubectl: knowing which cluster and namespace you are talking to, reading state fast, shaping output with jsonpath and custom columns, the debugging path from "Pending" to the real cause, changing things safely (apply, diff, rollouts, patches), reading RBAC and admission errors, and the habits that prevent the classic accidents.
Every example below was run for real against a throw-away local cluster (kind, Kubernetes v1.37, kubectl v1.37) in an empty namespace called demo, including the failures: the error messages are the ones the cluster produced. Because pod names, ages and IP addresses change on every run, those parts are replaced by fixed stand-ins (for example web-6d9f8c7b5d-x7k2p and "3m") so the output stays readable; everything else is untouched. Commands that need a metrics server, a cloud, or an interactive terminal are shown without output.

kubectl talks to the cluster named by the current CONTEXT in your kubeconfig (a context = cluster + user + default namespace). Checking it before every change is the single most useful habit.
$ kubectl config current-context
kind-hb
$ kubectl config get-contexts
CURRENT   NAME      CLUSTER   AUTHINFO   NAMESPACE
*         kind-hb   kind-hb   kind-hb    demo
$ kubectl config view --minify | grep -E 'cluster:|namespace:|user:|server:'
- cluster:
    server: https://127.0.0.1:6443
    cluster: kind-hb
    namespace: demo
    user: kind-hb
  user:
$ kubectl cluster-info | head -2
Kubernetes control plane is running at https://127.0.0.1:6443
CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
$ kubectl auth whoami
ATTRIBUTE                                           VALUE
Username                                            kubernetes-admin
Groups                                              [kubeadm:cluster-admins system:authenticated]
Extra: authentication.kubernetes.io/credential-id   [X509SHA256=<container-id>]
$ kubectl version | head -3
Client Version: v1.37.1
Kustomize Version: v5.8.1
Server Version: v1.37.0
$ kubectl config use-context prod-aks                       # switch cluster
$ kubectl config set-context --current --namespace=payments # switch the default namespace of THIS context
$ kubectl config rename-context aks-prod-westeurope prod    # short, clear names help you avoid mistakes
$ kubectl get pods --context staging -n payments            # one-off: override context and namespace for a single command
$ KUBECONFIG=~/.kube/config:~/.kube/aks.yaml kubectl config view --flatten > ~/.kube/merged   # merge several kubeconfig files
$ az aks get-credentials -g my-rg -n my-aks --overwrite-existing    # Azure AKS: fetch a kubeconfig entry (then kubelogin for Entra ID sign-in)
The namespace is the second half of "where am I": a command with no -n uses the context's default namespace (default if unset). Most "I deleted the wrong thing" stories are a wrong context or the default namespace. Show them in your prompt (kube-ps1, starship) and consider a different terminal colour for production.
A few useful settings: alias k=kubectl, source <(kubectl completion bash) (or zsh) for tab completion of resource names, and kubens/kubectx (or k9s) to switch quickly.

Everything below uses a small workload: a 3-replica web Deployment and a Service in the demo namespace. Creating things imperatively is fine for experiments (real systems should use manifests in Git).
$ kubectl create deployment web --image=nginx:1.27-alpine --replicas=3
deployment.apps/web created
$ kubectl expose deployment web --port=80
service/web exposed
$ kubectl create configmap app-config --from-literal=LOG_LEVEL=info --from-literal=REGION=eu-west
configmap/app-config created
$ kubectl get deployment,service,pods
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/web   3/3     3            3           3m

NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/web   ClusterIP   10.96.121.15   <none>        80/TCP    3m

NAME                      READY   STATUS    RESTARTS   AGE
pod/web-6d9f8c7b5-x7k2p   1/1     Running   0          3m
pod/web-6d9f8c7b5-q4m9z   1/1     Running   0          3m
pod/web-6d9f8c7b5-b8t3r   1/1     Running   0          3m
Generate a manifest instead of typing YAML (no cluster needed for --dry-run=client): the fastest way to start a new file.
$ kubectl create deployment api --image=nginx:1.27-alpine --replicas=2 --dry-run=client -o yaml | grep -vE 'creationTimestamp|status:|strategy:|resources:|^---$|^\s*$'
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: api
  name: api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - image: nginx:1.27-alpine
        name: nginx
$ kubectl run tmp --image=busybox:1.37 --dry-run=client -o yaml -- sleep 3600 | sed -n '1,12p'
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: tmp
  name: tmp
spec:
  containers:
  - args:
    - sleep
    - "3600"
    image: busybox:1.37

$ kubectl get pods
NAME                  READY   STATUS    RESTARTS   AGE
web-6d9f8c7b5-x7k2p   1/1     Running   0          3m
web-6d9f8c7b5-q4m9z   1/1     Running   0          3m
web-6d9f8c7b5-b8t3r   1/1     Running   0          3m
$ kubectl get pods -o wide | cut -c1-120
NAME                  READY   STATUS    RESTARTS   AGE   IP             NODE               NOMINATED NODE   READINESS GA
web-6d9f8c7b5-x7k2p   1/1     Running   0          3m    10.244.0.5   hb-control-plane   <none>           <none>
web-6d9f8c7b5-q4m9z   1/1     Running   0          3m    10.244.0.6   hb-control-plane   <none>           <none>
web-6d9f8c7b5-b8t3r   1/1     Running   0          3m    10.244.0.7   hb-control-plane   <none>           <none>
$ kubectl get pods --show-labels
NAME                  READY   STATUS    RESTARTS   AGE   LABELS
web-6d9f8c7b5-x7k2p   1/1     Running   0          3m    app=web,pod-template-hash=6d9f8c7b5
web-6d9f8c7b5-q4m9z   1/1     Running   0          3m    app=web,pod-template-hash=6d9f8c7b5
web-6d9f8c7b5-b8t3r   1/1     Running   0          3m    app=web,pod-template-hash=6d9f8c7b5
$ kubectl get pods -l app=web
NAME                  READY   STATUS    RESTARTS   AGE
web-6d9f8c7b5-x7k2p   1/1     Running   0          3m
web-6d9f8c7b5-q4m9z   1/1     Running   0          3m
web-6d9f8c7b5-b8t3r   1/1     Running   0          3m
$ kubectl get pods -l 'app in (web,api)'
NAME                  READY   STATUS    RESTARTS   AGE
web-6d9f8c7b5-x7k2p   1/1     Running   0          3m
web-6d9f8c7b5-q4m9z   1/1     Running   0          3m
web-6d9f8c7b5-b8t3r   1/1     Running   0          3m
$ kubectl get pods --field-selector status.phase=Running
NAME                  READY   STATUS    RESTARTS   AGE
web-6d9f8c7b5-x7k2p   1/1     Running   0          3m
web-6d9f8c7b5-q4m9z   1/1     Running   0          3m
web-6d9f8c7b5-b8t3r   1/1     Running   0          3m
$ kubectl get pods -o name
pod/web-6d9f8c7b5-x7k2p
pod/web-6d9f8c7b5-q4m9z
pod/web-6d9f8c7b5-b8t3r
$ kubectl get all
NAME                      READY   STATUS    RESTARTS   AGE
pod/web-6d9f8c7b5-x7k2p   1/1     Running   0          3m
pod/web-6d9f8c7b5-q4m9z   1/1     Running   0          3m
pod/web-6d9f8c7b5-b8t3r   1/1     Running   0          3m

NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/web   ClusterIP   10.96.121.15   <none>        80/TCP    3m

NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/web   3/3     3            3           3m

NAME                            DESIRED   CURRENT   READY   AGE
replicaset.apps/web-6d9f8c7b5   3         3         3       3m
$ kubectl get deploy,svc,cm,pods -l app=web
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/web   3/3     3            3           3m

NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/web   ClusterIP   10.96.121.15   <none>        80/TCP    3m

NAME                      READY   STATUS    RESTARTS   AGE
pod/web-6d9f8c7b5-x7k2p   1/1     Running   0          3m
pod/web-6d9f8c7b5-q4m9z   1/1     Running   0          3m
pod/web-6d9f8c7b5-b8t3r   1/1     Running   0          3m
$ kubectl get pods -A | head -5
NAMESPACE            NAME                                       READY   STATUS    RESTARTS   AGE
demo                 web-6d9f8c7b5-x7k2p                        1/1     Running   0          3m
demo                 web-6d9f8c7b5-q4m9z                        1/1     Running   0          3m
demo                 web-6d9f8c7b5-b8t3r                        1/1     Running   0          3m
kube-system          coredns-7c6cdd646b-n2v6w                   1/1     Running   0          3m
$ kubectl get pods -w                                       # watch changes live (Ctrl+C to stop)
$ kubectl get pods --sort-by=.metadata.creationTimestamp     # oldest first, newest last
-l selects by label (app=web, env!=dev, 'tier in (a,b)'); --field-selector filters by fields the API knows (status.phase, spec.nodeName, metadata.name); -A means all namespaces. "all" is a convenience alias that does NOT include everything (ConfigMaps and Secrets are not in it, which is why it is listed again above).
Short names save typing: po pods, deploy deployments, svc services, ns namespaces, cm configmaps, sa serviceaccounts, pvc persistentvolumeclaims, ing ingresses, rs replicasets, ds daemonsets, sts statefulsets, no nodes. The full list, with short names and whether each is namespaced:
$ kubectl api-resources | sed -n '1p;/^pods /p;/^services /p;/^deployments /p;/^configmaps /p;/^namespaces /p'
NAME                                SHORTNAMES   APIVERSION                        NAMESPACED   KIND
configmaps                          cm           v1                                true         ConfigMap
namespaces                          ns           v1                                false        Namespace
pods                                po           v1                                true         Pod
services                            svc          v1                                true         Service
deployments                         deploy       apps/v1                           true         Deployment
$ kubectl api-resources --namespaced=false -o name | head -5
componentstatuses
namespaces
nodes
persistentvolumes
mutatingadmissionpolicies.admissionregistration.k8s.io

describe is the first thing to run on a pod that is not Running: the Events section at the bottom is the story.
$ kubectl describe deployment web | sed -n '1,14p'
Name:                   web
Namespace:              demo
CreationTimestamp:      Sun, 08 Mar 2026 10:00:00 +0000
Labels:                 app=web
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=web
Replicas:               3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=web
  Containers:
   nginx:
$ kubectl describe pod -l app=web | sed -n '/^Events:/,$p' | head -8
$ kubectl events --for deployment/web
LAST SEEN   TYPE     REASON              OBJECT           MESSAGE
3m          Normal   ScalingReplicaSet   Deployment/web   Scaled up replica set web-6d9f8c7b5 from 0 to 3
$ kubectl get events --sort-by=.lastTimestamp | tail -4
2s          Normal   SuccessfulCreate    replicaset/web-6d9f8c7b5   Created pod: web-6d9f8c7b5-q4m9z
2s          Normal   SuccessfulCreate    replicaset/web-6d9f8c7b5   Created pod: web-6d9f8c7b5-x7k2p
2s          Normal   SuccessfulCreate    replicaset/web-6d9f8c7b5   Created pod: web-6d9f8c7b5-b8t3r
2s          Normal   ScalingReplicaSet   deployment/web             Scaled up replica set web-6d9f8c7b5 from 0 to 3
Events expire after about an hour, so look soon after a problem. The newer kubectl events command is sorted for you and can filter (--for, --types=Warning, --watch).
$ kubectl events -A --types=Warning                         # every warning in the cluster

Default tables hide most of the information. -o yaml and -o json show everything; the rest of the skill is cutting that down to the one thing you want.
$ kubectl get pods -o yaml | grep -E '^    (name|nodeName|phase):|^  (nodeName|phase):' | head -4
    name: web-6d9f8c7b5-x7k2p
    nodeName: hb-control-plane
    phase: Running
    name: web-6d9f8c7b5-q4m9z
$ kubectl get pods -o jsonpath='{.items[*].metadata.name}'; echo
web-6d9f8c7b5-x7k2p web-6d9f8c7b5-q4m9z web-6d9f8c7b5-b8t3r
$ kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\t"}{.spec.nodeName}{"\n"}{end}'
web-6d9f8c7b5-x7k2p	Running	hb-control-plane
web-6d9f8c7b5-q4m9z	Running	hb-control-plane
web-6d9f8c7b5-b8t3r	Running	hb-control-plane
$ kubectl get pods -o custom-columns='POD:.metadata.name,IMAGE:.spec.containers[*].image,NODE:.spec.nodeName,IP:.status.podIP'
POD                   IMAGE               NODE               IP
web-6d9f8c7b5-x7k2p   nginx:1.27-alpine   hb-control-plane   10.244.0.5
web-6d9f8c7b5-q4m9z   nginx:1.27-alpine   hb-control-plane   10.244.0.6
web-6d9f8c7b5-b8t3r   nginx:1.27-alpine   hb-control-plane   10.244.0.7
$ kubectl get pods -o custom-columns='NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount' --no-headers
web-6d9f8c7b5-x7k2p   0
web-6d9f8c7b5-q4m9z   0
web-6d9f8c7b5-b8t3r   0
$ kubectl get deployment web -o jsonpath='{.spec.template.spec.containers[0].image}'; echo
nginx:1.27-alpine
$ kubectl get cm app-config -o jsonpath='{.data}'; echo
{"LOG_LEVEL":"info","REGION":"eu-west"}
$ kubectl get pods -o json | jq -r '.items[] | select(.status.phase=="Running") | .metadata.name'
web-6d9f8c7b5-x7k2p
web-6d9f8c7b5-q4m9z
web-6d9f8c7b5-b8t3r
$ kubectl get pods -o json | jq -r '[.items[].spec.containers[].image] | unique[]'
nginx:1.27-alpine
$ kubectl get pods -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | head -2
web-6d9f8c7b5-x7k2p
web-6d9f8c7b5-q4m9z
$ kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.kubeletVersion}'; echo
v1.37.0
jsonpath uses {.path} with [*] for "every item" and {range}...{end} for loops; custom-columns builds a table from paths; jq is the most flexible when it is installed. In Bash, put the whole expression in SINGLE quotes. List every image running in the cluster, a common audit:
$ kubectl get pods -A -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' | sort | uniq -c | sort -rn | head -3
      3 nginx:1.27-alpine
      2 registry.k8s.io/coredns/coredns:v1.14.6
      1 registry.k8s.io/kube-scheduler:v1.37.0

$ kubectl logs deployment/web --tail=3
Found 3 pods, using pod/web-6d9f8c7b5-x7k2p
2026/10/02 00:24:25 [notice] 1#1: start worker process 42
2026/10/02 00:24:25 [notice] 1#1: start worker process 43
2026/10/02 00:24:25 [notice] 1#1: start worker process 44
$ kubectl logs -l app=web --tail=1 --prefix --max-log-requests=5
[pod/web-6d9f8c7b5-x7k2p/nginx] 2026/10/02 00:24:25 [notice] 1#1: start worker process 44
[pod/web-6d9f8c7b5-q4m9z/nginx] 2026/10/02 00:24:25 [notice] 1#1: start worker process 44
[pod/web-6d9f8c7b5-b8t3r/nginx] 2026/10/02 00:24:25 [notice] 1#1: start worker process 44
$ kubectl logs deployment/web --since=1h --timestamps | head -2
Found 3 pods, using pod/web-6d9f8c7b5-x7k2p
2026-10-02T00:24:25.791927344Z /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
2026-10-02T00:24:25.791965873Z /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
$ kubectl exec deployment/web -- nginx -v
nginx version: nginx/1.27.5
$ kubectl exec deployment/web -- sh -c 'echo "$HOSTNAME" | cut -c1-8; wget -qO- http://127.0.0.1/ | head -3'
web-7c6c
<!DOCTYPE html>
<html>
<head>
$ kubectl exec deployment/web -- cat /etc/os-release | head -2
NAME="Alpine Linux"
ID=alpine
$ kubectl logs -f deployment/web                            # follow (Ctrl+C to stop);  -c NAME picks a container;  --previous shows the crashed one
$ kubectl exec -it deployment/web -- sh                      # an interactive shell
$ kubectl port-forward svc/web 8080:80                       # then browse http://localhost:8080 (Ctrl+C to stop; the forward ends with the pod)
$ kubectl cp web-xxxx:/etc/nginx/nginx.conf ./nginx.conf     # copy a file out (needs tar in the image)
Logs of a deployment show ONE pod (kubectl picks it); -l app=web --prefix reads all pods with the pod name in front. logs --previous reads the container that crashed BEFORE the current one (use it when the pod has already restarted and the current container has no useful output yet), which is how you read a CrashLoopBackOff.
Talk to a Service from inside the cluster with a temporary pod (it deletes itself; the short sleep keeps the pod alive long enough for kubectl to attach, a quirk of very fast commands):
$ kubectl run tmp --rm -i --restart=Never --image=busybox:1.37 -- sh -c 'wget -qO- http://web | head -4; sleep 2'
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
pod "tmp" deleted from demo namespace
$ kubectl run dns --rm -i --restart=Never --image=busybox:1.37 -- sh -c 'nslookup web | grep -E "Name|Address" | tail -2; sleep 2'
Name:	web.demo.svc.cluster.local
Address: 10.96.121.15
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
pod "dns" deleted from demo namespace

Create five broken pods on purpose and read how Kubernetes reports each.
$ kubectl run badimage --image=nginx:no-such-tag
pod/badimage created
$ kubectl run crasher --image=busybox:1.37 -- sh -c 'echo "starting"; echo "fatal: cannot connect to database" >&2; exit 1'
pod/crasher created
$ kubectl run toobig --image=nginx:1.27-alpine --overrides='{"spec":{"containers":[{"name":"toobig","image":"nginx:1.27-alpine","resources":{"requests":{"cpu":"64","memory":"1Ti"}}}]}}'
pod/toobig created
$ kubectl create -f - <<'EOF'
> apiVersion: v1
> kind: Pod
> metadata: {name: nocfg}
> spec:
>   containers:
>   - name: app
>     image: busybox:1.37
>     command: ["sleep", "3600"]
>     envFrom:
>     - configMapRef: {name: does-not-exist}
> EOF
pod/nocfg created
$ kubectl create -f - <<'EOF'
> apiVersion: v1
> kind: Pod
> metadata: {name: hungry}
> spec:
>   containers:
>   - name: app
>     image: busybox:1.37
>     command: ["sh", "-c", "x=$(head -c 200000000 /dev/zero | tr '\\0' 'x'); sleep 3600"]
>     resources:
>       limits: {memory: 32Mi}
> EOF
pod/hungry created
$ kubectl get pods
NAME                  READY   STATUS                       RESTARTS       AGE
badimage              0/1     ImagePullBackOff             0              3m
crasher               0/1     CrashLoopBackOff             4 (45s ago)    3m
hungry                0/1     OOMKilled                    4 (45s ago)   3m
nocfg                 0/1     CreateContainerConfigError   0              3m
toobig                0/1     Pending                      0              3m
web-6d9f8c7b5-x7k2p   1/1     Running                      0              3m
web-6d9f8c7b5-q4m9z   1/1     Running                      0              3m
web-6d9f8c7b5-b8t3r   1/1     Running                      0              3m
Read the STATUS column like this:
  ErrImagePull / ImagePullBackOff   the image cannot be pulled: wrong name or tag, no such registry, or missing credentials (imagePullSecrets). Look at: describe pod
  Error / CrashLoopBackOff          the container starts and exits, repeatedly, with growing delays: the status alternates between Error (it has just failed) and CrashLoopBackOff (Kubernetes is waiting before the next restart); both mean the same thing. Look at: logs --previous, and the exit code in describe
  Pending                           no node could take the pod (resources, taints, affinity) or a volume cannot bind. Look at: describe pod, the Events message
  CreateContainerConfigError        the pod spec references something missing: a ConfigMap, Secret or key. Look at: describe pod
  OOMKilled (shown in describe)     the container used more memory than its limit
  Running with READY 0/1            the readiness probe is failing, or the app is not listening: the pod gets no traffic
  Terminating (stuck)               finalizers, or a node that vanished: see "Stuck deletes"
Now find the cause of each one with describe and logs (the Events and the last state tell you directly):
$ kubectl describe pod badimage | sed -n '/^Events:/,$p' | tail -3
  Warning  Failed     72s (x4 over 2m33s)  kubelet            spec.containers{badimage}: Error: ErrImagePull
  Normal   BackOff    12s (x9 over 2m33s)  kubelet            spec.containers{badimage}: Back-off pulling image "nginx:no-such-tag"
  Warning  Failed     12s (x9 over 2m33s)  kubelet            spec.containers{badimage}: Error: ImagePullBackOff
$ kubectl logs crasher
starting
fatal: cannot connect to database
$ kubectl get pod crasher -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'; echo
{"containerID":"containerd://<container-id>","exitCode":1,"finishedAt":"2026-03-08T10:00:00Z","reason":"Error","startedAt":"2026-03-08T10:00:00Z"}
$ kubectl describe pod toobig | sed -n '/^Events:/,$p' | tail -2
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  2m35s  default-scheduler  0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
$ kubectl describe pod nocfg | sed -n '/^Events:/,$p' | tail -2
  Normal   Pulled     3s (x13 over 2m34s)  kubelet            spec.containers{app}: Container image "busybox:1.37" already present on machine and can be accessed by the pod
  Warning  Failed     3s (x13 over 2m34s)  kubelet            spec.containers{app}: Error: configmap "does-not-exist" not found
$ kubectl get pod hungry -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason} exit={.status.containerStatuses[0].lastState.terminated.exitCode}'; echo
OOMKilled exit=137
Exit codes in lastState: 0 clean exit, 1 the application failed, 137 killed (128+9: OOM or force-killed), 139 crashed (segfault), 143 stopped (128+15, normal for SIGTERM). The reason field says OOMKilled when the memory limit was the cause. A one-line summary of every pod that is not healthy (a handy triage command):
$ kubectl get pods -o json | jq -r '.items[] | select(.status.phase != "Running" or ([.status.containerStatuses[]?.ready] | all | not)) | [.metadata.name, .status.phase, ((.status.containerStatuses[0].state // {}) | keys[0] // "unscheduled"), ((.status.containerStatuses[0].state // {}) | to_entries[0].value.reason // "")] | @tsv' | column -t
badimage  Pending  waiting      ImagePullBackOff
crasher   Running  waiting      CrashLoopBackOff
hungry    Running  terminated   OOMKilled
nocfg     Pending  waiting      CreateContainerConfigError
toobig    Pending  unscheduled  
Clean up the experiments:
$ kubectl delete pod badimage crasher toobig nocfg hungry --wait=false
pod "badimage" deleted from demo namespace
pod "crasher" deleted from demo namespace
pod "toobig" deleted from demo namespace
pod "nocfg" deleted from demo namespace
pod "hungry" deleted from demo namespace

The Service exists, the pods run, and nothing answers: nearly always the selector does not match the pod labels, so the Service has no endpoints.
$ kubectl create -f - <<'EOF'
> apiVersion: v1
> kind: Service
> metadata: {name: web-typo}
> spec:
>   selector: {app: wbe}
>   ports: [{port: 80}]
> EOF
service/web-typo created
$ kubectl get endpointslices -l 'kubernetes.io/service-name in (web,web-typo)' -o custom-columns='SERVICE:.metadata.labels.kubernetes\.io/service-name,ADDRESSES:.endpoints[*].addresses[0]'
SERVICE    ADDRESSES
web        10.244.0.5,10.244.0.6,10.244.0.7
web-typo   <none>
$ kubectl describe svc web-typo | grep -E 'Selector|Endpoints'
Selector:                 app=wbe
Endpoints:                
$ kubectl get pods -l app=wbe
No resources found in demo namespace.
$ kubectl get svc web -o jsonpath='{.spec.selector}'; echo
{"app":"web"}
An EndpointSlice with no addresses (shown as <unset>) means no ready pod matches the Service's selector. (The older kubectl get endpoints still works but prints a deprecation warning on current versions: EndpointSlices replaced it.) Fix the selector (or the pod labels), or check that the pods are READY: a failing readiness probe removes a pod from the endpoints on purpose. Compare the Service's port and targetPort with the port the container really listens on.
$ kubectl delete svc web-typo
service "web-typo" deleted from demo namespace

Prefer declarative changes: keep manifests in Git and apply them. First a manifest, then look at what would change BEFORE you change it.
$ cat > manifests/web.yaml <<'EOF'
> apiVersion: apps/v1
> kind: Deployment
> metadata: {name: web, labels: {app: web}}
> spec:
>   replicas: 3
>   selector: {matchLabels: {app: web}}
>   template:
>     metadata: {labels: {app: web}}
>     spec:
>       containers:
>       - name: nginx
>         image: nginx:1.27-alpine
>         ports: [{containerPort: 80}]
>         resources:
>           requests: {cpu: 50m, memory: 32Mi}
>           limits: {memory: 64Mi}
> EOF
$ kubectl diff -f manifests/web.yaml | grep -E '^[+-] ' | head -8
-  generation: 1
+  generation: 2
-        resources: {}
+        ports:
+        - containerPort: 80
+          protocol: TCP
+        resources:
+          limits:
$ kubectl apply -f manifests/web.yaml
Warning: resource deployments/web is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
deployment.apps/web configured
$ sed -i 's/replicas: 3/replicas: 2/' manifests/web.yaml
$ kubectl diff -f manifests/web.yaml | grep -E '^[+-] '
-  generation: 2
+  generation: 3
-  replicas: 3
+  replicas: 2
$ kubectl apply -f manifests/web.yaml
deployment.apps/web configured
$ kubectl apply -f manifests/web.yaml
deployment.apps/web unchanged
$ kubectl apply -f manifests/web.yaml --dry-run=server
deployment.apps/web unchanged (server dry run)
kubectl diff prints what would change and exits with 1 if there are differences (0 if none): good for CI. (The warning on the first apply appears because the Deployment was created with kubectl create, so it has no "last applied" record yet; apply adds one, and later applies are quiet.) The second apply shows "configured"; applying an unchanged file says "unchanged". --dry-run=server sends the request to the API server, which validates it (including admission policies) without storing it, which is stronger than --dry-run=client. apply -k DIR applies a Kustomize folder; apply --server-side lets the server track field ownership (better when several tools edit the same object).
Imperative edits, for quick experiments and emergencies:
$ kubectl scale deployment web --replicas=4
deployment.apps/web scaled
$ kubectl set image deployment/web nginx=nginx:1.27-alpine
$ kubectl set env deployment/web LOG_LEVEL=debug
deployment.apps/web env updated
$ kubectl label pod -l app=web tier=frontend --overwrite | head -2
pod/web-5b8d4f7c9a-c5j7h labeled
pod/web-5b8d4f7c9a-d9l1f labeled
$ kubectl annotate deployment web owner=platform-team --overwrite
deployment.apps/web annotated
$ kubectl patch deployment web -p '{"spec":{"replicas":2}}'
deployment.apps/web patched
$ kubectl patch deployment web --type=json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/image","value":"nginx:1.27-alpine"}]'
deployment.apps/web patched (no change)
$ kubectl get deployment web -o jsonpath='{.spec.replicas} {.metadata.annotations.owner}'; echo
2 platform-team
$ kubectl edit deployment web                               # opens the live object in $EDITOR; the change is applied when you save (and is NOT in Git)
$ kubectl replace --force -f pod.yaml                        # delete and recreate: for immutable fields
Imperative changes drift away from your manifests: the next apply may undo them (or not, depending on field ownership). For anything that stays, change the file.

Roll out a bad image on purpose and watch it stall, then undo:
$ kubectl set image deployment/web nginx=nginx:does-not-exist
deployment.apps/web image updated
$ kubectl rollout status deployment/web --timeout=10s
Waiting for deployment "web" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "web" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "web" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "web" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "web" rollout to finish: 1 out of 3 new replicas have been updated...
error: timed out waiting for the condition
# (exit status 1)
$ kubectl get pods -l app=web
NAME                   READY   STATUS         RESTARTS   AGE
web-68f5c9b7d4-g3s8k   0/1     ErrImagePull   0          3m
web-79d6b8c5f2-m6w2t   1/1     Running        0          3m
web-79d6b8c5f2-p4r7v   1/1     Running        0          3m
web-79d6b8c5f2-z8h5n   1/1     Running        0          3m
$ kubectl rollout history deployment/web
deployment.apps/web 
REVISION  CHANGE-CAUSE
1         <none>
2         <none>
3         <none>
4         <none>
$ kubectl rollout undo deployment/web
Warning: resource deployments/web was previously managed with 'kubectl apply'. Rolling back will not update the kubectl.kubernetes.io/last-applied-configuration annotation, which may cause unexpected behavior on future 'kubectl apply' operations. Consider using 'kubectl apply' with your previous configuration file instead.
deployment.apps/web rolled back
$ kubectl rollout status deployment/web --timeout=90s
deployment "web" successfully rolled out
$ kubectl get pods -l app=web
NAME                   READY   STATUS        RESTARTS   AGE
web-68f5c9b7d4-g3s8k   0/1     Terminating   0          3m
web-79d6b8c5f2-m6w2t   1/1     Running       0          3m
web-79d6b8c5f2-p4r7v   1/1     Running       0          3m
web-79d6b8c5f2-z8h5n   1/1     Running       0          3m
The rolling update never removed the working pods while the new ones could not start (maxUnavailable protects you), so the site stayed up. rollout undo returns to the previous ReplicaSet. Other rollout commands:
$ kubectl set image deployment/web nginx=nginx:1.27-alpine --record      # the old --record flag is deprecated: use the change-cause annotation below instead
$ kubectl rollout restart deployment/web                     # roll all pods (same spec): picks up a changed ConfigMap or Secret
$ kubectl rollout pause deployment/web; kubectl rollout resume deployment/web      # batch several changes into one rollout
$ kubectl rollout history deployment/web --revision=2        # what was in revision 2
$ kubectl annotate deployment/web kubernetes.io/change-cause="bump nginx to 1.27" --overwrite     # a readable CHANGE-CAUSE column in the history
$ kubectl wait --for=condition=Available deployment/web --timeout=60s
deployment.apps/web condition met
kubectl wait is the right way to block a script until something is true (--for=condition=Ready pod/NAME, --for=delete, --for=jsonpath='{.status.phase}'=Running).

$ kubectl auth can-i create deployments
yes
$ kubectl auth can-i delete nodes
Warning: resource 'nodes' is not namespace scoped

yes
$ kubectl auth can-i '*' '*'
yes
$ kubectl auth can-i get secrets --as=system:serviceaccount:demo:default
no
# (exit status 1)
$ kubectl auth can-i list pods --as=system:serviceaccount:demo:default
no
# (exit status 1)
$ kubectl get pods --as=system:serviceaccount:demo:default
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:demo:default" cannot list resource "pods" in API group "" in the namespace "demo"
# (exit status 1)
auth can-i answers "may this identity do that?"; --as impersonates another user or ServiceAccount (it needs permission to impersonate): the way to test an RBAC rule from the outside. Give a ServiceAccount a narrow role and verify it:
$ kubectl create serviceaccount viewer
serviceaccount/viewer created
$ kubectl create role pod-reader --verb=get,list,watch --resource=pods
role.rbac.authorization.k8s.io/pod-reader created
$ kubectl create rolebinding viewer-pods --role=pod-reader --serviceaccount=demo:viewer
rolebinding.rbac.authorization.k8s.io/viewer-pods created
$ kubectl auth can-i list pods --as=system:serviceaccount:demo:viewer
yes
$ kubectl auth can-i delete pods --as=system:serviceaccount:demo:viewer
no
# (exit status 1)
$ kubectl auth can-i --list --as=system:serviceaccount:demo:viewer | head -4
Resources                                       Non-Resource URLs                      Resource Names   Verbs
selfsubjectreviews.authentication.k8s.io        []                                     []               [create]
selfsubjectaccessreviews.authorization.k8s.io   []                                     []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                                     []               [create]
Forbidden messages say exactly what was denied: read the identity, the verb, the resource and the namespace:
  Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:demo:default" cannot list resource "pods" in API group "" in the namespace "demo"
Admission policies (Gatekeeper, Kyverno, Pod Security, ValidatingAdmissionPolicy) reject requests that are allowed by RBAC but break a rule. The message names the policy. Here a built-in ValidatingAdmissionPolicy refuses the :latest tag (the same idea as a Gatekeeper constraint):
$ kubectl run floating --image=nginx:latest
The pods "floating" is invalid: : ValidatingAdmissionPolicy 'no-latest' with binding 'no-latest-binding' denied request: images must carry an explicit tag and may not use :latest
# (exit status 1)
$ kubectl run floating --image=nginx
The pods "floating" is invalid: : ValidatingAdmissionPolicy 'no-latest' with binding 'no-latest-binding' denied request: images must carry an explicit tag and may not use :latest
# (exit status 1)
$ kubectl run pinned --image=nginx:1.27-alpine
pod/pinned created
In an admission error, "admission webhook ... denied the request" (Gatekeeper, Kyverno) or "ValidatingAdmissionPolicy ... denied request" tells you WHICH policy blocked you and usually why. The fix is to change the manifest (pin the tag, add the required label, set resource limits), not to bypass the policy. To see what policies exist: kubectl get constraints (Gatekeeper), kubectl get clusterpolicies (Kyverno), kubectl get validatingadmissionpolicies.
$ kubectl delete pod pinned --wait=false
pod "pinned" deleted from demo namespace

$ kubectl get nodes
NAME               STATUS   ROLES           AGE   VERSION
hb-control-plane   Ready    control-plane   3m   v1.37.0
$ kubectl describe node hb-control-plane | sed -n '/^Allocated resources:/,/^Events:/p' | head -8
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests     Limits
  --------           --------     ------
  cpu                1100m (13%)  0 (0%)
  memory             386Mi (1%)   532Mi (1%)
  ephemeral-storage  0 (0%)       0 (0%)
  hugepages-1Gi      0 (0%)       0 (0%)
$ kubectl top nodes; kubectl top pods -A --sort-by=memory | head      # live usage: needs metrics-server installed in the cluster
$ kubectl cordon node-2                    # stop NEW pods landing on a node
$ kubectl drain node-2 --ignore-daemonsets --delete-emptydir-data       # evict pods safely before maintenance (respects PodDisruptionBudgets)
$ kubectl uncordon node-2                  # allow scheduling again
$ kubectl taint nodes node-2 dedicated=gpu:NoSchedule                   # keep ordinary pods off a node
Requests are what the scheduler RESERVES; limits are what the kernel ENFORCES (memory over the limit means OOMKilled, CPU over the limit means throttling). A pod that requests more than any node has free stays Pending with "Insufficient cpu" or "Insufficient memory". Compare what is requested to what is allocatable:
$ kubectl get nodes -o custom-columns='NODE:.metadata.name,CPU:.status.allocatable.cpu,MEMORY:.status.allocatable.memory,PODS:.status.allocatable.pods'
NODE               CPU   MEMORY       PODS
hb-control-plane   8     32541204Ki   110
$ kubectl get pods -A -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,MEM_LIM:.spec.containers[*].resources.limits.memory' | head -4
NS                   POD                                        CPU_REQ   MEM_LIM
demo                 pinned                                     <none>    <none>
demo                 web-79d6b8c5f2-m6w2t                       50m       64Mi
demo                 web-79d6b8c5f2-p4r7v                       50m       64Mi

$ kubectl delete deployment,service -l app=web
deployment.apps "web" deleted from demo namespace
service "web" deleted from demo namespace
$ kubectl delete configmap app-config
configmap "app-config" deleted from demo namespace
$ kubectl get all
NAME                       READY   STATUS        RESTARTS   AGE
pod/pinned                 0/1     Terminating   0          3m
pod/web-79d6b8c5f2-m6w2t   1/1     Terminating   0          3m
pod/web-79d6b8c5f2-p4r7v   1/1     Terminating   0          3m
pod/web-79d6b8c5f2-z8h5n   1/1     Terminating   0          3m
$ kubectl delete pod NAME --grace-period=0 --force          # last resort: removes the API object without waiting for the node (the container may keep running)
$ kubectl delete ns demo                                    # deletes EVERYTHING in the namespace: check the name, then the context, then press Enter
Everything in a namespace, including what "get all" hides (ConfigMaps, Secrets, PVCs, Ingresses, RoleBindings and custom resources); events are filtered out here because there are so many:
$ kubectl api-resources --verbs=list --namespaced -o name | xargs -I{} sh -c 'kubectl get {} -o name 2>/dev/null' | grep -v '^event' | sort | head -8
configmap/kube-root-ca.crt
role.rbac.authorization.k8s.io/pod-reader
rolebinding.rbac.authorization.k8s.io/viewer-pods
serviceaccount/default
serviceaccount/viewer
Namespace stuck in Terminating: something inside has a finalizer that no controller removes. List what is left (the command above) and look at its metadata.finalizers; fix the controller, or as a last resort kubectl patch the object to clear them. Pods stuck Terminating usually sit on an unreachable node (kubectl get nodes shows NotReady).

$ kubectl explain pod.spec.containers.resources | head -12
KIND:       Pod
VERSION:    v1

FIELD: resources <ResourceRequirements>


DESCRIPTION:
    Compute Resources required by this container. Cannot be updated. More info:
    https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
    ResourceRequirements describes the compute resource requirements.
    
FIELDS:
$ kubectl explain deployment.spec.strategy.rollingUpdate
GROUP:      apps
KIND:       Deployment
VERSION:    v1

FIELD: rollingUpdate <RollingUpdateDeployment>


DESCRIPTION:
    Rolling update config params. Present only if DeploymentStrategyType =
    RollingUpdate.
    Spec to control the desired behavior of rolling update.
    
FIELDS:
  maxSurge	<IntOrString>
    The maximum number of pods that can be scheduled above the desired number of
    pods. Value can be an absolute number (ex: 5) or a percentage of desired
    pods (ex: 10%). This can not be 0 if MaxUnavailable is 0. Absolute number is
    calculated from percentage by rounding up. Defaults to 25%. Example: when
    this is set to 30%, the new ReplicaSet can be scaled up immediately when the
    rolling update starts, such that the total number of old and new pods do not
    exceed 130% of desired pods. Once old pods have been killed, new ReplicaSet
    can be scaled up further, ensuring that total number of pods running at any
    time during the update is at most 130% of desired pods.

  maxUnavailable	<IntOrString>
    The maximum number of pods that can be unavailable during the update. Value
    can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%).
    Absolute number is calculated from percentage by rounding down. This can not
    be 0 if MaxSurge is 0. Defaults to 25%. Example: when this is set to 30%,
    the old ReplicaSet can be scaled down to 70% of desired pods immediately
    when the rolling update starts. Once new pods are ready, old ReplicaSet can
    be scaled down further, followed by scaling up the new ReplicaSet, ensuring
    that the total number of pods available at all times during the update is at
    least 70% of desired pods.
$ kubectl get --raw /healthz; echo
ok
$ kubectl get --raw '/readyz?verbose' | tail -3
[+]poststarthook/apiservice-openapiv3-controller ok
[+]shutdown ok
readyz check passed
$ kubectl api-versions | head -5
admissionregistration.k8s.io/v1
apiextensions.k8s.io/v1
apiregistration.k8s.io/v1
apps/v1
authentication.k8s.io/v1
explain documents any field of any resource (add --recursive for the whole tree); api-resources and api-versions show what the cluster supports ("no matches for kind X" usually means a missing CRD or an old apiVersion, such as extensions/v1beta1 on a modern cluster). Debug what kubectl sends: -v=6 prints each API call, -v=8 the request and response bodies (careful: tokens can appear at high verbosity).
Plugins: kubectl krew installs plugins (kubectx, kubens, neat, tree, view-secret, who-can); kubectl plugin list shows installed ones; any executable named kubectl-NAME on the PATH works as kubectl NAME.
Debugging with an ephemeral container (works even on images with no shell), and on a node:
$ kubectl debug -it web-xxxx --image=busybox:1.37 --target=nginx          # adds a temporary container to the running pod, sharing its processes
$ kubectl debug node/node-2 -it --image=ubuntu                            # a pod on the node with the host file system at /host
$ kubectl debug web-xxxx --copy-to=web-debug --container=nginx -- sleep 1d   # a copy of the pod with a different command, for investigation

Before anything that changes or deletes: kubectl config current-context, then the namespace, then the command with --dry-run=server or diff.
Name your contexts so production is obvious, and keep production in a different terminal profile.
Prefer manifests in Git and kubectl apply (or a GitOps tool) over edit and patch on live systems.
Delete by exact name or by a label you have just listed (kubectl get -l ... first), never kubectl delete ... --all in an unknown namespace.
Give people and automation the least RBAC needed (view for most), and test it with auth can-i --as.
Set resource requests and limits, readiness and liveness probes, and pinned image tags: most of the failures above come from missing ones.
Read the Events before restarting anything. Restarting pods hides the clue.
$ kubectl get secret db-password -o jsonpath='{.data.password}' | base64 -d           # secrets are only base64, not encrypted: limit who can read them, and never paste them into tickets

Message                                              Meaning and first move
Unable to connect to the server: dial tcp ...        wrong or dead cluster in the kubeconfig, VPN down, expired or missing credentials: kubectl config current-context; kubectl cluster-info
error: You must be logged in to the server (Unauthorized)   the token or certificate expired (Azure: az login, kubelogin; cloud CLIs refresh it)
Error from server (Forbidden): ... cannot ...       RBAC: you lack that verb on that resource in that namespace: kubectl auth can-i; ask for a Role/RoleBinding
admission webhook "..." denied the request          a policy (Gatekeeper, Kyverno, Pod Security) refused the manifest: read the message, fix the manifest
the namespace "x" not found / not found            wrong namespace: kubectl get ns; check -n and the context's default namespace
no matches for kind "X" in version "Y"               the CRD is not installed, or the apiVersion is outdated: kubectl api-resources | grep -i X
field is immutable                                  you cannot change that field of an existing object: delete and recreate (replace --force), or use a new name
exceeded quota                                       a ResourceQuota in the namespace is used up: kubectl describe quota
Pending PVC, Pod stays ContainerCreating             no storage class or volume could not attach/mount: kubectl describe pvc / pod, check the storage provider
0/N nodes are available: ... Insufficient cpu        requests too big for the nodes, or taints and affinity block placement: describe pod shows each reason
Evicted                                              the node ran low on memory or disk: kubectl describe node (Conditions: MemoryPressure, DiskPressure)
Error: ImagePullBackOff on a private registry        missing imagePullSecrets, expired token, or the node cannot reach the registry
x509: certificate signed by unknown authority        a proxy or private CA: add the CA to the kubeconfig (certificate-authority-data) or fix the trust store

Always know which context and namespace you are in. Make it visible in your prompt.
describe (Events), logs --previous and get -o yaml answer most "why" questions. Learn jsonpath and custom-columns once, use them forever.
Create manifests with --dry-run=client -o yaml, review them with kubectl diff, apply them from Git.
A Service without endpoints is a selector or readiness problem, not a networking problem.
Rollouts are your safety net: rollout status in CI, rollout undo when it goes wrong, and readiness probes so bad pods never receive traffic.
Test permissions with auth can-i --as before you hand out a role. Treat admission errors as the policy talking: fix the manifest.
kubectl is a thin client for an API: kubectl get --raw and -v=6 show what it does, and the same calls work from curl or any language.
"It works with kubectl exec but not through the Service" is the beginning of every Kubernetes debugging story, and the endpoints list is usually the end of it.