Kubernetes Scaling, Probes and Troubleshooting with kubectl

Pager goes off: the ticket-gate API at a ferry terminal shows CrashLoopBackOff and the morning rush is twenty minutes away. The worst response is to start changing things. The best one is a fixed sequence of questions, each answered by one kubectl command. This lesson covers the settings that prevent most of these incidents (scaling, resource requests and probes) and then walks through the sequence for the two failures you will meet most often.

Scaling: manual first, then automatic

Manual scaling is a single field: kubectl scale deployment/ticket-gate --replicas=5. Automatic scaling uses a HorizontalPodAutoscaler (HPA), which watches a metric and adjusts replicas between a minimum and maximum. CPU utilisation is the usual starting metric. Two prerequisites trip people up. The cluster needs a metrics source such as metrics-server, and the Pods must declare CPU requests, because utilisation is calculated as a percentage of the request.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ticket-gate
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ticket-gate
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Read it as: keep average CPU around 70 percent of each Pod's request, never below two Pods or above ten. If you also set replicas in the Deployment manifest and re-apply it, you will fight the autoscaler, so omit that field once an HPA owns the count. Autoscaling Pods does not add nodes; node-level scaling is a separate component (a cluster autoscaler) and depends on your platform.

Requests, limits and probes

A request is what the scheduler reserves for a container. A limit is the ceiling the kernel enforces. Exceed a memory limit and the container is killed with OOMKilled. Exceed a CPU limit and it is throttled instead. Three probes tell the kubelet about health, and mixing them up is a classic source of outages.

containers:
  - name: gate
    image: ghcr.io/example-org/ticket-gate:3.1.2
    ports:
      - containerPort: 8080
    resources:
      requests:
        cpu: 250m
        memory: 256Mi
      limits:
        memory: 512Mi
    startupProbe:
      httpGet:
        path: /health
        port: 8080
      failureThreshold: 30
      periodSeconds: 2
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      periodSeconds: 5
    livenessProbe:
      httpGet:
        path: /health
        port: 8080
      periodSeconds: 10
      failureThreshold: 3

The debugging walkthrough: read, then ask why

Start the same way every time. kubectl get pods shows the status, and kubectl describe pod <name> shows the evidence. Everything below is about what to look for in describe and why.

Case 1: CrashLoopBackOff

This status means the container starts, exits, and Kubernetes keeps restarting it with growing delays. The image pulled fine, so the problem is what happens after start. Work through these questions in order.

1. What exit code did it leave with? In describe, under Containers, read Last State: Terminated, Reason and Exit Code. Exit code 1 or 2 usually means the application itself failed. 137 means it was killed with SIGKILL, and a reason of OOMKilled confirms it hit its memory limit. 126 or 127 means the command could not be executed or was not found, typically a wrong command or entrypoint.
2. What did it say before dying? kubectl logs <pod> --previous prints the last terminated container's logs. Plain logs often shows the new, empty attempt. Missing environment variables, an unreachable database host or a bad config file path usually appear here.
3. Is the liveness probe killing a healthy app? Look for Liveness probe failed in Events. If the app needs 40 seconds to boot and the probe starts at 10, the kubelet restarts it forever. Add a startup probe or lengthen the delay.
4. Do the config objects exist and have the right keys? A wrong key name in a configMapKeyRef fails differently (CreateContainerConfigError), but a present-but-wrong value fails inside the app and shows in the logs.
5. Reproduce it interactively. Run the image with a sleep command in a debug Pod, or use kubectl debug to attach a debug container to the failing Pod, then run the real start command by hand and watch it fail.

Containers:
  gate:
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
    Restart Count:  6
    Limits:
      memory:  512Mi

That block says the process was killed for exceeding 512Mi, so reading logs will not help much. The fix is raising the limit, or finding the memory leak or oversized batch behind it.

Case 2: ImagePullBackOff (and its sibling ErrImagePull)

Here the container never started, because the node could not fetch the image. The Events section at the bottom of describe carries the reason, and three causes cover most cases.

1. Name or tag is wrong. An event like manifest unknown or not found means the registry has no such tag. Compare the Pod's image string character by character against what was pushed (1.4.1x versus 1.4.1, a missing registry hostname, upper case in a repository name).
2. Credentials are missing. An event like pull access denied or unauthorized for a private registry means the node had no valid login. Create a docker-registry Secret and reference it under imagePullSecrets in the Pod spec or on the ServiceAccount, in the same namespace as the Pod.
3. The node cannot reach the registry. Events mentioning timeouts, TLS errors or DNS lookup failures point to network, proxy or firewall issues rather than the image. Try pulling the same image from the node, or from a debug Pod, to confirm.

kubectl get pods -l app=ticket-gate
kubectl describe pod <pod-name>
kubectl logs <pod-name> --previous
kubectl get events --sort-by=.metadata.creationTimestamp
kubectl rollout undo deployment/ticket-gate
kubectl get endpoints ticket-gate

Those six commands are the sequence in miniature: see the status, read the evidence, read the previous logs, scan recent cluster events, roll back if customers are affected while you investigate, and finally confirm the Service has endpoints again. Rolling back first and investigating second is a perfectly professional order when the incident is live.

Common mistakes and the symptoms they cause

• No CPU requests with an HPA. Symptom: the HPA shows unknown for its target and never scales. Fix: set requests and confirm metrics-server is installed.
• Memory limit far below real use. Symptom: restarts with OOMKilled and exit code 137 under load only.
• Liveness probe pointing at a slow dependency. Symptom: all Pods restart together when the database is slow.
• Probe on the wrong port or path. Symptom: Readiness probe failed: HTTP probe failed with statuscode: 404 and a Pod that is Running but never Ready.
• Fixing only the live Pod. Symptom: a hand-edit works until the next rollout. Fix: change the manifest in version control and apply it.

How this shows up in real projects and interviews

"A Pod is in CrashLoopBackOff, what do you do?" is perhaps the most common practical Kubernetes interview question, and the interviewer wants a method, not a guess: status, describe, exit code, previous logs, probes, config. Say aloud what each command tells you. On call, the discipline is the same, and teams that pair it with automated deployment checks (see CI/CD) catch many of these failures before they reach production.