Zero-downtime deployments on Kubernetes: probes, rolling updates and safe migrations

Kubernetes makes rolling deployments the default, which leads many teams to assume they already have zero-downtime releases. Then they watch their error rate during a deploy and see a short burst of 502s and connection resets. The rollout itself worked as designed; the application and its configuration did not cooperate with it.

Genuinely invisible deployments depend on four things working together: the scheduler knowing when a pod can take traffic, pods shutting down without dropping in-flight requests, enough capacity remaining during the rollout, and database changes that are compatible with both the old and the new version of the code. This article covers each in turn.

How a rolling update actually works

When you change a Deployment's pod template, Kubernetes creates a new ReplicaSet and gradually scales it up while scaling the old one down. Two settings control the pace: maxSurge is how many extra pods may exist above the desired count, and maxUnavailable is how many may be missing. Setting maxUnavailable: 0 means a pod is only removed after its replacement is Ready, which is what you want for user-facing services.

The crucial word is Ready. Kubernetes has no idea whether your app can serve requests unless you tell it, and that is the job of probes.

Probes: startup, readiness and liveness

• Startup probe: runs first and holds off the other probes until it passes. Use it for slow-booting apps so liveness does not kill them during startup.
• Readiness probe: decides whether the pod receives traffic from Services. Failing it removes the pod from endpoints without restarting it.
• Liveness probe: decides whether the container should be restarted. Failing it kills the container.

The most common mistake is pointing liveness at an endpoint that checks dependencies. If the database has a hiccup, every pod fails liveness, Kubernetes restarts them all, and a brief database blip becomes a full outage. Keep liveness trivial: is this process able to respond at all? Readiness can be more thorough, but even there be careful about shared dependencies, because failing readiness everywhere at once also leaves no pods serving.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  revisionHistoryLimit: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      terminationGracePeriodSeconds: 45
      containers:
        - name: api
          image: ghcr.io/example/api:1.4.2
          ports:
            - containerPort: 3000
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              memory: 512Mi
          startupProbe:
            httpGet:
              path: /healthz
              port: 3000
            periodSeconds: 2
            failureThreshold: 30
          readinessProbe:
            httpGet:
              path: /readyz
              port: 3000
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:
            httpGet:
              path: /healthz
              port: 3000
            periodSeconds: 10
            failureThreshold: 3
          lifecycle:
            preStop:
              exec:
                command: ["sleep", "10"]

Graceful shutdown: the part most teams miss

When Kubernetes terminates a pod, two things happen in parallel: the kubelet sends SIGTERM to the container, and the control plane removes the pod from Service endpoints. Those updates propagate to kube-proxy, ingress controllers and load balancers asynchronously. For a short window, traffic can still be routed to a pod that has already started shutting down. If the process exits immediately on SIGTERM, those requests fail.

The fix has two parts. A preStop hook that sleeps for a few seconds delays SIGTERM until routing has caught up; this is why the manifest above sleeps for 10 seconds. Then the application handles SIGTERM by refusing new connections, finishing in-flight requests and closing resources before exiting. The whole sequence must fit within terminationGracePeriodSeconds, after which Kubernetes sends SIGKILL.

import http from "node:http";
import { app } from "./app";
import { pool } from "./db";

const server = http.createServer(app);
let shuttingDown = false;

// Readiness reports unavailable as soon as shutdown begins.
app.get("/readyz", (_req, res) => {
  res.status(shuttingDown ? 503 : 200).send(shuttingDown ? "draining" : "ok");
});
app.get("/healthz", (_req, res) => res.status(200).send("ok"));

server.listen(3000);

process.on("SIGTERM", () => {
  shuttingDown = true;
  console.log("SIGTERM received, draining connections");

  // Stop accepting new connections; the callback fires when in-flight requests finish.
  server.close(async () => {
    await pool.end();
    process.exit(0);
  });

  // Close idle keep-alive sockets so server.close() can complete.
  server.closeIdleConnections();

  // Safety net: exit before Kubernetes sends SIGKILL.
  setTimeout(() => process.exit(1), 30_000).unref();
});

Protect capacity during rollouts and node maintenance

Rolling updates are not the only thing that removes pods. Node upgrades, autoscaler scale-downs and manual drains evict pods too. A PodDisruptionBudget tells Kubernetes how many pods of a workload must remain available during these voluntary disruptions.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api

Combine this with topology spread constraints or pod anti-affinity so replicas land on different nodes and zones. Three replicas on one node give you no protection when that node is drained. Also set realistic resource requests; if a rollout's surge pod cannot be scheduled because the cluster is full, the rollout stalls.

Long-lived connections need extra thought. WebSockets, server-sent events and gRPC streams are not drained by a few seconds of sleep; they stay open until one side closes them. Have the server send a close or reconnect signal during shutdown, make clients reconnect with backoff, and set the grace period long enough for connections to move over. The same applies to background workers: stop pulling new jobs on SIGTERM, finish or release the current one, and rely on the queue to redeliver anything that was not acknowledged.

Database migrations: expand, migrate, contract

During a rolling update, old and new versions of your code run at the same time against the same database. Any schema change must therefore work with both versions. Renaming a column in one step breaks whichever version expects the other name. The safe approach is the expand and contract pattern, spread across several deploys.

1. Expand: add the new column or table. Make it nullable or give it a default. Old code ignores it.
2. Dual write: deploy code that writes to both old and new columns, and reads from the old one.
3. Backfill: copy existing data in batches, outside the deploy, so you never hold long locks.
4. Switch reads: deploy code that reads from the new column.
5. Contract: once nothing uses the old column, drop it in a later release.

-- Safe on large tables in Postgres:
ALTER TABLE orders ADD COLUMN fulfilment_status text;          -- nullable, no rewrite
CREATE INDEX CONCURRENTLY orders_fulfilment_status_idx
  ON orders (fulfilment_status);                               -- does not block writes

-- Add a NOT NULL rule without a long exclusive lock:
ALTER TABLE orders
  ADD CONSTRAINT orders_fulfilment_status_not_null
  CHECK (fulfilment_status IS NOT NULL) NOT VALID;
ALTER TABLE orders VALIDATE CONSTRAINT orders_fulfilment_status_not_null;

-- Always bound how long a migration may wait for a lock.
SET lock_timeout = '5s';

Run migrations as a separate step before the rollout, for example as a Kubernetes Job triggered by your pipeline, rather than on application startup. Running them on startup means every replica races to migrate, and a slow migration blocks pods from becoming ready.

Verify and roll back quickly

Make the pipeline wait for the rollout to finish and fail loudly if it does not. kubectl rollout status exits non-zero when the deployment exceeds its progress deadline, and kubectl rollout undo returns to the previous ReplicaSet. Because the schema is backwards compatible, rolling back the code is always safe.

Readiness only proves a pod can answer a health check, not that the release is healthy. Watch error rates and latency for the service during and immediately after each rollout, ideally on a dashboard linked from the deploy notification. For higher-risk services, progressive delivery tools can shift a small share of traffic to the new version first and promote or abort automatically based on those metrics. That is a worthwhile step once the basics in this article are in place, not a substitute for them.

kubectl apply -f k8s/
kubectl rollout status deployment/api --timeout=5m || {
  echo "Rollout failed, rolling back"
  kubectl rollout undo deployment/api
  exit 1
}

Key takeaways

• Use maxUnavailable: 0 and a surge so capacity never drops during a rollout.
• Keep liveness trivial, make readiness meaningful, and add a startup probe for slow boots.
• Add a preStop sleep and handle SIGTERM by draining connections within the grace period.
• Protect capacity with PodDisruptionBudgets and spread replicas across nodes.
• Ship schema changes with expand and contract, run migrations as a separate step, and always set a lock timeout.
• Gate the pipeline on rollout status and keep rollback a single command.

Our DevOps and cloud team sets up Kubernetes platforms where deploys are routine, not risky. Talk to us about your release process.