DEV Community: softverseAdmin

Restore Drills in 2026: You Can't Automate the Declaration, Only the Comparison

softverseAdmin — Mon, 02 Nov 2026 06:10:10 +0000

Restore Drills in 2026: You Can't Automate the Declaration, Only the Comparison

The previous post in this little series — Backup Verification in 2026: Exit Code 0 Is Not a Backup — grew out of a comment thread under a dev.to article by dafili, "My Backup Exited 0 for Three Weeks and Nothing Paged Me". The conversation did not stop at publication. Dafili corrected one claim in that post (the correction is in, and credited), and then pushed the whole idea one level deeper — to a place the four-level ladder does not reach on its own.

His point, in his words: "You cannot automate the declaration, only the comparison."

This post is about what that sentence means, because once you hear it, a hole opens up in the middle of most restore drills — including the tidy ones from the ladder post. The ladder tells you how to check a backup. It never tells you what "correct" means. And if the definition of correct comes from the same system you are checking, your drill can pass beautifully while verifying nothing at all.

TL;DR: A restore drill is only as honest as the expectations it compares against. If those expectations are derived from the backup or the system being verified — row counts read from the dump itself, file lists generated by the backup tool — the check is circular: it catches corruption, never wrongness. The fix is a fingerprint: a short, hand-written declaration of what a correct restore must contain (an expected file list, one file's sha256, a row count, one config value). A human re-declares it quarterly; automation compares against it nightly and shouts on drift. And the drill itself gets monitored with a days_since_last_successful_manual_restore gauge that pages at about 120 days — because an unmonitored drill quietly stops happening.

Originally published on DevOps Enginer.

The ladder checks the climb, not the destination

The four levels from the previous post — the artifact exists, it is intact, it can be read, it has actually been restored — still stand. Level 4 is where trust lives: restore into a scratch environment and verify the result like a user would.

But look closely at how a typical level-4 drill "verifies." It restores the dump, then runs queries against the restored database: count the rows in users, check that orders has recent entries, confirm the application boots. Where do the expected values for those checks come from? Usually from the restored data itself, or from the same monitoring that watched the original system. The drill asks the backup: do you agree with yourself? The backup always says yes.

That is how a drill passes on data that is perfectly intact and completely wrong. Dafili made this distinction in the same thread: an integrity check catches media or repository corruption, but only a restore catches data that is wrong — and even then, only if the restore is compared against something that did not come from the restore. An empty mount backed up faithfully, checksummed faithfully, restored faithfully. A dump of the wrong database, in perfect condition. Every level green. Nothing recovered that you actually wanted.

He explored the same trap from another angle in a follow-up article, "Three checks that look like they passed but validated nothing": checks that execute correctly, return success, and prove nothing, because they validate a different property than the one you care about. A restore drill with self-derived expectations is exactly that shape of check — bigger, more expensive, and just as capable of validating nothing.

The fingerprint: four things a human writes down

Dafili's answer is a fingerprint — a small file a human writes by hand, declaring what a correct restore must contain. Not generated. Not exported from the database. Declared, the way you would declare what matters before a trip: passport, keys, the one document you cannot replace. His fingerprint has four components:

An expected file list. The files (or tables, or object prefixes) that must be present in the restore. Not all of them — the ones whose absence would mean the backup missed its point.
One file's sha256. Pick one file whose content you can predict — a config file, a seed data file, a known document — and record its hash. The restore must reproduce it bit for bit.
A row count. One table that matters, with a count (or a sane range) a human states from knowing the business, not from querying the backup. Orders should be in the hundreds of thousands by now; if the restore says 41, something is deeply wrong even if the restore "worked."
One config value. A single setting whose correctness you can declare: the application's environment name, a feature flag that must be on, the schema version the code expects. Config drift is how "successful" restores fail at boot time.

Four declarations. Small enough to write in fifteen minutes, small enough to read at a glance, and — this is the whole design — none of them can be produced by the system they verify. The fingerprint stands outside the backup pipeline and judges it.

An example, as a plain file kept with the runbook:

# restore-fingerprint.yaml — declared by hand, re-declared quarterly
system: billing-db
expected_files:
  - /srv/app/config/app.env
  - /srv/app/data/seed-countries.csv
sha256:
  /srv/app/config/app.env: "9f2c…e41a"   # recorded at declaration time
row_counts:
  orders: ">= 400000"                      # stated from the business, not the dump
config_values:
  schema_version: "2026.09"                # what the current code expects
declared: 2026-11-02
next_review: 2027-02-02

The exact format does not matter. What matters is the direction of the arrow: expectations flow from a human's knowledge into the check, never from the checked system outward.

You cannot automate the declaration

Why insist a human writes it? Because the declaration is not data entry — it is a review disguised as data entry. To write the fingerprint, someone has to sit down and answer questions that no monitoring dashboard asks: What in this system would we actually miss? Which table is the business? Which config value, if wrong, turns a good restore into a bad morning? Those answers live in people's heads and in the incident history, not in the database.

In dafili's practice, the human re-declares the fingerprint quarterly — deliberately on the same cadence as the full restore drill. Systems drift legitimately over a quarter: new tables appear, row counts grow, config values change on purpose. The quarterly re-declaration is the moment a human looks at that drift on purpose, in calm conditions, and says: yes, this is what correct looks like now. Skip it, and the fingerprint slowly becomes a declaration about a system that no longer exists — which is its own quiet failure mode, and exactly why the declaration cannot be a one-time setup task.

This is also why automating the declaration defeats the purpose. A generator that rebuilds the fingerprint from production every night produces a fingerprint that is always exactly right and therefore says nothing: it re-declares whatever the system currently is, including its mistakes. The value of the fingerprint is precisely that it can disagree with the system — and that a human chose the terms of the disagreement in advance.

The comparison is the automation's job

Everything after the declaration should be boring, mechanical, and nightly. Restore the latest backup into a scratch environment, extract the actuals, and diff them against the fingerprint:

#!/usr/bin/env bash
# compare-restore.sh — runs after the nightly restore-to-scratch
set -euo pipefail

FAIL=0

# 1. Expected files present?
while read -r f; do
  [ -f "/tmp/restore-test$f" ] || { echo "MISSING: $f"; FAIL=1; }
done < <(yq -r '.expected_files[]' restore-fingerprint.yaml)

# 2. The declared hash matches?
sha256sum -c <<<"$(yq -r '.sha256."/srv/app/config/app.env"' restore-fingerprint.yaml)  /tmp/restore-test/srv/app/config/app.env" \
  || { echo "HASH DRIFT: app.env"; FAIL=1; }

# 3. The declared row count is in range?
ORDERS=$(psql restore_test -tAc "SELECT count(*) FROM orders;")
[ "$ORDERS" -ge 400000 ] || { echo "ROW COUNT DRIFT: orders=$ORDERS"; FAIL=1; }

# 4. The declared config value survived?
psql restore_test -tAc "SELECT value FROM settings WHERE key='schema_version';" \
  | grep -qx "2026.09" || { echo "CONFIG DRIFT: schema_version"; FAIL=1; }

exit $FAIL

The details will differ with your stack — the pattern is the point. Automation compares; it does not get an opinion. When the comparison fails, it shouts: a failed fingerprint comparison is a page, not a log line, because it means one of two things, both important — the backup is wrong, or the world changed and nobody re-declared it.

Drift is a conversation, not just an alarm

That second case deserves respect rather than dread. Most fingerprint failures, in a healthy team, will be legitimate change: a migration added a table, the row count outgrew the declared range, a config value rolled forward. The correct response is not to silence the check but to re-declare deliberately — a human reviews the drift, confirms it is intended, and updates the fingerprint as its own small, visible act.

Contrast that with the alternative everyone actually lives with: expectations embedded in the check script, edited quietly at 2 a.m. to make the red go away, until the check asserts nothing. The fingerprint does not prevent someone from weakening a check. It makes the weakening a declared event, in a file with a date on it, that a second person can review. That is the difference between a system that learns and a system that merely quiets down.

Monitor the drill itself

There is one more layer, and it might be the most practical idea in the whole thread. Automated nightly comparisons are excellent — right up until the human parts of the practice quietly die. The quarterly manual restore is the first thing skipped in a busy quarter. Then it is skipped twice. Nothing alarms, because the automation is still green; the drill's judgment — a person restoring by hand, noticing the weird thing no fingerprint anticipated — has been absent for a year and nobody knows.

Dafili's countermeasure is to monitor the drill itself with a gauge: days_since_last_successful_manual_restore, and to page when it crosses about 120 days. Not a ticket. A page — the same severity as a failed backup, because that is what it is: past roughly one quarter, you no longer have evidence that a human can restore this system, only evidence that a script once could.

In Prometheus terms it is a timestamp gauge the drill updates when — and only when — a manual restore passes its fingerprint comparison:

# Written by the drill checklist's final step, after a human-run
# restore passes the fingerprint comparison:
echo "last_successful_manual_restore_timestamp $(date +%s)" \
  > /var/lib/node_exporter/textfile/restore_drill.prom

# Page when no successful manual restore in ~120 days
(time() - last_successful_manual_restore_timestamp) / 86400 > 120

Readers of the ladder post will recognise the shape: it is the dead-man's-switch from that article, pointed at the humans. There, the monitored absence was verification success — page when the checks stop succeeding. Here, the monitored absence is the drill itself — page when the people stop practising. Automation watches the backups; the gauge watches the watchers' habits. A restore you have not performed in four months is a hypothesis, and the gauge exists to say so out loud while it is still cheap to fix.

Where does the fingerprint live?

One honest open question, because it came up in the thread and I do not have dafili's answer yet: where should the fingerprint be stored? I asked him directly in the conversation; while his reply is pending, the trade-offs are worth stating plainly, because the obvious answer has a thorn in it.

With the backup configuration is convenient — the drill finds it without ceremony. But it risks the fingerprint drifting with the thing it verifies: the same change that breaks the backup can quietly rewrite the expectations, the way a scale that recalibrates itself to whatever it weighs can never report a wrong weight.
In a separate repository, owned like a runbook, keeps the declaration independent and reviewable — changes arrive as commits a second person can see. The cost is one more place to keep current, and a drill that must fetch its expectations from somewhere it trusts.
On paper, or in the incident binder, sounds like a joke until you remember what the fingerprint is for: it is the declaration that survives the system. Four values fit on an index card.

Wherever it lives, the property to protect is independence: the fingerprint must not be writable by the pipeline it judges. When dafili answers, this section gets an update — his reasoning has improved this series twice already.

Restore-drill fingerprint checklist

A fingerprint file exists for each system that matters, written by hand — not generated from the system it verifies
It declares all four: an expected file list, one file's sha256, one row count (or range), one config value
A human re-declares it quarterly, on the same cadence as the full restore drill
The nightly drill restores to scratch and compares actuals against the fingerprint automatically
A failed comparison pages — and legitimate drift is resolved by deliberate re-declaration, never by editing the check to be quiet
A days_since_last_successful_manual_restore gauge (or equivalent) exists, with a page at about 120 days
The fingerprint is stored somewhere the backup pipeline itself cannot rewrite
Expectations in older checks that were derived from the checked system have been replaced with declared ones

FAQ

Isn't this just the level-4 verification from the previous post?
It is level 4 done honestly. The ladder says restore and verify; the fingerprint answers the question that post left open — verify against what? If the answer is "against values taken from the restore itself," the verification is circular. Declared expectations are what break the circle.

Four checks seem too few. Shouldn't the fingerprint cover everything?
Coverage is the automation's job — checksums, listings, and sampled reads already sweep the whole artifact nightly. The fingerprint has a different job: to be an independent witness. A witness who testifies about everything testifies about nothing; four declarations a human can hold in their head, and re-declare with full attention each quarter, beat four hundred generated ones nobody reads.

What if the business can't state a row count in advance?
Declare a range or a floor, stated from business knowledge ("orders grow by roughly ten thousand a month; a restore below last quarter's count is wrong"). Precision is not the point — independence is. An expectation that is roughly right and genuinely external catches the catastrophic wrongness the drill exists to catch.

Does the gauge replace scheduling the quarterly drill?
No — it catches the schedule failing. Calendars get ignored under load; that is precisely the failure the gauge exists to surface. Schedule the drill and monitor its successful completion, the same way you schedule backups and monitor verification success.

Summary

The ladder from the previous post — exists, intact, readable, restored — tells you how to check a backup. Dafili's fingerprint tells you what makes the checking mean something: expectations a human declared, standing outside the system, that the system cannot rewrite to flatter itself. You cannot automate the declaration, only the comparison. So declare by hand, re-declare quarterly, compare nightly, shout on drift — and keep a gauge on the drill itself, paging at about 120 days, so the practice cannot die in silence.

Credit where it is due: the fingerprint doctrine, the declaration/comparison line, and the drill gauge all come from dafili, in the comment thread under "My Backup Exited 0 for Three Weeks and Nothing Paged Me" — the same thread that produced the ladder post and corrected it. His follow-up, "Three checks that look like they passed but validated nothing", is the natural next read. This series keeps getting better because a reader keeps arguing with it in public. Long may that continue.

Backup Verification in 2026: Exit Code 0 Is Not a Backup

softverseAdmin — Sun, 01 Nov 2026 05:19:56 +0000

Backup Verification in 2026: Exit Code 0 Is Not a Backup

A story made the rounds in a DevOps community recently, and it should make anyone who owns a cron job uncomfortable. An engineer checked on a backup script that had been running perfectly by every visible measure — green in the scheduler, exit code 0, night after night — and found it had been "succeeding" at backing up nothing at all for three weeks, ever since something upstream had quietly changed. No error. No page. Just three weeks of confidence with nothing underneath it.

TL;DR: A backup job's exit code tells you the script ran, not that you have a backup. A backup is only trustworthy at four levels: (1) the artifact exists, (2) it is structurally intact, (3) it can be opened and read, and (4) it has actually been restored and the data verified. Most teams verify level 1 and stop. This guide walks a practical verification ladder — checksums, restic check, pg_restore --list, scheduled restore drills — plus the monitoring change that matters most: page on verification failure, not job failure.

Originally published on DevOps Enginer.

This is deliberately not a Kubernetes backup guide — cluster tooling like Velero and etcd snapshots is covered in the Kubernetes Backup and Disaster Recovery post. This is about the layer underneath every stack: database dumps, file archives, and object-storage backups, and how to prove they work before the day you need them.

Why exit code 0 lies

Exit code 0 means one narrow thing: the last command in your script did not report an error. It says nothing about what the script actually captured. Backup scripts fail silently in a handful of classic ways:

Wrong or empty source paths. A directory gets moved, a mount fails to come up, or a path is renamed upstream. The archiver dutifully compresses an empty directory and exits 0. Your backup is a perfectly valid archive of nothing.
Expired or rotated credentials. The dump tool fails to authenticate and writes a 0-byte file — or the wrapper script swallows the error, logs a warning nobody reads, and still exits successfully because the upload of that empty file worked fine.
Silent partial captures. A database dump taken without the right flags can complete while missing schemas, extensions, or large objects. The file is large, recent, and wrong.

The pattern underneath all of these: the job was monitored, but the backup was never verified. Those are different jobs, and they need different checks.

The four levels of backup confidence

A useful way to think about backup verification is as a ladder. Each level proves something the level below cannot:

Level 1 — The artifact exists. A file or snapshot with the expected name appeared in the expected place, recently, with a plausible size. This is what most monitoring checks. It proves a process ran.

Level 2 — The artifact is structurally intact. The archive is not truncated or bit-rotted. Checksums match; the backup tool's own integrity check confirms its internal structures — indexes, metadata, pack files — are consistent. This proves the bits survived storage.

Level 3 — The artifact can be opened and read. The restore tool can actually parse the backup: list the table of contents of a dump, enumerate the files in an archive, read back a sample of the stored data. This proves the format is valid and the content is what you think it is — not an empty tree, not the wrong source.

Level 4 — The artifact has been restored and the data verified. You restored it into a real (throwaway) environment and checked the result: the database answers queries, row counts and key tables look right, application files open. This is the only level that proves you can recover. Everything below it is inference.

Notice what the ladder implies: levels 1–3 are cheap and automatable on every backup. Level 4 is more expensive, so you schedule it — sampled nightly or weekly, full restores on a rotation. What you cannot do is pretend level 1 is level 4. That gap is exactly where the three-weeks-of-nothing story lives.

The verification ladder in practice

Here is how to test yours, level by level. The examples use common tools — restic, PostgreSQL dumps, plain tarballs in object storage — but the pattern transfers to whatever you run.

Level 1: existence, freshness, and size sanity

Check that the newest artifact exists, is younger than your backup interval (plus slack), and is within a sane size band of its predecessors. A backup that suddenly shrinks by 90% is a failed backup wearing a success costume.

# Fail loudly if no backup newer than 26 hours exists in the bucket
aws s3 ls s3://my-backups/postgres/ --recursive \
  | sort | tail -n 1

# Size sanity: compare today's artifact against the 7-day median
# and alert if it deviates by more than ~50%

Tools like restic make this easier because the repository keeps snapshot metadata:

restic snapshots --last 1 --json   # newest snapshot time, paths, tags
restic stats                       # repository size trend

Encode the rule: a missing, stale, or suddenly-tiny backup is a verification failure, even if the job that produced it exited 0.

Level 2: structural integrity with checksums and tool-native checks

If you produce plain archives, generate a checksum at backup time and verify it at check time:

sha256sum -c postgres-2026-11-01.dump.sha256
gzip -t nightly-files.tar.gz   # quick structural test for gzip streams

restic has integrity checking built in:

restic check

restic check verifies the repository structure — all packs, indexes, and metadata present and consistent. It is fast and belongs in the nightly job. What it does not do by default is re-read your backup data — that is level 3.

Level 3: prove it can be read back

For archives, listing the contents is the minimum viable read test — it forces the tool to walk the whole structure:

tar -tzf nightly-files.tar.gz > /dev/null   # fails on truncation/corruption

For PostgreSQL custom-format dumps, pg_restore --list parses the dump's table of contents without restoring anything:

pg_restore --list postgres-2026-11-01.dump | head -n 50

Scan that listing for what should be there: your critical tables, extensions, and the expected object count. An empty dump has a valid header and almost nothing under it — this is the check that catches the "backed up nothing" failure mode.

For restic, the read-back check is:

restic check --read-data-subset=5%

Understand the caveat before you rely on it: --read-data-subset=5% reads and verifies 5% of the data packs, not 5% of each file. It is sampling. A corrupt pack outside the sample will not be found tonight. Two practices make sampling honest:

Rotate with the n/t form, not the percentage. --read-data-subset=5% picks a random subset on every run, so runs can overlap and some packs may never get sampled at all. The n/t form is the deterministic one: restic splits the packs into t groups and checks group n. Pin it to the calendar — --read-data-subset=$(date +%u)/7 checks a different seventh each night and guarantees every pack is read within a week, for the same total I/O as one full check spread out.
Run a full read periodically. restic check --read-data (100%) on a monthly cadence closes any remaining gaps.

Also add a content spot-check at this level: restore a handful of known files to a temp directory and diff them, or query the snapshot for a file you know changed yesterday. Existence of expected content is a much stronger signal than existence of an artifact.

Level 4: restore drills — the only level that counts

Restore into a throwaway environment and verify the result like a user would. For PostgreSQL:

# Restore last night's dump into a scratch database
createdb -T template0 restore_test
pg_restore --no-owner --dbname=restore_test postgres-2026-11-01.dump

# Then verify like the application would
psql restore_test -c "SELECT count(*) FROM users;"
psql restore_test -c "SELECT max(created_at) FROM orders;"

Row counts in a plausible range and a recent max(created_at) on your busiest table prove the restore contains live, current data — not a skeleton schema and not last month's state.

For file backups, restore the latest snapshot to a scratch path and verify file counts and a sample of checksums against expectations:

restic restore latest --target /tmp/restore-test
find /tmp/restore-test -type f | wc -l

The drill is not done when the restore command exits 0. The drill is done when a check of the restored data passes. And time the drill: the duration you measure is your real recovery time, not the one in the runbook.

Make verification a separate job that can fail

The single most important structural change: verification must be its own job, with its own failure state, that runs after the backup job. Folded into the backup script, a verification failure disappears into the same green checkmark. Give it its own exit code and its own alert path.

#!/usr/bin/env bash
# verify-backups.sh — runs after the backup job, fails independently
set -euo pipefail

LATEST=$(restic snapshots --last 1 --json | jq -r '.[0].id')

restic check                                   # level 2
restic check --read-data-subset=5%             # level 3 (sampled)
restic restore "$LATEST" --target /tmp/restore-test --include /srv/app/config
test -s /tmp/restore-test/srv/app/config/app.env   # expected content present

echo "backup_verification_success $(date +%s)" \
  > /var/lib/node_exporter/textfile/backup_verification.prom

The script writes a success timestamp only if every level passed — and that timestamp is what you monitor.

Monitor the verification, not the job

Alerting on "backup job failed" catches crashes. It does not catch the failure mode this article is about, because in that failure mode the job succeeds. Change what you page on:

Page on verification failure. Any level failing — stale artifact, size anomaly, checksum mismatch, unreadable dump, failed drill — is a page-worthy event, because it means you currently do not have a proven backup.
Page on missing success. Track "time since last successful verification" and alert when it exceeds your threshold (for a nightly cycle, ~30 hours). This dead-man's-switch pattern catches the silent cases: the verification job itself dying, the scheduler stopping, credentials expiring before the check even runs. Absence of failure is not presence of success — so monitor the presence of success.
Track your verified restore point. The metric that matters in an incident is "how old is the newest backup that has passed level 4?" If that number quietly grows to three weeks, you want to know before the incident, not during it.

If you use Prometheus, the textfile pattern above plus one alert — time() - backup_verification_success > 30 * 3600 — covers most of this. The tooling is not the hard part; deciding that verification failure pages like a production outage is.

A restore-drill cadence you can actually keep

Verification fails as a practice when it depends on someone remembering. Put it on a calendar:

Cadence	Scope	Levels
Nightly (automated)	Every backup, right after it lands	1–3, with n/t-rotated read-data subsets
Weekly (automated)	Partial level-4: restore one database or one snapshot subset to a scratch environment; run content checks	4 (partial)
Monthly (automated + reviewed)	Full restore of one production system on rotation, timed end to end	4 (full)
Quarterly	Full restore drill treated like a game day: restore, verify, record duration and surprises, update the runbook	4 (full, measured)

The rotation matters: every system should get a full restore at least once a quarter, rather than an annual hero-drill on the one system everybody already understands. Record each drill's duration — that number is your measured RTO, and it is almost always worse than the assumed one. Finding that out in a drill is the entire point.

Cloud snapshots deserve a specific warning here: a snapshot of a disk or database is level 1 by default. Snapshots faithfully preserve corruption, they can be silently incomplete if taken mid-write without coordination, and "we have snapshots" has ended more than one incident review badly. Snapshots need the same ladder — restore one into an isolated instance, open it, query it — before they count as backups.

Backup verification checklist

Every backup has a separate verification job with its own pass/fail, running after the backup completes
Level 1: freshness and size-band checks on every artifact, every cycle
Level 2: checksums verified for plain archives; restic check (or your tool's equivalent) nightly
Level 3: read-back tests — pg_restore --list, tar -t, restic check --read-data-subset rotated with the n/t form, plus expected-content spot checks
Level 4: automated partial restores weekly; full restores monthly on rotation; quarterly timed drill
Monitoring pages on verification failure and on missing verification success (~30h for nightly cycles)
"Age of newest level-4-verified backup" is a visible metric, not a surprise
Restore durations are recorded, so your RTO is measured rather than assumed
Snapshots are restore-tested like any other backup before being trusted

FAQ

Is restic check on its own enough?
It covers level 2 well and, with --read-data-subset, samples level 3. It never restores anything, so it cannot prove level 4. Treat it as the nightly baseline, not the whole strategy.

Doesn't all this verification double my costs?
Levels 1–3 are metadata, checksums, listings, and sampled reads — cheap. Level 4 restores cost scratch compute for a few hours a month. Compare that with the cost of discovering, mid-incident, that the backups were empty.

My backups are managed by my cloud provider. Am I covered?
Managed backup services still produce artifacts that can be empty, misconfigured, or pointed at the wrong source. The provider runs the job; verification that the artifact restores your data is still yours.

What is the one change to make first?
Add the dead-man's-switch alert on "time since last successful verification." It is a small change, and it converts the worst failure mode — silent, indefinite, total — into a page within a day.

Summary

An exit code tells you a script ran. A trustworthy backup has been checked at four levels: it exists, it is intact, it can be read, and it has been restored with the data verified. Automate the first three nightly; schedule the fourth — weekly partials, monthly full restores on rotation, quarterly timed drills. And page when verification fails or goes missing, not merely when a job crashes. The engineer in the opening story had monitoring. What they did not have was a single check that asked the only question that matters: if I needed this backup tonight, could I restore it? Build the checks that answer that question while the answer is still cheap.

Prometheus in 2026: Metrics, PromQL and Alerting That Pages Only When It Matters

softverseAdmin — Sat, 31 Oct 2026 11:58:56 +0000

Prometheus in 2026: Metrics, PromQL and Alerting That Pages Only When It Matters

TL;DR: Prometheus fails teams in two predictable ways: cardinality explosions from unbounded labels, and alert noise that trains everyone to ignore pages. The fixes are disciplines, not features. Model metrics as counters, gauges, and histograms with small, bounded label sets; measure services with RED (rate, errors, duration) and resources with USE (utilisation, saturation, errors); compute ratios and quantiles with recording rules so dashboards and alerts stay fast; and alert on user-visible symptoms with an action attached — via Alertmanager grouping and inhibition — never on every cause you can measure. Keep Prometheus itself for what it is (superb short-to-medium-term metrics), and remote-write to long-term storage when history matters.

Originally published on DevOps Enginer.

The model in one picture

Prometheus pulls: on an interval, it scrapes an HTTP endpoint (conventionally /metrics) on each target, parses the exposition format, and stores samples in a time-series database. A time series is a metric name plus a set of labels — http_requests_total{job="api", status="200"} — and each unique combination is a separate series with its own storage and query cost. That one sentence explains almost every Prometheus behaviour and failure mode in this guide.

Three metric types do the work:

Counters only increase (requests served, errors seen). You almost never graph a counter raw; you graph its rate() — which also handles counter resets when a process restarts.
Gauges go up and down (queue depth, memory in use, in-flight requests). Current value, min/max/avg over a window — no rate needed.
Histograms count observations into buckets (le labels), which is what makes server-side quantiles possible: histogram_quantile() aggregates across instances, which summary-type metrics cannot do correctly. Choose bucket boundaries around the latencies you actually care about — your SLO threshold must live inside your buckets to be measurable.

What to measure: RED and USE

Instrumentation fails by excess more often than by absence. Two checklists keep it purposeful:

RED for every service and endpoint that matters: request Rate, Errors (ratio, not just count), Duration (histogram). This is the user-visible truth of a service, and the raw material of every good alert below.
USE for every resource: Utilisation (busy fraction), Saturation (queued work — the earliest warning), Errors. CPU steal aside, saturation metrics are the ones that predict incidents instead of narrating them.

Everything else — business metrics, per-feature counters, debug detail — is optional gravy. High-cardinality detail belongs in logs (the logging guide's correlation IDs are exactly for this), not in labels.

Cardinality: the budget everyone blows

Series count is the product of label-value combinations. A metric with 10 endpoints × 5 status codes × 4 instances is 200 series — fine. Add a user_id label and it becomes millions — the database slows, memory climbs, queries time out, and the postmortem writes itself. The rules:

Labels identify dimensions with few, bounded values: endpoint (normalised — /users/123 becomes /users/:id before it is a label), status class or code, method, instance, zone, version.
Never label with unbounded values: user IDs, request IDs, session tokens, full URLs, error message text. Each is a series factory.
Scrape-time relabeling is the safety net: Prometheus can drop or rewrite label values during scraping — the place to strip an offending label from a third-party exporter you cannot change.
Watch the budget: the targets and metrics with the highest series counts should be known numbers in your stack, reviewed like cloud spend (the Kubernetes cost guide makes the same argument for compute).

PromQL that earns its keep

Four patterns cover most production questions:

Error ratio (the alert-worthy form of errors):

sum(rate(http_requests_total{job="api", status=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="api"}[5m]))

Latency quantile from a histogram:

histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket{job="api"}[10m])))

Saturation — in-flight work versus capacity is usually a gauge ratio; the shape matters more than the exact metric names.

Absence detection — absent() catches the series that stopped, which is how dead exporters and vanished jobs announce themselves; pair it with a dead man's switch on the alerting pipeline itself, so "no alerts" can be distinguished from "alerting is broken."

Two habits separate useful PromQL from fragile PromQL: always rate() counters (never subtract samples by hand), and aggregate with sum by (...) naming exactly the labels you intend to keep — a query that silently sums across a label you cared about produces confident, wrong dashboards.

Recording rules: pay for queries once

Dashboards and alerts re-run the same expressions relentlessly. Recording rules precompute them into new series on an interval:

groups:
  - name: api-slo
    interval: 30s
    rules:
      - record: job:http_error_ratio:5m
        expr: |
          sum(rate(http_requests_total{job="api", status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="api"}[5m]))

Now the alert is trivial, fast, and testable against the recorded series:

      - alert: ApiHighErrorRate
        expr: job:http_error_ratio:5m > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "API 5xx ratio above 5% for 10 minutes"
          runbook_url: "https://wiki.example.com/runbooks/api-errors"

Note the anatomy of a good alert: a symptom (error ratio, sustained), a for duration that ignores blips, a severity, and a runbook link. An alert without a next action is a notification, not an alert.

Alerting philosophy: symptoms page, causes ticket

Alert fatigue is an availability risk in its own right — a team paged twenty times a night stops reading page one. The discipline that fixes it:

Page on symptoms users feel: error ratio, latency burn, saturation of the user path. These have a natural threshold — the SLO — and a natural owner.
Ticket (or dashboard) the causes: disk filling, a replica down, a certificate ageing. Important, rarely 3 a.m.-shaped; a Deployment with one of three replicas down is healthy by design and should page nobody.
Burn-rate thinking: rather than a single static threshold, alert on how fast the error budget is being consumed (fast burn pages, slow burn tickets). Recording rules make multi-window burn alerts practical; the concept matters more than any specific window pair.
Alertmanager does the traffic control: grouping collapses a hundred instance alerts from one outage into one notification; inhibition suppresses downstream symptom alerts while a known root alert fires; silences mute deliberately during maintenance; routing sends each alert to the team that owns the action. Configure these on purpose — default Alertmanager behaviour is a starting point, not a design.

Architecture and storage honesty

Local retention is short by design. A Prometheus server's local TSDB commonly retains around 15 days by default; that is a feature of its role — fast, recent, operational metrics — not a limitation to apologise for.
Long-term history goes elsewhere: remote-write to Thanos, Grafana Mimir, or VictoriaMetrics (the common open options) when you need months of data for capacity and trend work. Keep the operational Prometheus queryable even if the long-term tier is down.
Federation is rarely the answer it first appears to be; hierarchical scraping duplicates data and queries painfully. Prefer remote-write and per-team/per-cluster servers with a global query layer.
In Kubernetes, the Prometheus Operator's ServiceMonitor/PodMonitor resources turn scrape config into the same GitOps-reviewed YAML as everything else in this series (the observability guide covers the wider stack); kube-state-metrics supplies object state (pending pods, failed jobs, replica mismatches) and node exporter supplies host USE metrics. Watch the operator's own cardinality — Kubernetes state is prolific.
Short-lived jobs can't be scraped on an interval; the Pushgateway exists for them, with real caveats: it doesn't expire series for you, and pushed metrics lose the scrape's health signal. Prefer recording outcomes from the platform side (the Jobs guide's failure conditions) or pushing sparingly and deliberately.

Common mistakes

Unbounded labels, the cardinal sin above — most often a well-meant user_id, path, or error label that turns a metric into a million series over a weekend.
Alerting on causes. CPU at 80% pages; the error budget untouched. Pages spent on non-symptoms are how real symptoms get muted.
Counters graphed raw — ever-climbing lines that say nothing; or worse, alerts on them. rate() exists; use it.
Averaging quantiles. Averaging per-instance p95s produces a number that is no instance's p95 and nobody's truth. Histograms plus histogram_quantile over the aggregate is the correct shape.
No recording rules on heavy dashboards — the same 30-day, high-cardinality query from every open browser tab, grinding the server that is supposed to diagnose the incident.
Treating Prometheus as the archive. Local retention quietly rotates; the quarter's capacity review finds 15 days of data. Decide the retention tiers like log retention (logging guide): hot and short in Prometheus, long in the remote tier.
Silent alerting pipeline. Alertmanager down, webhook expired, routing typo — discovered during the incident it existed to announce. A dead man's switch (an always-firing alert whose absence pages) is the standard guard.

Where this fits

Prometheus is the metrics pillar of the observability guide's triad: metrics say what and how much, logs say what exactly, traces say where in the path. The discipline rhymes across all three — bounded cardinality here, volume control in logging, sampling in tracing — because all three are the same budget wearing different clothes: keep the signal, shed the noise, and make sure the alerting end of the pipeline deserves the trust the team puts in it.

FAQ

Prometheus or Grafana — do I need both?
They are different layers: Prometheus collects and stores metrics and evaluates alerts; Grafana visualises (Prometheus has a basic built-in UI for queries, not dashboards). Most stacks run Prometheus (+ Alertmanager) for data and alerting, Grafana for dashboards — Grafana also fronts logs and traces, which is where the triad becomes one investigation.

Pull or push — which is right?
Pull, by default: it gives Prometheus target health for free (up series), centralises interval control, and survives target restarts gracefully. Push (Pushgateway) is the exception for genuinely short-lived batch work, used with its caveats in mind.

How long should Prometheus retain data?
Operationally, two to four weeks local is common and usually plenty for incident work. Anything longer belongs in a remote-write long-term tier, where storage is cheaper and the queries are trend-shaped rather than incident-shaped.

Prometheus or OpenTelemetry?
Complementary, not rivals. OpenTelemetry is instrumentation and pipelines (APIs, SDKs, the Collector); Prometheus is a metrics backend with its own exposition format, which OTel can produce and feed (directly or via remote-write-compatible paths). Many stacks instrument with OTel and still query with PromQL.

When have I outgrown a single Prometheus?
When scrape volume, cardinality, or query load makes the single server the bottleneck — symptoms are scrape duration creeping toward the interval, slow range queries, and memory pressure dominated by series churn. The step is sharding by team/cluster plus a remote-write tier, not a bigger instance forever.

Kubernetes Jobs and CronJobs in 2026: Batch Work That Finishes (or Fails) Cleanly

softverseAdmin — Sat, 31 Oct 2026 11:48:58 +0000

Kubernetes Jobs and CronJobs in 2026: Batch Work That Finishes (or Fails) Cleanly

TL;DR: Deployments run forever; Jobs run to completion — and that one difference changes how you configure everything. A production Job needs an explicit contract: restartPolicy: Never (or OnFailure, knowingly), a backoffLimit you chose, an activeDeadlineSeconds so nothing runs forever, and ttlSecondsAfterFinished so finished work cleans itself up. CronJobs add three decisions on top: a timeZone, a concurrencyPolicy that matches whether runs may overlap, and a startingDeadlineSeconds so a controller outage doesn't fire a pile of missed runs at once. Batch failures are normal — design for retries that are idempotent, and alert on failure rather than discovering it in the data.

Originally published on DevOps Enginer.

The workload Deployments can't express

Everything earlier in this series assumes services: replicas that should always exist, replaced when they die. Batch work inverts the assumption — a migration, a report, a backup, an import has a done, and success means the pods stop existing. Running batch work in a Deployment produces the classic pathologies: a completed container restarted forever, a failing task CrashLooping with no concept of "give up," no record of whether the run succeeded.

The Job controller exists to own that lifecycle: create pods, watch them finish, count successes against completions, count failures against backoffLimit, and stop. CronJob is a Job factory on a schedule — nothing more mystical than that, which is the right way to reason about both.

The Job contract, field by field

apiVersion: batch/v1
kind: Job
metadata:
  name: nightly-export
spec:
  completions: 1
  parallelism: 1
  backoffLimit: 3
  activeDeadlineSeconds: 3600
  ttlSecondsAfterFinished: 86400
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: export
          image: registry.example.com/export:1.4.2
          args: ["--date", "yesterday"]

The fields that decide behaviour:

restartPolicy: Never (or OnFailure) — Jobs require one of these two; Always is rejected. The choice matters: with Never, a failed container means the pod fails and the controller creates a replacement pod (visible, countable, each with its own logs). With OnFailure, the kubelet restarts the container inside the same pod — fewer pod objects, murkier failure history. Default to Never unless you have a reason; it makes failures legible.
backoffLimit (default 6) — how many pod failures before the Job itself is marked Failed. Each retry waits with exponential backoff (10s, 20s, 40s… capped at 6 minutes). Six silent retries of a broken image is how a bad deploy burns an hour unnoticed; for most batch work 2–4 is a saner, louder choice. Finer control exists via pod failure policies — rules that fail the Job immediately on specific exit codes (a config error will never fix itself) while ignoring disruptions like node drains.
activeDeadlineSeconds — the Job's total time budget; exceed it and the Job fails and its pods are terminated. This is the field that prevents the "export job from March is still running" class of surprise.
ttlSecondsAfterFinished — after the Job completes or fails, wait this long, then delete the Job and its pods. Without it, finished Jobs and their pod corpses accumulate forever (and with them, cluttered kubectl get pods output and etcd weight). A day is a reasonable default window for debugging; the logging pipeline from the logging guide is what preserves evidence beyond it.
completions / parallelism — for work queues: completions is the total successes required, parallelism the pod ceiling. Leave both unset for the simple one-shot case (one success, one pod). Indexed Jobs (completionMode: Indexed) give each pod a stable index (JOB_COMPLETION_INDEX) for sharded work — partition the input by index instead of running a separate queue service.

Idempotency: the requirement retries impose

A retried Job runs your code again against a world the first attempt may have half-changed. Batch code must therefore be idempotent or transactional: write to a staging key and publish atomically, upsert rather than insert, name outputs deterministically (by date/partition), and check-before-create where the sink allows. The failure this prevents is specific and common: attempt one writes 90% of the export and dies; attempt two appends a second 100%; downstream reads 190% and nobody knows which records are real. Retries are the controller doing its job — making them safe is yours.

CronJobs: a schedule plus three decisions

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-export
spec:
  schedule: "15 2 * * *"
  timeZone: "Asia/Tokyo"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 900
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      backoffLimit: 3
      activeDeadlineSeconds: 3600
      ttlSecondsAfterFinished: 86400
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: export
              image: registry.example.com/export:1.4.2
              args: ["--date", "yesterday"]

timeZone — set it, always. Without it the schedule runs in the controller-manager's timezone (effectively UTC on most clusters), and "2:15 a.m." silently means something else after a cluster migration. The field takes an IANA name and handles daylight-saving transitions the way the name implies.
concurrencyPolicy — the overlap decision, made explicitly:
- Allow (the default): runs may stack up. Right for independent, cheap, idempotent work; dangerous when a slow run plus a punctual schedule compounds load.
- Forbid: skip a scheduled run while the previous is still going. The right default for most sequential batch work — exports, backups, syncs.
- Replace: kill the running Job and start fresh. For "latest state wins" work where a stale run is worse than none.
startingDeadlineSeconds — if the controller misses a start (downtime, upgrades), how late a run may still fire. Without it, a controller recovering after an outage can launch a burst of stale runs; with it, missed runs beyond the window are skipped. Roughly 100 missed starts is the controller's own safety cap, but your deadline should be far tighter and deliberate.
History limits — successfulJobsHistoryLimit (default 3) and failedJobsHistoryLimit (default 1) bound how many finished Job objects linger for inspection. Failed history is debugging gold and set lowest by default — raise it if you investigate on Mondays what failed on Fridays.
suspend: true — the pause button. CronJobs honour it immediately; it is the clean way to stop a schedule during an incident without deleting the object.

Failure is a normal state — observe it like one

A service failing pages someone; a Job failing at 2 a.m. writes a status field nobody reads. Close that gap deliberately:

Alert on Job failure and on CronJob staleness. The signals exist in the metrics stack from the observability guide: job-failed conditions, and time since a CronJob's last successful run versus its schedule. "The nightly export hasn't succeeded in 26 hours" is an alert; discovering it from a stale dashboard is an incident review.
Logs outlive pods only if you shipped them. Failed pods are deleted by TTL and history limits by design — the pipeline from the logging guide is the memory. At minimum, make the failure visible where the team already looks (the alert carries the job name; the runbook says where its logs live).
Distinguish "failed" from "never ran." Scheduler outages, suspended CronJobs, image-pull failures before any pod starts — the Job object tells different stories in each case (conditions, events, pod statuses), and the fixes differ. The troubleshooting guide's habit applies: read the object's status before touching anything.

Resource and disruption behaviour for batch

Requests still matter, limits protect neighbours. Batch jobs are the classic noisy neighbour — bursty CPU during everyone else's peak. Set requests honestly (requests-and-limits guide) and consider scheduling heavy batch into off-peak windows or lower-priority classes so services win contention; a report can wait 20 minutes, a checkout cannot.
Disruption budgets apply differently. A Job's pods are not protected by PDBs the way replicas are (the PDB guide covers voluntary disruptions) — node drains during a long Job mean pod termination and a controller retry. Long jobs should checkpoint progress or be restartable from partial state; the idempotency work above is what makes that survivable rather than catastrophic.
One-off manual runs: kubectl create job --from=cronjob/nightly-export export-manual-1 clones the template for an immediate run — the clean pattern for backfills and tests, far better than hand-writing a twin Job manifest that drifts from the scheduled one.

When a Job is the wrong tool

Long-running or never-finishing work is a service — use a Deployment.
Per-node work (agents, collectors) is a DaemonSet's shape, not a Job's.
Complex DAG workflows — fan-out/fan-in pipelines, artefact passing, per-step retries with dependencies — outgrow the Job controller quickly. That is the territory of workflow engines (Argo Workflows and kin); the Job primitives remain the right answer for the large simple majority: one task, a schedule or a trigger, a definition of done.

Common mistakes

restartPolicy: Always copied from a Deployment. The API rejects it for Jobs — usually discovered at apply time in a hurry. The template is not interchangeable.
No deadline. A hung Job holds resources and blocks Forbid-policy successors indefinitely; the schedule looks fine and nothing runs. activeDeadlineSeconds is the difference between a failure and a mystery.
Retries without idempotency. The double-write pattern above; the controller retries by design, so the hazard is guaranteed to fire eventually.
Default backoffLimit and no alerting. Six quiet retries, a Failed status, and silence. Batch observability is an alerting decision, not a dashboard.
CronJobs in the wrong timezone. The 2 a.m. job running at 11 a.m. after a migration is a rite of passage; the timeZone field is how you skip it.
Allow concurrency on heavy work. One slow run plus an hourly schedule equals a self-inflicted load test by morning.
Secrets via environment for one-shot jobs. Jobs end; their specs (and any env values in them) linger in history objects and etcd. The ConfigMaps and Secrets guide's file-based patterns apply with extra force to objects designed to be kept around as records.

Where this fits in the series

Batch is the third workload shape in this series, beside services (Deployments) and per-node agents (DaemonSets): it leans on the requests-and-limits guide for fair scheduling, the storage guide when jobs move real data, the backup guide — whose restores and snapshots are themselves usually Jobs — and the autoscaling guide's cluster-autoscaler behaviour, which treats pending batch pods as the elastic demand they are. Model the done-state honestly and the controller does the rest.

FAQ

Job or CronJob for a task I trigger from CI?
A Job, created by the pipeline when the event happens (a migration after deploy, an export on demand). CronJob is for wall-clock schedules; forcing event-driven work into a polling CronJob adds latency and pointless runs.

Why did my CronJob not run at the expected time?
In order of likelihood: timezone (no timeZone set, cluster in UTC), the previous run still going under Forbid, suspend: true left on, or a missed start beyond startingDeadlineSeconds after controller downtime. The CronJob's status and events record which.

How many times will a failing Job retry?
backoffLimit pod failures (default 6), with exponential backoff between attempts — then the Job is marked Failed and stops. With pod failure policies, specific exit codes can fail it sooner, on the first occurrence.

Can a Job run on a schedule and be triggered manually?
Yes — define the CronJob, and create one-off Jobs from its template with kubectl create job --from=cronjob/<name>. The manual run uses the same spec; it just isn't part of the schedule's history.

Should batch jobs share the cluster with services?
Usually, yes — with honest requests, sane priorities, and schedules that respect peak hours. Dedicated node pools earn their cost when batch is heavy, spiky, or needs special hardware; reach for isolation when contention is measured, not imagined.

Kubernetes RBAC in 2026: Roles, Bindings and Least Privilege That Actually Holds

softverseAdmin — Sat, 31 Oct 2026 11:40:23 +0000

Kubernetes RBAC in 2026: Roles, Bindings and Least Privilege That Actually Holds

TL;DR: Kubernetes RBAC fails in practice not because the model is weak but because grants accrete: a wildcard here, a cluster-admin binding there, a "temporary" service account that becomes permanent. The model itself is simple — subjects get verbs on resources in a scope, grants are additive only, and nothing is denied, only not granted. Least privilege that holds means: one service account per workload with token automount off by default, namespace-scoped Roles as the default unit, groups (not individuals) as human subjects, secrets treated as the crown jewels they are, and a regular audit of who can do what — verified with kubectl auth can-i, not assumed.

Originally published on DevOps Enginer.

Why RBAC is where cluster security is won or lost

The security-basics guide in this series covered the checklist; this guide goes deep on the one control that decides what a compromised credential can actually do. Most cluster incidents follow the same shape: an attacker (or a fat-fingered script) obtains a token — from a pod, a CI job, a leaked kubeconfig — and the blast radius is whatever that token may do. Network policies limit where it can talk; RBAC limits what it can be. A read-only token in one namespace is an embarrassment. A cluster-admin token is a breach report.

And RBAC sprawl is the default outcome, because every grant is a small act of unblocking someone: the deploy pipeline "needs" broader rights, the dashboard "needs" cluster-wide secrets, the new tool "needs" admin "for setup." Each is reasonable on the day. Together they are the incident, pre-positioned.

The model, precisely

Four ideas carry the whole system:

Subjects — who: a User (external identity, typically from a certificate or OIDC token), a Group, or a ServiceAccount (in-cluster identity).
Rules — what: verbs (get, list, watch, create, update, patch, delete, and a few specials) on resources (pods, deployments, secrets…) within API groups ("" for core, apps, batch, rbac.authorization.k8s.io…).
Scope — where: a Role grants within one namespace; a ClusterRole grants cluster-wide or for cluster-scoped resources (nodes, persistent volumes, namespaces themselves).
Bindings — the connection: a RoleBinding attaches a Role or ClusterRole to subjects inside one namespace (a ClusterRole bound by a RoleBinding is effectively trimmed to that namespace); a ClusterRoleBinding attaches a ClusterRole everywhere.

Two properties shape everything else:

Additive only. There are no deny rules. A subject's rights are the union of all its bindings. "Remove the bad grant" always means deleting or narrowing a binding — and "who granted what" questions are answered by enumerating bindings, never by finding an exception.
Default deny. With RBAC as the authorisation mode, a subject with no grants can do nothing. Every capability is an explicit decision — which is why the accretion pattern above is a process failure, not a model failure.

A worked example: read-only log access for support

The support team needs to read pods and logs in one production namespace — nothing else, nowhere else:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: shop-prod
  name: log-reader
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: shop-prod
  name: log-reader-support
subjects:
  - kind: Group
    name: support-engineers
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: log-reader
  apiGroup: rbac.authorization.k8s.io

Details doing quiet work here: the subject is a Group (people join and leave groups; bindings should not name individuals), pods/log is a subresource that must be granted explicitly, and there is no secrets, no create, no second namespace. Verify it the way an auditor would:

kubectl auth can-i get pods --namespace shop-prod --as jane --as-group support-engineers   # yes
kubectl auth can-i get secrets --namespace shop-prod --as jane --as-group support-engineers # no
kubectl auth can-i list pods --all-namespaces --as jane --as-group support-engineers         # no

kubectl auth can-i with --as/--as-group impersonation is the ground-truth test for any identity — build the habit of checking grants with it instead of reading YAML and reasoning hopefully.

Service accounts: the identities that actually get stolen

Humans are rarely the compromised identity; workloads are. Service-account discipline is therefore the highest-value RBAC work:

One service account per workload. The default service account in each namespace should have no grants. A dedicated account per deployment makes grants reviewable and revocation surgical.
Turn off token automount unless needed. Most applications never call the Kubernetes API. For them the token is pure attack surface:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: shop-api
  namespace: shop-prod
automountServiceAccountToken: false

(The same field exists per-pod in the pod spec, which wins when both are set.) Where a token is needed, modern clusters issue short-lived projected (bound) tokens rather than long-lived secrets — prefer them, and treat any long-lived service-account token secret as a credential to inventory.

Deployer accounts are prime targets. A CI service account that can create deployments in production can, by design, run arbitrary images with that namespace's service accounts — near-admin inside its scope. Narrow its verbs to exactly the resources the pipeline applies, keep it namespace-scoped, and protect its credential like a production key (the OIDC guide's keyless pattern applies to how the pipeline authenticates to the cloud; the in-cluster account deserves the same suspicion).

ClusterRoles: handle with both hands

Cluster-wide grants are sometimes right — an observability agent reading node and pod metadata everywhere, an operator managing its resource type across namespaces. The disciplines:

Prefer a trimmed custom role over the built-ins' extremes. The default cluster-admin is total; admin, edit, and view are namespace-oriented defaults with specific shapes (notably, view excludes secrets — a deliberate, instructive choice). For cluster-wide read, write the ClusterRole that lists exactly the resources the tool reads; "cluster-reader" patterns from the community are a starting point, not a destination.
Aggregation composes, so audit the aggregate. A ClusterRole with an aggregationRule collects rules from labelled ClusterRoles (this is how the built-in admin/edit/view roles grow with CRDs). Powerful for platforms, dangerous when a stray label quietly widens a widely bound role — include aggregated roles in audits.
ClusterRoleBindings deserve a list, kept short. Every entry is a cluster-wide statement. Review the full list on a schedule; in a healthy cluster it fits on one screen and every line has an owner who can explain it.

The dangerous verbs and resources

Audits should hunt specific shapes, because damage concentrates:

Wildcards — verbs: ["*"] or resources: ["*"] in anything but a deliberately tiny set of admin roles. A wildcard silently includes tomorrow's new subresources.
get/list on secrets — especially cluster-wide. Reading secrets is effectively holding every credential in scope (the ConfigMaps and Secrets guide covers why). Treat any new secrets grant as a security review, not a config change.
create on pods — pod creation with a chosen service account and mounts is a well-trodden path to privilege escalation; so are create on rolebindings/clusterrolebindings and the special bind and escalate verbs, which exist precisely to gate grant-making itself. impersonate is equally sensitive: it is the other identity, for whoever holds it.
deletecollection and broad delete — the availability side of the same coin.

Human access: groups, short-lived credentials, no shared kubeconfigs

Map your identity provider's groups to Group subjects once, and manage membership in the IdP — not by editing bindings per person. Joiner/leaver processes then work through the system people already trust for access changes.
Issue short-lived credentials: exec-plugin/OIDC flows from your cloud or IdP, so a copied kubeconfig expires instead of opening a permanent door. A long-lived client certificate cannot be revoked (Kubernetes has no certificate revocation for this) — another reason to prefer token-based, expiring auth.
Break-glass, deliberately. Keep an emergency admin path, alarm on its use, and test it — an incident is the wrong time to discover nobody can grant the rights to fix it. Emergency access is a designed feature; routine access through it is the failure.

Auditing: make "who can do what" a query, not a debate

kubectl auth can-i --list --as <identity> prints an identity's effective rights in a namespace — the fastest spot check.
Enumerate bindings as code review: RoleBindings and ClusterRoleBindings are YAML in your GitOps repo (the Argo CD guide's pattern) — changes to them should be pull requests with security-aware reviewers, not kubectl improvisation.
Tooling in the kubectl-who-can / RBAC-lookup family answers the reverse question ("who can get secrets in prod?") across the cluster; run it in CI or on a schedule and diff the output — new entries are events worth a human glance.
API-server audit logs record who did what (the logging guide covers collecting them); RBAC denials in those logs are a signal of both attacks and of teams fighting their own permissions — both worth knowing.

Common mistakes

Binding cluster-admin "temporarily." Temporary grants are permanent until the audit that never happens. Time-box with a ticket, an owner, and a removal date — or better, grant the specific role the task needs.
One service account for everything in the namespace. Every pod inherits the union of rights; one compromised container spends them all.
Secrets access for convenience. Dashboards and debug tools asking for cluster-wide secrets are asking for the keys to everything. Scope to the namespace, or redesign the tool's access.
Confusing RoleBinding-to-ClusterRole scope. Binding a ClusterRole via a RoleBinding grants it only in that namespace — teams regularly believe they granted (or limited) something else. Read the binding kind, not just the role kind.
Stale bindings. People change teams; projects end; vendor tools are uninstalled and their service accounts and bindings remain, valid and forgotten. The audit diff is how they surface.
Testing with your own admin account. It works for you, so permissions "work." Test as the workload's identity (--as=system:serviceaccount:ns:name or by running the pod) before the 3 a.m. discovery.

Where this fits in the series

RBAC is the identity layer the rest of the series assumes: the security-basics guide sets the posture, the network policies guide constrains where identities can reach, the ConfigMaps and Secrets guide protects what grants can expose, and the observability and logging guides watch it all — including the audit stream that records every grant being used. Get RBAC boring, and the other controls get to do their jobs.

FAQ

Can RBAC deny a specific action?
No. RBAC is additive — a subject's permissions are the union of its grants, and there is no deny rule to carve exceptions. If a grant is too broad, narrow the role or split it; for true deny semantics (blocking an action even for the granted), admission-time policy tools are the layer that does it.

Role or ClusterRole for a cross-namespace tool?
If it genuinely acts in many namespaces, a ClusterRole — bound cluster-wide only if it must act everywhere, or via RoleBindings in selected namespaces when its reach should be enumerated namespace by namespace. The second pattern keeps the grant list explicit and reviewable.

Should each pod have its own service account?
Each workload identity should: one per deployment/workload is the practical unit, shared by its replicas. Per-pod accounts multiply objects without adding a trust boundary; per-namespace sharing erases the boundary that exists.

How do I see everything a service account can do?
kubectl auth can-i --list --as=system:serviceaccount:<namespace>:<name> -n <namespace> for a namespace view, repeated per namespace or answered cluster-wide with RBAC lookup tooling. Make it a habit after every binding change.

Are the built-in admin/edit/view roles safe to use?
They are reasonable, maintained defaults — with shapes you should know: view excludes secrets and most write access, edit adds writes but not role management, admin adds role management within the namespace but not namespace deletion or quota changes. The risk is rarely the built-ins; it is the custom wildcard beside them.

Terraform Testing in 2026: Unit Tests With terraform test and Mock Providers

softverseAdmin — Sat, 31 Oct 2026 11:23:47 +0000

Terraform Testing in 2026: Unit Tests With terraform test and Mock Providers

TL;DR: Most Terraform changes are still reviewed by reading a plan and hoping. The native test framework (terraform test, stable since Terraform 1.6) changes that: test files declare inputs, run a plan or apply, and assert on the results — and with mock providers (since 1.7) the logic tests run in CI in seconds with no cloud account and no credentials. Test your modules' contracts — outputs, conditional resources, naming and tagging rules — not the cloud provider's behaviour. Keep a small number of real apply tests for critical modules in an ephemeral sandbox, and let linters and scanners do their separate jobs alongside.

Originally published on DevOps Enginer.

The problem: the plan is not a test

Application code gets unit tests as a matter of course. Infrastructure code — which can delete a database just as dead as any application bug — usually gets a human skimming terraform plan output in a pull request. That review catches typos. It does not catch the conditional that inverted last refactor, the variable default that quietly changed a bucket from private to public, or the count logic that creates zero replicas of the thing production needs three of.

The reason was tooling: for years, testing Terraform meant heavyweight frameworks or applying to a real account and inspecting by hand. The native framework removes the excuse. Tests live next to the module, run with one command, and — the decisive part — can run against mocked providers, so the suite executes like unit tests: fast, hermetic, credential-free.

What is worth testing (and what is not)

Test your module's decisions, not the provider's implementation:

Outputs and contracts. Callers depend on output values and shapes. If bucket_name changes format, downstream code breaks — a test should break first.
Conditional logic. Every count, for_each, and ternary is a decision your module makes: versioning on/off, optional resources created or skipped, single vs multi-AZ shapes. Each branch deserves a run.
Naming, tagging, and policy invariants. The rules platform teams actually care about: names compose from project/environment correctly, required tags are always present, encryption is enabled, nothing is public unless an explicit variable says so.
Variable edge cases. Empty lists, zero counts, nulls — the inputs real callers will eventually pass.

Do not test that AWS creates an S3 bucket when asked, that an attribute AWS generates looks a certain way beyond your control, or other provider internals. Those tests are slow, brittle, and verify someone else's software.

The framework in one module

A small module to test:

# main.tf
variable "project"     { type = string }
variable "environment" { type = string }

variable "enable_versioning" {
  type    = bool
  default = true
}

resource "aws_s3_bucket" "this" {
  bucket = "${var.project}-${var.environment}-assets"
}

resource "aws_s3_bucket_versioning" "this" {
  bucket = aws_s3_bucket.this.id

  versioning_configuration {
    status = var.enable_versioning ? "Enabled" : "Suspended"
  }
}

output "bucket_name" {
  value = aws_s3_bucket.this.bucket
}

And its test file (conventionally in a tests/ directory inside the module):

# tests/bucket.tftest.hcl
mock_provider "aws" {}

variables {
  project     = "shop"
  environment = "prod"
}

run "versioning_on_by_default" {
  command = plan

  assert {
    condition     = aws_s3_bucket_versioning.this.versioning_configuration[0].status == "Enabled"
    error_message = "Versioning must be enabled by default"
  }

  assert {
    condition     = output.bucket_name == "shop-prod-assets"
    error_message = "Bucket name must compose project and environment"
  }
}

run "versioning_can_be_disabled" {
  command = plan

  variables {
    enable_versioning = false
  }

  assert {
    condition     = aws_s3_bucket_versioning.this.versioning_configuration[0].status == "Suspended"
    error_message = "Versioning should suspend when explicitly disabled"
  }
}

Run it with terraform test from the module root. The anatomy:

mock_provider "aws" {} replaces the real provider. Terraform generates fake computed values, so a plan (or even an apply) runs without credentials, network, or a cloud account — in milliseconds.
variables {} at file level sets shared inputs; a run block can override them per case.
run blocks are the test cases. command = plan is the default workhorse; command = apply executes against the (possibly mocked) provider for shape checks that only appear after apply.
assert pairs a condition over any value in the configuration — resources, outputs, variables, locals — with the error_message a human reads when it fails. Write the messages as explanations, not echoes.

Where a mocked computed value needs to be specific — an ARN a policy references, an ID another resource consumes — override_resource and override_data blocks pin those values inside a run. Use them sparingly; a test that overrides everything asserts nothing.

Mock providers: what they change, and their honest limits

Before mocks, a "unit test" of a module still configured the real provider, which meant credentials in CI, provider API calls in every plan, and suites measured in minutes. Mocks invert all three: the pipeline job needs no cloud credentials at all for the test stage (which, per the keyless-CI pattern in the GitHub Actions OIDC guide on this site, is exactly where you want the fewest credentials), tests run at unit-test speed, and nobody's sandbox account accumulates test debris.

The limits are equally clear, and stating them keeps the suite trustworthy:

Mocks validate your configuration logic against the provider's schema, not against the real API. A value the real AWS API would reject (an invalid name pattern, an unsupported combination) can sail through a mocked plan.
Data sources return generated values unless overridden — tests depending on a data source's real content need overrides or a real read.
Provider-side quirks and eventual-consistency behaviour are, by definition, absent.

So the strategy is layered: mocked tests for logic, in bulk; a few real tests for reality, on purpose.

The top of the pyramid: real apply tests, kept few

For a critical module — the network foundation, the shared database module — a mocked plan cannot prove the one thing that matters: it actually builds. Keep a small suite of command = apply runs against the real provider in a sandbox account with three disciplines:

Ephemeral by design. Tests create, assert, and the framework destroys what it created at the end of the run (that is the framework's contract for test-created resources). Verify the destroy happened; orphaned test resources are a cost and security smell — the state-management guide's hygiene rules apply to test state too.
Separate credentials, narrow scope. The sandbox role can create and destroy only the resource types under test, in the sandbox account only. The OIDC guide's per-environment role pattern fits exactly here.
Scheduled, not per-commit. Real applies take minutes and cost real (if small) money. Run them nightly or on module release, while the mocked suite gates every pull request.

Where tests sit among the other tools

Terraform already has a quality neighbourhood, and the test framework does not evict anyone:

Variable validation blocks — the module rejecting bad input at plan time. First line of defence, and the test suite should cover that they fire.
terraform fmt / validate — shape and syntax. Free; run always.
tflint and friends — lint rules, naming conventions, deprecated syntax. Style and hygiene, not behaviour.
Checkov / tfsec-style scanners — policy and security posture of the configuration. They check what the config says; tests check what the module decides given inputs. Complementary, not overlapping.
terraform test — behaviour: outputs, conditionals, invariants across input variations.

A sane pipeline order: fmt check → validate → lint/scan → terraform test (mocked) → plan for human review. Each stage is cheap; together they mean the human reviews intent, not correctness arithmetic.

Modules are where tests pay rent

For a root configuration used once, tests are nice. For modules — shared, versioned, consumed by teams who did not write them — they are the contract enforcement mechanism. The modules guide in this series covers structuring reusable modules; tests are what let a maintainer refactor a module's internals while proving its callers see the same outputs, the same resource decisions, the same invariants. That confidence is precisely what a module library runs on, and it is why test files should ship inside the module repository, versioned with the code they describe.

Common mistakes

Testing the provider. Asserting that a mocked (or real) AWS resource has AWS-chosen properties teaches the suite to fail for other people's changes. Assert your decisions.
Overriding everything. Heavy override_resource use pins the fake world so completely that the test passes regardless of the configuration. Overrides are scalpels for specific computed values, not a substitute world.
One giant run block. A failed assert should name the broken decision. Many small runs with descriptive names beat one mega-run with forty asserts.
Apply tests against shared accounts. Test resources in an account other teams use will eventually collide, linger, or be "cleaned up" by someone mid-run. Sandbox account, always.
Mocked-only confidence for critical paths. The suite is green, the module cannot actually apply. Keep the few real apply tests for the modules whose failure would hurt.
Ignoring test state. terraform test manages its own state for runs, but interrupted real applies can leave residue; a periodic sandbox sweep belongs in the routine.
Writing tests once, at the end. The suite earns its keep when a refactor lands six months later. Add the run block when you add the conditional — that is when the intent is clearest and cheapest to encode.

FAQ

terraform test or Terratest?
They answer different questions. terraform test is native, HCL-based, and ideal for module logic and contracts — no Go toolchain, no extra framework. Terratest (Go) remains strong for end-to-end verification that spans tools and real cloud behaviour beyond one module. Many teams use both: native tests in every module repo, Terratest-style end-to-end checks for platform assemblies.

Does terraform test cost money?
Mocked and plan-only runs create nothing and cost nothing. command = apply runs against a real provider create real resources for the duration of the test — priced accordingly, then destroyed by the framework. Keep them few, small, and in a sandbox.

Can tests run in CI without cloud credentials?
Mocked-provider tests: yes, completely — that is their headline feature. Plan runs against a real provider still need whatever credentials that provider requires to configure itself, which is one more reason the mocked suite should carry the bulk of the assertions.

Where do test files live?
In the module root or a tests/ subdirectory (the conventional choice), as *.tftest.hcl. terraform test discovers them in both places; keep one convention per repository.

What about OpenTofu?
OpenTofu implements the same test framework model — test files, run blocks, and mock providers are compatible concepts, so the approach in this guide carries across. Check your version's documentation for exact feature parity before assuming a specific block behaves identically.

Kubernetes Logging in 2026: From kubectl logs to a Pipeline That Scales

softverseAdmin — Sat, 31 Oct 2026 11:13:41 +0000

Kubernetes Logging in 2026: From kubectl logs to a Pipeline That Scales

TL;DR: In Kubernetes, logs die with the pod — and the pods you most need logs from are the ones that just died. The fix is an architecture, not a tool: applications write structured logs to stdout/stderr, a node-level agent (Fluent Bit, usually) collects and enriches them with Kubernetes metadata, and a backend (Loki, OpenSearch, or your cloud's) stores them with retention you chose on purpose. Add correlation IDs so a log line connects to a request and a trace, control the volume before it controls your budget, and test the pipeline by killing a pod and finding its last words.

Originally published on DevOps Enginer.

The problem: the evidence deletes itself

On a VM, a crashed application's logs sat on disk until someone looked. In Kubernetes, the sequence runs the other way: the pod crashes, gets rescheduled or replaced, the node eventually cleans up, and the logs — the primary evidence of what happened — evaporate on a schedule driven by everything except your investigation.

kubectl logs papers over this for exactly one generation: it shows the current container, and with -p the previous one. Before that? Gone. On another node after a reschedule? Gone. From the DaemonSet that was OOM-killed at 3 a.m.? Never existed as far as you can prove.

So the troubleshooting guide in this series starts with logs — and production logging is the discipline of making sure they are somewhere else before the pod takes them away.

The model: stdout is the application's only logging API

The rule from which everything else follows: a containerised application logs to stdout and stderr, and nowhere else. No log files inside the container, no application-managed rotation, no "the file is on the volume somewhere." The container runtime captures those two streams to files on the node (/var/log/containers/…), and the platform takes it from there.

This buys three things:

Uniformity. Every workload, every language, same contract. The pipeline never needs per-application file paths.
Rotation for free (with a caveat below): the kubelet and runtime rotate the node-side files; the application cannot fill its own container filesystem with logs.
Streams stay separable. stderr is not automatically "errors" — plenty of frameworks log everything to stdout — but having both captured beats parsing one merged file inside a dying container.

The application-side obligations that make this work: one event per line, and structured — JSON lines with a timestamp, a level, a message, and the fields a future searcher will filter on (service, version, request ID). Plain-text prose logs are a tax every incident pays at the worst exchange rate.

The collector: a node-level agent

The standard shape is one log collector per node, running as a DaemonSet, tailing the runtime's log files, enriching each record with Kubernetes metadata (namespace, pod, container, labels — fetched from the API), parsing structure, and shipping to the backend.

The usual choice in 2026 is Fluent Bit: small, fast, CNCF-graduated, written for exactly this job. Fluentd (its older sibling) remains capable, especially where a team already runs its plugin ecosystem; a few platforms ship their own agents. What matters more than the brand:

Metadata enrichment is non-negotiable. A log line without namespace/pod/labels is nearly unsearchable at cluster scale. Verify your pipeline adds them — silently missing metadata is the most common logging bug in production clusters.
Buffering and backpressure. When the backend is unreachable, the agent must buffer (to disk, with a cap) and retry — and its buffer behaviour must not be allowed to fill the node disk it is protecting you from. Size it, alert on it.
The agent is production software. It runs on every node, touches every log line, and fails silently when misconfigured. It belongs in the observability stack's alerting (the observability guide covers the wider stack): DaemonSet pods healthy, shipping lag near zero, drop counters at zero.

A note on audit logs: the API server's audit stream is a different animal — who did what to the cluster, the security record from the security-basics guide — collected at the control plane, retained separately, and never mixed into the application pipeline.

Where logs land: pick the backend by the questions you ask

Loki — indexes only labels, stores compressed log chunks in object storage. Cheap to run, excellent when your queries are "this service, this namespace, this time window, containing this string" — which is most incident queries. Pairs with Grafana next to your metrics. Its discipline: labels are precious and few (service, namespace, level, environment). A user ID or request ID in a label explodes the index (the cardinality trap); those belong in the log line or in structured fields, not labels.
Elasticsearch / OpenSearch — full-text indexing of everything: expensive in storage and operations, unmatched when you genuinely need arbitrary field search across huge volumes and long retention.
Cloud-native sinks — CloudWatch Logs, Google Cloud Logging, Azure Monitor: zero extra infrastructure on the matching managed service, pricing that rewards a retention policy, and query languages your team may like less than LogQL. For many EKS/GKE/AKS clusters this is the pragmatic default, with the node agent either replaced by the provider's or forwarding to it.

Any of the three works. The expensive mistakes are the two defaults nobody chose: keeping everything forever, and discovering your per-GB ingestion bill during the incident that tripled log volume.

The sidecar pattern: when node-level isn't enough

Some applications will not log to stdout — legacy software writing to files, multi-line stack traces an application insists on assembling itself, or a workload needing local pre-processing. The pattern: a sidecar container in the same pod that reads the file from a shared volume and writes it to its own stdout (so the node agent picks it up normally), or ships directly to the backend. It works, at the price of a container per pod — resources, complexity, one more thing to upgrade. Reach for it per-workload, knowingly, not as the cluster default.

Log survival details that bite

Rotation is configured at the node, and it has a size. Runtime/kubelet log rotation (commonly around 10 MiB per file, a handful of files) means a chatty container's history on the node may cover minutes, not hours. That is fine because the agent ships lines as they are written — but it is why "the agent was down for an hour" translates directly into "that hour's logs exist only where the agent later recovered them, if it could." Agent health is log retention.
Node disk is a shared fate. Logs, images, and emptyDir volumes all eat the same disk; a runaway logger can trigger disk-pressure evictions of innocent pods (the storage guide covers the disk side). Resource discipline — requests and limits from that guide — applies to your loggiest workloads first.
kubectl logs still matters. For live debugging it is the fastest tool that exists: kubectl logs -l app=api --since=10m --all-containers=true across a label selector, --previous for the crashed instance, --follow during a rollout. The pipeline is for history and search; kubectl is for now. Engineers should be fluent in both.

Correlation: logs that connect to everything else

A log stream alone answers "what did this service print." Production debugging asks a bigger question: "what happened to this request." The bridge is a correlation ID: generated at the edge (or taken from an incoming header), propagated through every service call, and printed in every log line and trace span. With that one discipline, an incident moves from grepping five services by timestamp to one query per system, all keyed by the same ID — and the metrics–traces–logs triad from the observability guide becomes a single investigation instead of three.

Two supporting habits:

Levels with meaning. ERROR should mean "a human may need to act," not "a retry happened." When everything is an error, nothing is; alert on error rate changes, not on the existence of error lines.
Never log secrets or personal data. Tokens, passwords, full request bodies with credentials, unnecessary personal fields — logs are copied, retained, and widely readable by design; treat them as the least-trusted storage you own. Redaction at the application beats scrubbing in the pipeline, and both beat explaining a credential found in a log bucket (the ConfigMaps and Secrets guide covers the secret-handling side).

Volume, retention, and the bill

Log volume scales with traffic and verbosity, and both spike during incidents — exactly when you can least afford dropped data or a surprise invoice. The controls, in the order teams usually meet them:

Retention tiers. Hot storage (fast search) for days-to-weeks; cheaper object storage for months; deletion after the period your compliance actually requires. "Forever" is a decision too — an expensive one nobody remembers making.
Drop or sample the noise. Health-check access logs from internal probes, debug output from one misbehaving workload, a namespace that logs every cache hit — filter at the agent, where dropping is free, rather than at the backend, where you have already paid to ingest.
Per-namespace visibility. Know which workloads produce the volume. The cost guide in this series makes the same point about compute; logs deserve the same per-team accounting, because the fix is always a conversation with one workload's verbosity.

Prove it: the kill-a-pod test

Like backups and disruption budgets before it in this series, a logging pipeline is a rumour until exercised. The test is five minutes:

Pick a workload. In its logs, find (or generate) a unique marker line — a distinctive request ID works.
Delete the pod. Wait for the node to forget it ever existed.
From the backend alone, retrieve that pod's final log lines, filtered by namespace, pod name, and the marker. Check the metadata is attached and the timestamps are ordered and correct.

If step 3 fails, you do not have a logging pipeline — you have a logging intention. Run the test after every agent upgrade and every backend change, and monitor the agent continuously in between; silent gaps are this system's natural failure mode.

Common mistakes

Applications logging to files inside containers. The data is invisible to the platform, dies with the pod, and fills the container filesystem on the way out.
Unstructured prose logs. Every incident becomes a regex-writing contest. JSON lines with stable fields, from day one.
High-cardinality labels. Request IDs, user IDs, and pod IPs as index labels: the backend slows, costs climb, and the fix (re-indexing your mental model) is painful. Labels identify the stream; fields describe the event.
No one owns agent health. The DaemonSet falls behind or dies on two nodes; the gap is discovered mid-incident, which is the only time it cannot be fixed.
Debug level in production, forever, because one incident needed it once. Verbosity is a per-workload dial with a cost meter attached.
Secrets in logs. The widest-read storage in your stack is the worst place for the most sensitive strings you own.
Retention set by the tool's default instead of by your actual compliance and debugging needs — too short loses investigations, too long buys nothing but storage bills.

Where this fits in the series

Logging is the third pillar the observability guide stands on, and the raw material the troubleshooting guide consumes first. It leans on the storage guide's disk discipline, the requests-and-limits guide's resource accounting, the network policies guide's protection for the shipping path, and the security-basics guide's audit and redaction rules. Get it right and it disappears — every other guide in this series quietly assumes the logs were there when needed.

FAQ

Fluent Bit or Fluentd?
For a new cluster pipeline, Fluent Bit: lighter, faster, and the current centre of gravity for Kubernetes log collection. Fluentd remains reasonable where its plugin ecosystem or an existing deployment carries weight. The architecture matters more than the choice.

Loki or Elasticsearch?
If your queries are service-and-time-window shaped (most incident work), Loki's label model is dramatically cheaper to run. If you need deep, arbitrary full-text analytics over long retention, Elasticsearch/OpenSearch earns its cost. Many clusters start on their cloud provider's sink and graduate to one of these when query needs or bills demand it.

Should the application ever ship logs directly to the backend?
Almost never. It couples application deploys to logging infrastructure, duplicates shipping logic in every service, and loses the node agent's uniform metadata and buffering. The stdout contract exists precisely to avoid this.

How long should logs be retained?
Long enough to cover your slowest investigation and your compliance floor — commonly 2–4 weeks hot and a few months in cheap archive for application logs, longer for audit logs under policy. Decide per class of log, write it down, and let the tiers enforce it automatically.

Do managed services (EKS, GKE, AKS) solve this for me?
They provide a destination (CloudWatch, Cloud Logging, Azure Monitor) and sometimes an agent option, but the architecture decisions remain yours: stdout discipline, metadata, correlation IDs, retention, and cost control. The provider removes the backend operations, not the design.

GitHub Actions to AWS Without Stored Keys: OIDC Federation in 2026

softverseAdmin — Sat, 31 Oct 2026 11:00:10 +0000

GitHub Actions to AWS Without Stored Keys: OIDC Federation in 2026

TL;DR: If your GitHub Actions workflows authenticate to AWS with a stored access key, that key is a permanent credential sitting in a second system, waiting to leak and rarely rotated. OIDC federation removes it entirely: each workflow run gets a short-lived token from GitHub, presents it to AWS, and assumes an IAM role scoped to exactly that repository, branch, or environment. Credentials live for about an hour and are never stored anywhere. Setup is one IAM identity provider, one role with a careful trust policy, and two lines in the workflow. This guide walks the whole thing, including the scoping mistakes that quietly undo the benefit.

Originally published on DevOps Enginer.

The problem: a permanent key in someone else's house

The classic setup is familiar: create an IAM user for CI, generate an access key pair, paste them into GitHub repository secrets, and reference them in the workflow. It works on day one. Then the years happen:

The key never rotates, because rotating it means touching every repo that copied it — and nobody remembers all of them.
The key outlives its purpose: the project is archived, the repo is forked into a test, the contractor who set it up left. The credential remains valid through all of it.
The key's permissions drift upward: it was scoped for one deployment, then someone needed one more action, then it quietly became near-admin because that was faster.
The key leaks like a secret, not like a session: pasted into a log, a debug dump, a screenshot. When a session token leaks you have an hour of exposure; when this leaks you have until someone notices and rotates — if anyone notices.

The deeper issue is architectural: a long-lived credential is a standing invitation. OIDC federation deletes the standing part. There is nothing to steal from the repository because the repository holds nothing — the workflow proves who it is at runtime instead.

How it works, in one paragraph

GitHub runs an OIDC identity provider. Every workflow run can request a signed JSON Web Token whose claims describe that run: the repository (subject claims like repo:my-org/my-repo:ref:refs/heads/main), the branch or tag, the environment, the workflow name. In AWS, you register GitHub as an IAM OIDC provider once per account, then create an IAM role whose trust policy says: "tokens from GitHub's provider may assume me — but only if the claims match these exact conditions." The workflow exchanges its token via AssumeRoleWithWebIdentity and receives ordinary short-lived AWS credentials (default lifetime one hour). AWS validates the token's signature against GitHub's published keys and checks the audience claim (sts.amazonaws.com) and your conditions. No shared secret exists on either side.

Step 1: register GitHub as an OIDC provider in IAM

One time per AWS account (Console: IAM → Identity providers → Add provider → OpenID Connect):

Provider URL: https://token.actions.githubusercontent.com
Audience: sts.amazonaws.com

That is the entire provider setup. Everything else lives on the role.

Step 2: the role and its trust policy — where the security actually is

Create an IAM role for web identity federation. The trust policy is the security boundary of this whole design, so write it deliberately:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:my-org/my-repo:ref:refs/heads/main"
        }
      }
    }
  ]
}

What each part earns:

The aud condition confirms the token was minted for AWS STS, not for some other service that also trusts GitHub tokens.
The sub condition is the scoping decision. repo:my-org/my-repo:ref:refs/heads/main means: only runs of this repo, on the main branch. Alternatives you will actually use:
- repo:my-org/my-repo:environment:production — only runs that deploy to the GitHub environment named production (which can itself require manual approval — a strong pattern for prod).
- repo:my-org/my-repo:ref:refs/tags/v* with StringLike — only tag pushes, for release roles.
Never a wildcard like repo:my-org/* on a powerful role, and never omit the sub condition. "Any workflow in any repo I own can assume the production deploy role" is the stored-key problem wearing a new coat.

Then attach the role's permissions policy — the least-privilege part. Scope it to what the pipeline does: update one ECS service, write to one S3 bucket, publish to one ECR repository. The AWS IAM least-privilege guide on this site covers how to build such policies without guessing; the OIDC work here assumes that discipline rather than replacing it.

One role per (repository, environment) pair is the pattern that scales: gha-myrepo-prod, gha-myrepo-staging, each boringly narrow.

Step 3: the workflow side

Two requirements, both small:

name: deploy

on:
  push:
    branches: [main]

permissions:
  id-token: write      # without this, no token is issued
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gha-myrepo-prod
          aws-region: ap-northeast-1

      - name: Deploy
        run: |
          aws s3 sync ./dist s3://my-prod-site-bucket --delete

Notes that save debugging time:

permissions: id-token: write is the step everyone forgets. Without it the token request fails before AWS is even involved. Setting any permissions block also means everything not listed drops to none — hence contents: read for checkout.
There are no secrets to configure: AWS_ACCESS_KEY_ID and friends disappear from the repo settings entirely.
The configure action exports the short-lived credentials to subsequent steps; tools that read the standard AWS credential chain (the AWS CLI, Terraform, SDKs) work unchanged.

Scoping patterns that matter

Environments are the strongest lever. GitHub environments can require a human approval before a job runs, and the token's claims carry the environment name. Combine them: the production role trusts only environment:production, so even a maintainer pushing to main cannot reach prod credentials until the environment's approval gate passes. Branch protection plus environment approval plus a scoped role is about as strong as CI identity gets without extra tooling.

Pull requests are a different animal. A pull_request run's claims do not look like a branch push — and for forks, the workflow runs in a context you should treat as untrusted. Practical rule: give PR workflows a read-only role if they need AWS at all (fetching artifacts, running plan), and never let a PR-triggered run assume a role that can write to production. terraform plan on PRs with a read-only role, apply only from main with the production role, is the canonical split — it pairs naturally with the Terraform state setup from this site's state-management guide.

Multiple accounts, one pattern. Staging and production in separate AWS accounts is common; OIDC handles it cleanly because the provider registration and role live per account. The same workflow can assume the staging role in the staging account and the production role in the production account, each trust policy scoped to its own environment claim.

Reusable workflows inherit the caller's identity carefully. If you use reusable workflows (this site's guide covers them), the token claims describe the caller repository and context — which is what you want, but it means a shared deploy workflow used by ten repos needs trust-policy entries or separate roles per caller repo. Plan for that instead of discovering it as a wildcard temptation.

Migrating off stored keys without breaking Friday's deploy

Inventory. Search your repos' secrets and workflows for AWS_ACCESS_KEY_ID. Note which IAM user each key belongs to and what it can do — this list is usually longer and scarier than expected, which is the argument for the project.
Create the provider and roles, one repo/environment at a time, permissions mirroring what the key was supposed to do (not what it accidentally can).
Switch the workflow to the OIDC pattern and deploy. Verify in CloudTrail that calls now arrive as the assumed role session, not the old user.
Watch, then delete. Leave the old key deactivated (not deleted) for a short window while you confirm nothing else uses it — scheduled jobs and forgotten repos surface quickly when a key stops working. Then delete the key, and if the CI IAM user has no remaining purpose, delete the user.
Sweep the leftovers: organisation-level secrets, third-party tools holding the same keys, and local copies in developer tooling.

Order matters less than completeness: a migration that moves nine repos and forgets the tenth has kept exactly the risk it meant to remove.

Common mistakes

Wildcard sub conditions. repo:my-org/* or a missing sub check turns one role into every repo's master key. The trust policy is the product; write it narrowly.
One role for everything. A single powerful role assumed by every workflow recreates the shared-key blast radius with better marketing. Role per repo per environment.
Forgetting id-token: write. The failure appears before AWS is contacted and confuses everyone who starts debugging in IAM.
Leaving the old keys active "just in case". The fallback is the vulnerability. Deactivate, watch, delete.
Granting write roles to pull-request contexts. Plan on PRs, apply from protected branches and environments — never both from the same trigger.
Skipping the audience check. The aud condition costs one line and prevents tokens minted for other purposes from being replayed at STS.
Assuming OIDC fixes permissions. Federation decides who may assume the role, not what the role may do. An admin role via OIDC is still an admin role; least privilege is a separate, continuing discipline.

FAQ

Does this cost anything?
No. OIDC providers and roles are free IAM features, and STS calls for this pattern are not a line item you will ever notice.

What if a workflow needs to run longer than the token lifetime?
Role sessions default to one hour and can be configured up to the role's maximum session duration (up to 12 hours). Design for re-assumption or shorter jobs instead of reaching for the maximum by default — long-lived sessions quietly reintroduce the exposure you removed.

Can self-hosted runners use this?
Yes — runners you host still request the token from GitHub's OIDC endpoint at runtime, so the same trust policy works. (GitHub Enterprise Server deployments use their own provider URL; check your instance's docs for its exact value.)

Does Terraform in GitHub Actions work with OIDC credentials?
Yes. The configure-aws-credentials step exports standard environment credentials, and the Terraform AWS provider consumes them through its normal credential chain — no provider block changes needed. Remote state locking and the backend setup stay exactly as they were.

What about other clouds?
The same pattern exists across the major providers — Azure and Google Cloud both accept GitHub-issued OIDC tokens for workload identity federation with no stored secrets. Once a team learns the shape on one cloud, the others are variations, and the destination of all of them is the same: CI with no permanent keys anywhere.

Kubernetes Pod Disruption Budgets in 2026: Graceful Shutdown Without Dropped Requests

softverseAdmin — Sat, 31 Oct 2026 10:51:44 +0000

Kubernetes Pod Disruption Budgets in 2026: Graceful Shutdown Without Dropped Requests

TL;DR: Kubernetes constantly disrupts your pods on purpose — node drains, cluster upgrades, autoscaler scale-downs, rollouts. Pod Disruption Budgets (PDBs) are how you negotiate those disruptions: "you may take one of my three replicas at a time, never two." But a PDB only covers the eviction handshake. The other half of graceful shutdown is the pod itself: handling SIGTERM, finishing in-flight work, and leaving the load balancer before the SIGKILL lands. Get both halves right and maintenance stops being an outage source; get either wrong and every upgrade is a small incident.

Originally published on DevOps Enginer.

The problem: most pod deaths are voluntary

When people imagine pods dying, they imagine failures — a node dying, an OOM kill, a crash. Those are involuntary disruptions, and Kubernetes handles them the only way it can: after the fact, by starting replacements.

But watch a healthy cluster for a week and most pod terminations are voluntary: an admin drains a node for maintenance, the cluster autoscaler removes an underused node, an upgrade rolls the nodes, a deployment replaces pods, the cluster itself gets upgraded version by version (the upgrades guide in this series is essentially a long, careful series of voluntary disruptions). Nothing failed — the platform is doing its job.

The trouble is the intersection: a drain that evicts all three replicas of your API in the same minute because nobody told it not to. A pod killed mid-request because it never got the memo — or got it and ignored it. Voluntary disruptions are scheduled, polite, and negotiable. You just have to actually negotiate.

Pod Disruption Budgets: the negotiation

A PDB is a small object that tells the eviction API how many pods of a workload may be voluntarily taken down at once:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
  namespace: shop
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api

This says: of the pods labelled app: api in shop, at least 2 must stay available; evictions that would break that are refused with an error, and the drain waits. Two ways to express the budget:

minAvailable — an absolute number (2) or a percentage (67%) of desired replicas that must remain up.
maxUnavailable — how many may be down at once (1, or 34%). Usually the better choice, because it tracks replica count: if you scale from 3 to 6 replicas, "at most 1 down" still means what you meant, while a stale minAvailable: 2 silently got weaker.

Rounding is asymmetric and worth knowing: percentages for minAvailable round up (67% of 3 = 3, not 2), and for maxUnavailable round down (34% of 3 = 1). If your PDB seems stricter than the percentage suggests, this is why.

How the handshake works in practice: kubectl drain, cluster upgrades, and well-behaved controllers evict pods through the eviction API, which checks the PDB first. If the budget allows it, the pod gets its termination grace period and shuts down; if not, the eviction is rejected and the tool retries later. Tools that delete pods directly (a raw kubectl delete pod) bypass PDBs entirely — a PDB is a contract with polite tooling, not a force field.

The trap: a PDB that can never be satisfied

Set minAvailable equal to your replica count — a very common "extra safe" instinct — and you have built a deadlock, not a safeguard. The node can never drain, because draining requires evicting a pod, and the budget forbids it. Cluster upgrades stall, autoscaler scale-downs stall, and someone eventually force-deletes the pod at the worst possible time, defeating the entire exercise under pressure.

The subtler version: the budget is satisfiable on paper, but the workload is unhealthy. If only 1 of 3 replicas is ready, a minAvailable: 2 PDB blocks all evictions — including evicting the broken pods — because the budget counts healthy pods, and there is no slack. Recent Kubernetes versions address this with spec.unhealthyPodEvictionPolicy (set it to AlwaysAllow so unhealthy pods can always be evicted; the default IfHealthyBudget is stricter). A blocked drain is information: it usually means the application was already below its own availability floor before maintenance started.

A working rule: budget for one disruption at a time. maxUnavailable: 1 for replicated stateless services is the sensible default — drains proceed pod by pod, replacements come up before the next eviction is allowed, and upgrades stay boring.

The other half: what the pod does when told to die

A PDB buys an orderly sequence. It does not make an individual shutdown graceful. That part happens inside the pod, on a clock:

The pod is marked Terminating. It is removed from Service endpoints — mostly. Endpoint removal and the kube-proxy / ingress updates that follow are fast but not instantaneous, so a terminating pod can receive a trickle of new connections for a short window. This race is normal and known.
Any preStop lifecycle hook runs.
The container receives SIGTERM. This is the application's cue: stop accepting new work, finish in-flight requests, flush and close.
After terminationGracePeriodSeconds (default 30), if the process is still alive, it receives SIGKILL — no negotiation, mid-whatever.

Graceful shutdown is an application feature, not a platform one. The checklist:

Handle SIGTERM in the process. Many frameworks do this out of the box now (Spring Boot's graceful shutdown, Go's signal.NotifyContext, Node's server close on signal) — but verify yours does, and that the handler drains: stop taking requests, wait for in-flight ones, close database connections cleanly.
Size the grace period to the work. A web API that finishes requests in 2 seconds is fine at 30s. A worker with 5-minute jobs at the default 30s grace period gets SIGKILLed mid-job every rollout — set terminationGracePeriodSeconds to match the longest unit of work you intend to finish, and make the work checkpoint or idempotent for the cases where it still gets cut.
The preStop sleep, honestly labelled. A common pattern is a preStop hook that sleeps a few seconds, giving endpoint removal time to propagate before SIGTERM makes the app stop listening. It looks like a hack because it is one — but it addresses the real endpoint race in step 1, and it prevents the classic "502s during every deploy" bug. Use it knowingly, keep it short.
Workers and queues: on SIGTERM, stop consuming, finish the current message, then exit — and make sure unprocessed messages return to the queue (visibility timeout / ack discipline) rather than dying with the pod.

Stateful and single-replica workloads: where PDBs can't save you

PDBs assume redundancy. Apply the arithmetic to a single-replica workload and minAvailable: 1 (or maxUnavailable: 0) means "never voluntarily evict this pod, ever" — the drain deadlocks on your one database pod, and the eventual forced eviction takes the downtime anyway, unscheduled. For singletons, the honest choices are:

Accept the disruption window: no PDB (or maxUnavailable: 1), maintenance takes the pod down briefly, planned and announced. Better a scheduled 60-second blip than a blocked upgrade escalating into an unplanned one.
Make it not a singleton if the availability requirement is real — which for databases means the database's own replication story, not a Kubernetes trick.

For StatefulSets, PDBs work normally (they select pods by label like anything else), but pair them with the storage and backup thinking from earlier in this series: an evicted stateful pod comes back to its own volume, so the shutdown risk is about in-flight writes — the application-level SIGTERM handling matters more, not less.

One more placement rule: do not put PDBs on batch Jobs. A Job's pods are supposed to terminate; a budget on them protects nothing and can wedge completions and node drains alike. PDBs are for long-running services whose absence a user would notice.

Proving it: test the drain before the upgrade does

Like every control in this series, a PDB is a rumour until exercised. The test fits in a staging afternoon:

Run a light, continuous request loop against the service (a small script recording status codes is enough).
Cordon and drain a node running the workload: kubectl cordon <node> then kubectl drain <node> --ignore-daemonsets.
Watch: evictions should proceed one at a time, the replacement should be Ready before the next eviction is allowed, and the request loop should show zero — or near-zero — failed requests. Failed requests with graceful PDB behaviour usually point at the application half: SIGTERM ignored, grace period too short, or the endpoint race.
Then test the blocked path deliberately: scale the service down to its floor and try another eviction. Being refused, and seeing the clear PDB error, is the control working. An upgrade night is the wrong time to learn this behaviour for the first time.

The cluster autoscaler respects the same handshake when it scales nodes down (covered in the autoscaling guide in this series) — which is why a too-strict PDB shows up twice in a cluster's life: once blocking your upgrade, once blocking your cost savings.

Common mistakes

minAvailable equal to the replica count. The deadlock special. Budget one disruption at a time.
Percentages that don't mean what you think. minAvailable: 50% of 3 replicas is 2 pods (it rounds up) — and of 1 replica it is a permanent eviction ban.
Expecting PDBs to cover failures. They govern voluntary evictions only. Node death, crashes and OOM kills are answered by replicas, topology spread and anti-affinity — the deployment-strategies guide covers that side.
Default 30-second grace for slow work. Long requests, report generation, queue jobs: if the work takes minutes, the grace period must too, or every rollout is a small data-loss event.
Ignoring SIGTERM "because Kubernetes restarts us anyway". It does — elsewhere. The in-flight request that died with the old pod does not get restarted; the client just sees an error.
PDBs on everything, including Jobs and singletons. Indiscriminate budgets create blocked operations across the cluster and train the team to force-delete — the worst possible outcome.
Never alerting on blocked evictions. A drain stuck for hours is a signal the app is below its availability floor. It belongs in the observability stack (the observability guide), not in tomorrow's surprise.

Where this fits in the series

PDBs are the politeness protocol that several earlier guides quietly assume: the upgrades guide's node-by-node dance only works when evictions are budgeted, the autoscaling guide's cluster autoscaler asks PDB permission before removing nodes, Argo Rollouts and the deployment-strategies guide manage the same risk from the release side, and the backup guide is the admission that even perfect politeness has limits. Voluntary disruption done well is invisible — which is precisely the point of it.

FAQ

Does a Pod Disruption Budget protect against node failure?
No. PDBs only constrain voluntary evictions through the eviction API. Involuntary loss — node death, kernel panic, a cloud reclaiming the instance — ignores budgets entirely. Redundancy (replicas, topology spread constraints, anti-affinity) is the tool for that side.

Should I use minAvailable or maxUnavailable?
maxUnavailable in most cases, typically 1. It states the disruption you tolerate rather than a floor that silently weakens or deadlocks as replica counts change. Reserve minAvailable for workloads whose floor is a hard contractual number.

Can a PDB block an upgrade forever?
Yes, if the budget can never be met — either by construction (floor equals replicas) or because the workload is too unhealthy to have slack. That is the PDB doing its stated job; the fix is to repair the workload's health or consciously relax the budget, not to force-delete pods under pressure.

Do Deployments create PDBs automatically?
No. A Deployment manages replicas; the budget is a separate object you write, select, and own. Some Helm charts ship one (check before assuming — the value may not match your replica count), and the Helm guide in this series shows where such templates live.

What happens to in-flight requests when a pod is evicted?
The pod leaves the Service endpoints, gets SIGTERM, and has its grace period to finish. Requests still in flight when the grace period expires die with the SIGKILL. Both halves of this guide exist to shrink that window to zero: PDBs sequence the evictions, and the application's shutdown handling empties the pod before the deadline.

Kubernetes Backup and Disaster Recovery in 2026: Backups You Have Actually Restored

softverseAdmin — Sat, 31 Oct 2026 10:43:19 +0000

Kubernetes Backup and Disaster Recovery in 2026: Backups You Have Actually Restored

TL;DR: High availability is not backup. A three-replica Deployment survives a node failure, but kubectl delete namespace deletes all three replicas just as happily, and no amount of HA brings back a dropped database. Real protection has three layers — cluster resources, persistent volume data, and the things that live outside the cluster — plus a restore you have actually performed and timed. Velero is the standard open-source tool for the first two layers. The work is not installing it; the work is proving, on a schedule, that a restore produces a working application.

Originally published on DevOps Enginer.

The problem: Kubernetes feels safer than it is

Kubernetes heals. Kill a pod and it comes back. Lose a node and workloads reschedule. This self-healing is real — and it quietly convinces teams they are protected against data loss, which is a different thing entirely.

Self-healing protects against failures of the machinery. Backups protect against everything else: an engineer deleting the wrong namespace, a bad migration dropping a table, a compromised account wiping resources, a storage failure corrupting a volume, a cloud account problem taking the whole region's control plane with it. In every one of those cases, the cluster faithfully heals itself into the new, broken state. The disaster is not that Kubernetes stops working — it is that it keeps working, on empty data.

The second misconception: "etcd is backed up, so we are covered." On a self-managed cluster, an etcd snapshot captures cluster state — the resource definitions. It does not contain the bytes inside your PersistentVolumes, and restoring it is a cluster surgery procedure, not an application recovery. On managed services (EKS, GKE, AKS) you cannot snapshot etcd at all — the provider runs it — which settles the question: your backup strategy lives at the resource and volume level whether you chose it or not.

What actually needs protecting: three layers

Cluster resources. Deployments, Services, ConfigMaps, Secrets, RBAC, NetworkPolicies — the declarative state. Cheap to back up (it is text in etcd), and the layer GitOps already half-covers: if everything is applied from Git, Git is a backup of intent, though not of generated state like Secrets created by operators or resources created by controllers.
Persistent data. The contents of PersistentVolumes: databases, uploaded files, queues on disk. This is the layer that hurts to lose and the layer most "we ran a backup tool once" setups quietly miss, because resource backup and volume backup are separate switches.
Outside the cluster. Container images (a registry that garbage-collects the tag you need is a DR failure), external databases and queues, DNS, certificates, and the cloud resources the cluster depends on — IAM roles, load balancer config, the object storage bucket your backups land in. A restore plan that discovers a missing layer at 3 a.m. is not a plan.

Backups also need their own threat model: they must survive the loss of the cluster and the loss of the account or credentials that administer it. Backup data stored only in the same account, deletable by the same compromised credential, is a copy, not a backup. Cross-account or cross-region copies with separate retention are the grown-up version.

Two numbers that decide the design: RPO and RTO

Before tooling, two definitions do all the real work:

RPO (recovery point objective): how much data you can afford to lose, measured in time. Hourly backups mean up to an hour of data loss. If the business answer is "five minutes", nightly volume snapshots are the wrong tool no matter how reliable they are — you need replication or database-native continuous backup.
RTO (recovery time objective): how long the business tolerates being down while you restore. This is the number restore tests measure, and it includes everything: provisioning a cluster, restoring resources, restoring data, DNS, verification. Teams routinely discover their restore takes six hours against a one-hour RTO — better discovered in a drill than in an incident.

Write both numbers down per application tier. They turn backup arguments ("should we also snapshot the volumes?") into arithmetic.

Layer 1 + 2 with Velero: the working setup

Velero is the standard open-source tool for Kubernetes backup: it copies cluster resources to object storage and, when asked, snapshots or copies the data inside PersistentVolumes. Two facts shape every Velero setup: backups land in object storage outside the cluster (S3, GCS, Azure Blob, or S3-compatible), and volume data needs an explicit choice — it is not automatic.

Install against your provider's plugin (AWS shown; bucket and credentials are placeholders):

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws \
  --bucket my-cluster-backups \
  --secret-file ./credentials-velero \
  --backup-location-config region=ap-northeast-1 \
  --use-node-agent

--use-node-agent deploys the agent that performs file-system-level volume backups — needed for the second volume option below.

There are two ways Velero captures volume data:

CSI snapshots — the storage driver snapshots the volume (on EKS, an EBS snapshot). Fast, storage-native, and the right default where the CSI driver supports snapshots. Since Velero 1.15 the CSI integration is built in rather than a separate plugin.
File-system backup — Velero copies the files out of the volume through the node agent (Kopia under the hood in current releases). Slower, but works on storage with no snapshot support and produces data that lives in the same object storage as the resource backup, which simplifies cross-cluster restores.

A scheduled backup of one namespace, with volume snapshots, is one object:

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: shop-daily
  namespace: velero
spec:
  schedule: "0 2 * * *"
  template:
    includedNamespaces:
      - shop
    snapshotVolumes: true
    ttl: 720h

And the manual equivalents you will use in drills:

velero backup create shop-drill --include-namespaces shop --snapshot-volumes --wait
velero restore create --from-backup shop-drill --namespace-mappings shop:shop-restore
velero backup describe shop-drill      # check phase: Completed, and read the warnings

Restoring into a different namespace (the --namespace-mappings above) is the drill pattern: the original workload keeps running while you prove the copy works.

The consistency problem: volumes are crash copies

A volume snapshot taken under a running database is a crash-consistent copy — equivalent to pulling the plug. Most databases recover from that on startup, which is why the pattern usually works, but "usually" is doing heavy lifting. For the workloads where consistency matters, you have three honest options:

Backup hooks. Velero can run a command in a pod before or after the snapshot (annotations on the pod template) — the classic pattern is freezing or flushing the database (for example fsfreeze or a database-specific lock) for the seconds the snapshot takes.
Database-native backup. For a critical PostgreSQL or MySQL, the serious answer is often the database's own tooling — WAL archiving / point-in-time recovery — with Velero protecting the manifests and configuration around it. Point-in-time recovery also beats any snapshot schedule on RPO.
Quiesce the workload. Scale to zero, snapshot, scale back. Only acceptable where a maintenance window exists — but for small internal systems it is the simplest correct answer.

Pick per workload, and write the choice down next to the RPO it achieves. The failure mode is assuming the default (crash-consistent) covers the database it does not.

The GitOps shortcut, and its limits

If your manifests live in Git and Argo CD (or equivalent) applies them, layer 1 mostly restores itself: point a new cluster at the repo and the workloads reappear. This is a genuine DR advantage of GitOps — the Argo CD guide in this series covers the setup. But know exactly what Git does not contain: Secrets (usually stored encrypted or externally — your restore needs the key material, e.g. the Sealed Secrets private key, or the external secret store), data created inside the cluster, and resources generated by operators. The standard production answer is both: GitOps for the declarative layer, Velero for the generated state and the volume data.

Prove it: restore drills are the actual product

Every guide in this series lands on the same principle — an untested control is a rumour — and backups are where it matters most, because backup failures are silent. Velero will happily report schedules running while the bucket policy expired months ago. The only verification that counts is a restore.

A workable drill rhythm:

Monthly: restore last night's backup of one real namespace into a scratch namespace on the same cluster. Verify the application answers, data row counts match expectations, and record how long it took — that number is your honest RTO for that tier.
Quarterly: restore into a different cluster (the migration path). This catches the things intra-cluster restores never touch: missing CRDs, storage class names that differ, image pull secrets, ingress differences.
Always: alert on backup failure, not just on schedule age. Wire backup status into the same observability stack as everything else — the observability guide in this series is where the backup metrics and alerts belong.

Notice what the drill produces beyond safety: a measured restore time, a list of missing pieces fixed while nothing was on fire, and a team that has performed the procedure before the night it counts.

Common mistakes

Backing up resources and calling volumes done. snapshotVolumes off (or no file-system backup configured) means your backup is the recipe, not the meal. Check a backup's contents, not just its green status.
Backups in the blast radius. Same account, same credentials, no versioning or retention lock on the bucket: one compromise deletes production and its backups in the same session.
No TTL / no cost control. Snapshots accumulate forever; the storage guide in this series covers the orphaned-snapshot problem from the cost side. Every schedule needs a ttl.
Secrets that cannot be restored. The backup contains the Sealed Secrets resources but the cluster that sealed them is the one that died. Keep the unsealing keys in the DR plan, stored outside the cluster.
Restoring everything at once. A full-cluster restore recreates namespaces in dependency order whether you plan it or not — and load balancers, ingress and DNS fight you on the way. Restore per-application, most critical first; that ordering is the RTO plan.
One backup tool for the database it cannot help. Crash-consistent volume copies of a busy database, with no hooks and no native backup, presented as a DR strategy.
Testing backups by reading their status. Green "Completed" is Velero's opinion. The restore is the test.

Where this fits in the series

Backup is the layer underneath several earlier guides: the storage guide explains the volumes being protected, the upgrades guide's blue/green cluster pattern is a restore-shaped procedure (and the quarterly cross-cluster drill doubles as upgrade rehearsal), the observability guide is where backup success and failure get monitored, and the network policies and security-basics guides cover the access control that keeps backup credentials and buckets out of an attacker's reach. Recovery is not a feature you add at the end; it is the property the rest of the stack is for.

FAQ

Does Velero back up etcd?
No — deliberately. Velero works at the API level, copying resources and volume data; etcd snapshots are a separate, control-plane procedure for self-managed clusters. Velero's model also restores into a cluster running a different Kubernetes version, which an etcd restore cannot do.

We run GitOps. Do we still need Velero?
Yes, for the parts Git cannot see: volume data, generated resources, and in-cluster state. GitOps shrinks the restore problem; it does not remove it.

How often should backups run?
As often as your RPO demands, minus honesty about restore cost: a common shape is daily schedules with a 30-day TTL for general namespaces, more frequent (or database-native continuous backup) for the tier whose RPO is measured in minutes. More schedules than that mostly buy storage cost.

Is a second cluster in another region enough — active-active instead of backup?
Replication solves some RPO problems and no deletion problems: a replicated kubectl delete is a deletion in two regions. Multi-cluster setups still need backups, and most teams should get backups plus practiced restores working before paying for a second always-on cluster.

Can Velero migrate between clusters?
Yes — backup on one cluster, restore on another is a first-class workflow, and the standard way to move workloads during a cluster rebuild or the blue/green upgrade pattern. That dual use is another reason the quarterly drill pays off: it rehearses two procedures at once.

Kubernetes Network Policies in 2026: Lock Down Pod Traffic Without Breaking Your App

softverseAdmin — Sat, 31 Oct 2026 10:33:40 +0000

Kubernetes Network Policies in 2026: Lock Down Pod Traffic Without Breaking Your App

TL;DR: By default, every pod in Kubernetes can talk to every other pod — a flat network with no internal walls. Network Policies are the built-in way to change that: declare which pods may talk to which, on which ports, and everything else is denied. Start with a default-deny policy per namespace, add back exactly the flows your app needs (including DNS — everyone forgets DNS), and test with real connection attempts, not hope. One catch: your CNI plugin must actually enforce policies, or you are writing rules nobody obeys.

Originally published on DevOps Enginer.

The problem: a flat network inside the cluster

Kubernetes networking guarantees that pods can reach each other. What it does not give you, out of the box, is any restriction on that reachability. A compromised frontend pod can scan the database. A batch job can reach the payment service. A pod in the dev namespace can talk to a pod in prod if they share a cluster.

Network Policies fix this at the platform layer. They are firewall rules expressed as Kubernetes objects: selectors pick the pods a policy applies to, and rules list the allowed sources (ingress), destinations (egress), and ports. Anything not explicitly allowed is denied — but only once at least one policy selects a pod. A pod with no policy selecting it remains wide open, which is the single most misunderstood part of the whole feature.

First, the prerequisite nobody checks: your CNI

NetworkPolicy is an API, not an implementation. The enforcement is done by your CNI plugin or network stack. Calico and Cilium enforce policies fully. Flannel, historically, does not enforce them at all — you can apply perfect policies and traffic flows anyway. On managed services it varies: Amazon's VPC CNI supports Network Policy enforcement in current EKS versions, GKE offers Dataplane V2 or Calico-backed policy, and AKS has Azure CNI powered by Cilium or its own policy manager options.

Before writing a single rule, verify enforcement in a scratch namespace: apply a default-deny, try a connection that should fail, and watch it fail. A policy that has never been tested is a rumour, not a control. (The security-basics guide in this series covers where Network Policies sit among the other cluster controls.)

The anatomy of a policy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-allow
  namespace: shop
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: web
      ports:
        - protocol: TCP
          port: 8080
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432

Read it as a sentence: Pods labelled app: api in namespace shop may accept TCP 8080 from pods labelled app: web, and may initiate TCP 5432 to pods labelled app: postgres. Everything else for these pods is denied.

Details that matter:

Empty podSelector: {} selects every pod in the namespace — that is how you write a namespace-wide default-deny.
policyTypes is a promise. If you list Egress but write no egress rules, all egress is denied. If you omit Egress from policyTypes while writing an ingress-only policy, egress stays open.
from/to entries in one list item are OR'd; separate list items are also OR'd. Within one item, a podSelector and namespaceSelector together are AND'd — the classic way to say "pods with this label, in namespaces with that label."
Rules are additive only. There is no deny rule and no ordering. Policies from multiple objects union together, which is why the default-deny-plus-allows pattern works.
Policies are namespaced. A policy in shop cannot select pods in another namespace directly; cross-namespace rules go through namespaceSelector.

Step 1: default-deny per namespace

The baseline move in every namespace that runs real workloads:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: shop
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Apply this and the namespace goes dark: nothing in, nothing out. You have not secured the app yet — you have made the allowed list explicit and empty. Now you earn each flow back deliberately. Do this in a staging namespace first, because the next thing that happens is always the same.

Step 2: allow DNS, the flow everyone forgets

The first casualty of default-deny egress is DNS. Pods resolve names through the cluster DNS service (kube-dns/CoreDNS in kube-system), and if egress to it is not allowed, every lookup fails and the app errors look like five unrelated bugs.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: shop
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Two subtleties: when a rule item contains both a namespaceSelector and a podSelector, they combine with AND — exactly what you want here. And allow TCP 53 as well as UDP 53; large responses and retries fall back to TCP, and "DNS mostly works" is a miserable state to debug.

Step 3: allow the real flows, one hop at a time

Now walk your architecture diagram and convert each arrow into a rule. For a typical three-tier app in one namespace — web → api → postgres — the api-allow policy at the top of this guide is the middle hop. The web tier needs ingress from the ingress controller's namespace (via namespaceSelector) and egress to the api. The database needs ingress only from the api and should usually have no egress at all beyond DNS.

Work namespace by namespace, and keep the policies in Git next to the deployments they describe. If you run GitOps (the Argo CD guide in this series), a policy change then gets the same review trail as a code change — which is precisely the audit story you want for a security control.

Testing policies: prove the negative

The failure mode of Network Policies is silent — both kinds. Too open, and nothing complains while the risk stays. Too strict, and the app breaks in ways that look like application bugs. Test both directions deliberately:

Positive test: the allowed flow works (web reaches api, api reaches postgres, names resolve).
Negative test: a forbidden flow fails. Exec into a pod — or spin up a throwaway debug pod in the namespace — and attempt the connection that should not work: the database port from the web tier, an external IP, another namespace's service. A timeout or refusal is the policy doing its job; a successful connect means the policy is not selecting what you think it is, or the CNI is not enforcing.

A debug pod recipe that covers most cases:

kubectl run nettest --rm -it --image=nicolaka/netshoot -n shop -- bash
# inside: dig api.shop.svc.cluster.local, nc -vz postgres 5432, curl to the api

(netshoot is a well-known troubleshooting image that bundles the usual network tools.) Label the debug pod like the tier you are testing — policies select by labels, so an unlabelled debug pod tests the default, not the app's reality.

Common mistakes

Applying policies where the CNI ignores them. Test enforcement first; a policy on Flannel-era setups is decoration.
Forgetting policyTypes. If you write egress rules but do not list Egress in policyTypes, your egress rules are not applied the way you expect — the API defaults policyTypes based on what you wrote, and the surprises are all in the direction of "more open than intended" or "mysteriously closed." Be explicit.
Denying DNS and chasing ghosts for an afternoon. Any default-deny egress rollout needs the DNS allow in the same change.
Selecting by namespace name strings. kubernetes.io/metadata.name is the label to match for a namespace; hand-rolled name labels drift.
Trying to write deny rules. There are none in the native API. Deny-by-exception designs belong to the CNI-specific policy types (Calico and Cilium both offer higher-level policy objects with ordering and explicit deny) — use them deliberately, not by accident.
One giant policy per namespace. Small, per-tier policies named after the flow (api-allow, postgres-ingress-from-api) stay readable and reviewable. A 300-line policy is where mistakes hide.
Ignoring egress. Ingress-only thinking leaves data exfiltration and command-and-control paths wide open. Locking egress to known destinations is the higher-value half of the work, and the half most teams skip.

Where this fits in the series

Network Policies are one layer, not the whole story. They sit alongside the controls from the security-basics guide (RBAC, Pod Security Standards, secrets handling from the ConfigMaps and Secrets guide), the namespace and quota discipline from the resource requests and limits guide, and the upgrade hygiene that keeps the CNI itself current. No single layer carries the cluster; the layers are the point.

FAQ

Do Network Policies replace a service mesh?
No. Policies operate at L3/L4 — pods, namespaces, ports. A mesh adds identity-based, L7-aware controls (which service may call which path). Many clusters run policies alone happily; regulated or multi-tenant environments often run both.

Can a policy allow traffic from outside the cluster?
Ingress rules accept ipBlock entries (CIDR ranges), which is how you allow a load balancer range or an office VPN. Prefer selecting the ingress controller's pods by label when the traffic enters through it — CIDRs of cloud load balancers change more often than labels do.

What about traffic between nodes, or to the node itself?
Native policies govern pod traffic. Host and node-level rules are CNI-specific territory again (Calico's GlobalNetworkPolicy and Cilium's host policies) — reach for those when you need them, knowingly.

How do I roll this out on a live cluster without an outage?
Namespace by namespace, staging first: apply default-deny where a copy of the workload runs, add the flows, run the positive/negative tests, then repeat in production during a quiet window with the Git revert one command away. The rollout is a series of small, reversible changes — treat it with the same respect as the upgrades in this series' cluster-upgrades guide and it stays boring, which is the goal.

Amazon EKS in 2026: Managed Kubernetes on AWS, Explained for Working Engineers

softverseAdmin — Sat, 31 Oct 2026 09:15:41 +0000

TL;DR: EKS gives you a managed Kubernetes control plane and leaves everything else — nodes, networking shape, IAM wiring, add-ons — as your design decisions. The choices that matter: nodes come as managed node groups (the sensible default), self-managed groups (when you need control), or Fargate (serverless pods, with real constraints); Karpenter increasingly replaces static groups for teams that want nodes to follow demand. Networking runs on your VPC — pods get real VPC IPs from your subnets, which makes the VPC design a Kubernetes decision. IAM meets Kubernetes through Pod Identity (the current, simpler answer to "how does this pod get AWS permissions"), and the cluster is only production-shaped once the add-ons land: EBS CSI for storage, the Load Balancer Controller for ingress, external-dns and cert-manager for the public face. What EKS removes is the control plane burden. Everything in the rest of this series still applies — it just runs on AWS now.

Originally published on DevOps Enginer.

Most teams choosing EKS are not really choosing Kubernetes — they already know it. They are choosing to stop operating a control plane. What surprises them is how much remains theirs: EKS is managed Kubernetes, not managed everything, and the gap between "cluster created" and "cluster production-ready" is a series of decisions this guide walks through.

What AWS manages, and what it pointedly does not

AWS runs the control plane — API servers and etcd across multiple AZs, patched and highly available — and charges a flat per-cluster hourly fee for it. That is the entire managed surface. Nodes are your EC2 instances (or Fargate tasks), the VPC is yours, IAM is yours, and the add-ons that make a cluster useful are yours to install and maintain. Understanding this boundary prevents the most common EKS disappointment: expecting an app platform and receiving excellent infrastructure primitives.

Cluster creation itself (eksctl, Terraform, console) is the easy afternoon. The decisions below are the ones with consequences.

Nodes: three ways to run pods, and a fourth that changes the question

Managed node groups — AWS handles the EC2 lifecycle (provisioning, AMI updates, draining on termination) behind a group definition: instance type, size range, subnets, labels. This is the default recommendation: boring, supported, and sufficient for most clusters.

Self-managed node groups — you own the Auto Scaling group and the bootstrap. Justified by custom AMIs, unusual instance configuration, or requirements the managed path doesn't expose. You also own every patch and every upgrade chore — the upgrades guide applies with extra steps.

Fargate — no nodes at all: each pod runs in an isolated micro-VM with a per-pod price. The constraints are the fine print: no DaemonSets, no persistent volumes beyond EFS, pod sizes from a fixed menu, and startup latency that batch and CI workloads forgive and latency-sensitive services don't. Excellent for spiky, stateless, low-count workloads; a poor default for a whole cluster.

Karpenter — the open-source autoscaler from AWS that increasingly replaces static groups and the cluster-autoscaler pattern: it watches pending pods and launches right-sized instances (type, family, AZ chosen per workload) in under a minute, then consolidates under-used nodes away. For clusters whose demand moves — most clusters — Karpenter is where the autoscaling story lands on AWS, and its cost effect is the rare one that shows up on the bill uninvited, in the good direction.

Networking: your VPC is the cluster network

EKS uses the AWS VPC CNI by default, which means pods receive real IP addresses from your VPC subnets. The consequences are pleasant (pods are first-class network citizens; security groups can even attach per-pod) and constraining (pod density per node is limited by ENI/IP capacity, and subnet IP space is a real planning input — small subnets plus big clusters is a known way to run out of addresses). Prefix delegation raises pod density substantially where needed. This is why the subnet sizing in the VPC guide leaned generous: Kubernetes spends IP addresses like it means it.

The public face of the cluster is the AWS Load Balancer Controller: annotate an Ingress or Service and it provisions an ALB/NLB with target groups wired to your pods — the AWS-native version of the ingress patterns covered earlier. Add external-dns (Route 53 records follow your ingresses automatically) and cert-manager with ACM or Let's Encrypt, and "deploy an app, get a public HTTPS URL" becomes the paved road rather than a manual ritual.

IAM inside the cluster: Pod Identity is the current answer

Pods need AWS permissions — read this bucket, publish to that queue — and the cardinal rule is unchanged from everywhere else in AWS: per-workload roles, never node-wide credentials. A node role big enough for every pod on the node is a privilege-escalation welcome mat, per the IAM least-privilege guide.

EKS has had two mechanisms. IRSA (IAM Roles for Service Accounts) works via OIDC and annotations and remains everywhere in existing clusters. EKS Pod Identity is the newer, simpler successor: associate an IAM role with a service account through the EKS API, an agent on the node handles credentials, and the annotation archaeology of IRSA goes away. New clusters should start on Pod Identity; existing ones can run both while migrating. Either way, the design is identical: one role per workload, scoped to the resources that workload touches, with the cluster components (CSI driver, LB controller, Karpenter, external-dns) each holding their own deliberately-sized roles.

The add-ons that make it production-shaped

A fresh EKS cluster is deliberately minimal. The standard load-out, whether installed as EKS add-ons or via Helm/Argo CD:

EBS CSI driver — without it, PersistentVolumes on EBS don't exist; the storage model (gp3 classes, Retain policies, snapshots) hangs off this driver on AWS.
AWS Load Balancer Controller — ingress and services, as above.
CoreDNS and kube-proxy — shipped as EKS add-ons; keep their versions in step with cluster upgrades rather than discovering skew during one.
Metrics and logs — metrics-server or the Prometheus stack for the observability baseline, plus a log route (Fluent Bit to CloudWatch Logs or your aggregator). A cluster nobody can see into is a cluster on borrowed time.
Cluster access — modern EKS uses access entries (API-based authentication mapping IAM principals to Kubernetes permissions), replacing the hand-edited aws-auth ConfigMap that caused a generation of "I locked everyone out of the cluster" incidents. Whoever provisions the cluster should set access policy deliberately on day one; the creator-admin trap (only the creating identity can administer, ever) is an old rite of passage best skipped.

Everything after installation is the standard Kubernetes operating model — GitOps delivery, progressive rollouts, security basics. EKS changes the substrate, not the discipline.

Upgrades and cost: the two recurring appointments

Upgrades follow the standard Kubernetes rhythm with AWS holding the control-plane end: you trigger the control-plane upgrade one minor version at a time, then roll node groups (or let Karpenter drift-replace nodes onto the new AMI), then bump add-ons. Support windows matter here: standard support for a Kubernetes version on EKS runs about 14 months from its EKS release, with paid extended support beyond — the calendar in the upgrades guide applies verbatim.

Cost has three components people chronically under-model: the per-cluster control-plane fee (multiplied by every forgotten dev cluster — see the Kubernetes cost guide for why cluster sprawl is the quiet line item), the nodes themselves (where Karpenter's right-sizing and consolidation do real work, plus Spot for fault-tolerant capacity), and the AWS perimeter — NAT gateways, load balancers and data transfer that the VPC design decided before the cluster existed.

Common mistakes

Expecting a platform. EKS is infrastructure with a managed control plane. Teams that budget zero platform work discover the add-on, IAM and networking decisions arriving anyway, unscheduled.
The node role that does everything. One broad role for the node group means every pod inherits it. Per-workload identity via Pod Identity is the entire fix, and it's cheap.
Subnets sized for the demo. Pods consume VPC IPs directly; small subnets cap cluster growth in a way that surfaces mid-incident. Size for the cluster you expect, with the VPC's spare range held back.
aws-auth archaeology. Hand-editing the access ConfigMap (on clusters still using it) locks teams out with one typo. Prefer access entries; if you must edit aws-auth, do it from a session that stays open until a second login is verified.
Add-ons drifting behind the cluster version. CSI and LB controller versions have compatibility windows; treat add-on upgrades as part of the cluster upgrade runbook, not a separate someday.
A cluster per team per environment, forever. The per-cluster fee plus duplicated add-ons and perimeter makes sprawl a budget line. Consolidate with namespaces and per-team IAM where isolation requirements genuinely allow it — and keep dedicated clusters where they don't (compliance, blast radius), knowingly.

FAQ

EKS vs self-managed Kubernetes on EC2?
Self-managed gives total control and total responsibility: control-plane HA, etcd backups, and patching are yours. EKS removes exactly that layer for a flat fee. Teams choose self-managed for specific control or cost-at-scale reasons; everyone else is buying back their weekends with EKS, correctly.

EKS vs ECS Fargate?
If the team knows Kubernetes or needs its ecosystem (Helm charts, operators, the tooling in this series), EKS. If the need is "run these containers on AWS with minimal concepts," ECS is simpler and entirely respectable. The honest test is which ecosystem your hiring, tooling and existing workloads already speak.

Can I run a cluster with no managed node groups at all?
Yes — Fargate-only clusters exist and suit small, spiky, stateless estates. The DaemonSet and storage constraints decide for most teams; mixed setups (node groups for the baseline, Fargate for burst jobs) are common.

How many clusters should we have?
Fewer than instinct suggests, more than one for anything serious: separate production from non-production at minimum. Beyond that, let blast radius, compliance boundaries and team autonomy argue it out — with the per-cluster and perimeter costs on the table while they do.

Conclusion

EKS is a trade stated plainly: AWS operates the control plane; you keep the architecture. Choose node strategy deliberately (managed groups by default, Karpenter when demand moves), design the VPC like the cluster depends on it — because it literally runs on it — wire IAM per workload with Pod Identity, and install the add-ons that turn a cluster into a platform. Do those things and EKS delivers what it promises: Kubernetes without the control-plane pager. The rest of running Kubernetes well is, as ever, on you — and it's what the other fifteen parts of this series are for.

The free Kubernetes Production Checklist (via the newsletter on the homepage) includes the production-readiness items this guide builds toward — access, backups, observability and upgrade readiness.