DEV Community: Nnamdi Felix Ibe

Day 77: A Pipeline and a Private Subnet

Nnamdi Felix Ibe — Mon, 02 Nov 2026 18:00:00 +0000

Day 77 of DevOps, Day 27 of Azure. Both tasks today were about a path that exists until it doesn't.

A two-line shell script deployed an application and quietly depended on three things nobody had checked. And an Azure VM built with no public IP turned out to be the exact configuration that the platform is in the middle of changing underneath.

One Jenkins task, one Azure task. Write a first declarative pipeline, then build a VM reachable only from inside its own network. The tasks come from the KodeKloud Engineer platform.

Task 1: A first declarative pipeline

The task: a Pipeline job named datacenter-webapp-job that pulls the latest code from master onto stapp01.

Step 1: The agent and a stale instruction

Plugins:  SSH Build Agents, Pipeline
Agent:    App Server 1, label stapp01, remote root /home/sarah/jenkins_agent

In every Jenkins task in this batch, I had to install Java 17 on the node, "as Jenkins (or its remoting agent) is compiled for Java 17".

The shape of that is right, and the version is two baselines stale. Jenkins' Java requirement is a project decision with dates on it, not a property of a Java release. It became 17 at the 2.463 weekly release in June 2024, covering "agent JVMs (i.e., JVMs running remoting.jar)" explicitly. And it has moved again since: "Beginning with Jenkins 2.555.1, Jenkins requires Java 21 or Java 25 on both the controller JVM and agent JVMs", with the Java 17 images retired.

So Java 17 is correct for this lab, which runs 2.541.3, the last LTS supporting it. As a general rule, it is already out of date. Match the controller's requirement, and upgrade the controller and agents together.

Day 75 covered what this prevents, and my notes undersell it by framing it as a compilation detail: "Running a Java 11 agent with a Java 17 controller produces an UnsupportedClassVersionError." The agent does not start at all.

Step 2: The pipeline

pipeline {
    agent { label 'stapp01' }

    stages {
        stage('Deploy') {
            steps {
                sh '''
                cd /var/www/html
                git pull origin master
                '''
            }
        }
    }
}

agent { label 'stapp01' } is documented plainly: it will "Execute the Pipeline, or stage, on an agent available in the Jenkins environment with the provided label." Label expressions work too, so 'my-label1 && my-label2' is valid once more than one agent shares labels.

The thing worth noticing is that the job names a label, not a machine. That is the payoff from Day 75's agent work: pointing this deployment at a different server is a label change, not a job rewrite.

And the sh '''...''' uses triple single quotes. Nothing in this script references a variable, so today it makes no difference whatsoever. Tomorrow it makes a large one.

What the docs recommend

Two lines of shell, and three assumptions underneath them.

That /var/www/html is already a git working copy with an origin remote. On a fresh server, it is a document root and nothing else, and git pull there fails with a message about not being a git repository, which reads like a path typo.

That the agent's user can write to it, this pipeline runs as sarah, and it works only if sarah owns the directory or has write access. Worth watching across this batch, because three days answer this one question three different ways: today leaves it implicit, Day 79 checks the ownership explicitly, and Day 80 reaches for sudo with a password in the build step.

That a pull is the right operation at all. This is the one I would change. git pull merges. A local edit in the document root produces a merge instead of a deployment, and a conflict leaves the web root half-merged with conflict markers in the files Apache is serving.

git fetch origin
git reset --hard origin/master

That is the deployment-shaped operation, because it makes the working tree match the remote regardless of what was there before. A deployment should not care about local state. A pull does, by design.

Task 2: A VM reachable only from inside

The task: Day 26 inverted. Same resource shapes, opposite reachability, with SSH permitted only from the VNet's own CIDR block.

Step 1: One flag decides the whole task

--nsg-rule NONE

--nsg-rule accepts {NONE, RDP, SSH}, and for a Linux VM it defaults to SSH, which creates default-allow-ssh with source *. That permits SSH from anywhere on the internet, which is the direct opposite of what this task asks for.

Day 26 left the default in place and was right to. Same command, opposite requirement, one flag, and it is easy not to think about precisely because it is the default.

So the verification has to list everything:

Name            Pri   Dir      Access  Proto  Src          Dst          Port
Allow-SSH-VNet  100   Inbound  Allow   Tcp    10.0.0.0/16  10.0.0.0/16  22

One row. A default-allow-ssh row sitting next to it would mean the VM was internet-reachable while every other check still looked fine, and the NSG would read as "scoped to the VNet" at a glance.

When the requirement is about what is not permitted, confirming the rule you added proves nothing. You have to enumerate what exists.

Step 2: Two layers, and only one of them is about inbound

"PubIP": "",  "PrivIP": "10.0.1.4"

--public-ip-address "" means no internet-routable address, so there is no inbound path. The NSG then restricts source even inside the VNet. Either alone would do it; both together survive a single mistake, because adding a public IP later would not open SSH and loosening the NSG would not either.

That much I had. What I had missed is that the absence of a public IP is also the condition that a platform change is now acting on, and it affects what this VM can do rather than what can reach it.

Step 3: This is the VM that loses its way out

How does a VM with no public IP reach the package mirrors at all? Microsoft: "when you deploy a virtual machine (VM) in a virtual network without an explicitly defined outbound connectivity method, Azure automatically assigns it an outbound public IP address."

That address is not yours. "This IP, known as the default outbound access IP, is owned by Microsoft and can change without notice", and "default outbound connectivity relies on an implicit platform behaviour and may be affected by platform-level changes."

And it is going away for new subnets. "Creating a subnet to be Private prevents any virtual machines on the subnet from utilising default outbound access to connect to public endpoints." Day 76 recorded the date, which has passed: subnets in new virtual networks now default to defaultOutboundAccess set to false.

So a VM built to exactly this specification, in a new virtual network today, has no inbound path and no outbound path.

The four explicit outbound methods are a NAT gateway on the subnet, an outbound-rule load balancer, a Standard public IP on the NIC, and a firewall or appliance behind a user-defined route. Day 26's VM had the third one by accident: the public IP that made it reachable also made its egress explicit. This VM has none of them, and a NAT gateway is the documented recommendation for most scenarios.

Two exceptions stop a private VM from being completely cut off, and they are worth knowing in opposite directions. "Virtual machines are still able to access Azure Storage accounts in the same region in a private subnet without an explicit method of outbound." But "Certain services don't function on a virtual machine in a private subnet without an explicit method of egress (examples are Windows Activation and Windows Updates)."

Microsoft's stated reasons read as an argument rather than a deprecation notice: "Security: Default internet access contradicts Zero Trust principles. Clarity: Explicit connectivity is preferred over implicit access. Stability: The default outbound IP isn't customer-owned and can change."

What the docs recommend

The enforcement here is entirely outside the VM, and it is worth being precise about that, because "SSH accessible from within the VNet only" sounds like host configuration and is not:

LISTEN 0 128 0.0.0.0:22  users:(("sshd",pid=861,fd=3))

sshd is bound to 0.0.0.0, accepting from any address. Nothing in the guest narrows it. Which means an NSG change takes effect immediately with no reboot, and also that an NSG mistake has no backstop: there is no firewall rule or ListenAddress on the host to catch it.

One more thing I did not do, and would rather say than imply. The task says SSH should work from inside the VNet, and I never tested that, because there is no second VM in the VNet to connect from. What I established is that sshd is listening, the NSG permits 22 from 10.0.0.0/16, no other inbound rule exists, and there is no public IP. That makes the conclusion sound, and it is an inference.

The documented way to actually test it is Azure Bastion, which deploys into an AzureBastionSubnet and brokers SSH without putting a public IP on the target. Given Day 38 failed precisely by treating configuration as proof of behaviour, the distinction is worth keeping visible.

What I learned

Both halves were about a path that works until something stops holding it up.

The pipeline deploys correctly, right now, on a server where someone previously made /var/www/html a git working copy and gave sarah write access to it. None of that is in the pipeline. The job is two lines, and the configuration it depends on is somewhere else entirely, undocumented, and discovered only when it is missing.

The Azure VM reaches the internet, right now, through an address Microsoft owns and assigned implicitly because the VM had no other way out. That is being retired, with a date that has already passed for new networks, and Microsoft's own first reason is that implicit internet access contradicts Zero Trust. The VM is not misconfigured. The thing it was resting on is being removed.

So here is the Day 77 question. What does your deployment job assume about the server it deploys to that the job itself does not say?

Day 77 down. Twenty-three to go.

Day 76: A Backup of Nothing and a Retiring Default

Nnamdi Felix Ibe — Sun, 01 Nov 2026 09:00:00 +0000

Day 76 of DevOps, Day 26 of Azure. Both tasks today went green, and both were quietly resting on something that will not hold.

A database backup job ran on schedule, produced a file, copied it across two machines, and backed up no tables at all. And an Azure network behaviour I had written in my notes as how Azure works turned out to be a platform default that Microsoft is in the middle of retiring.

One Jenkins task, one Azure task. Schedule a database backup across three servers, then put a VM in a purpose-built virtual network. The tasks come from the KodeKloud Engineer platform.

Task 1: A database backup on a ten-minute schedule

The task: a job named database-backup that dumps a database on stapp01 every ten minutes and copies the dump to the storage server.

Step 1: Two hops, two keys

Three machines are involved, and the Jenkins controller never touches the storage server directly. So each hop needs its own key, belonging to the account that opens it.

jenkins@jenkins:~$ ssh-keygen -t rsa -b 4096
Created directory '/var/lib/jenkins/.ssh'.
jenkins@jenkins:~$ ssh-copy-id tony@stapp01

jenkins@jenkins:~$ ssh tony@stapp01
[tony@stapp01 ~]$ ssh-keygen -t rsa -b 4096
[tony@stapp01 ~]$ ssh-copy-id natasha@ststor01

The second key had to be generated as tony, because tony@stapp01 is where the first hop lands and therefore the account that makes the second call. One key cannot do both hops.

This is the chain Day 74 could not account for: there, the transcript showed the second hop keyed and never showed the first, and the build still went green. Here both are on the record.

One detail on the second hop worth noticing:

The authenticity of host 'ststor01 (10.244.240.154)' can't be established.
ED25519 key fingerprint is SHA256:yEyN8qvzhNxfcKVE+H05zwQPmQMKCXj4JyGWuOP1HIg.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

That prompt is exactly what Day 75's agents switched off with their non-verifying host key strategy. Answered once interactively, recorded, done.

Step 2: The build step, and one thing it gets right

ssh tony@stapp01 "
  mysqldump -u kodekloud_roy -pasdfgdsd kodekloud_db01 > /tmp/db_\$(date +%F).sql &&
  scp /tmp/db_\$(date +%F).sql natasha@ststor01:/home/natasha/db_backups/
"

The && is inside the quotes, which is Day 71's bug written correctly. There, ssh host cmd1 && cmd2 split in the local shell and only the first half ever reached the remote machine. Here the whole conditional is one argument to ssh, so the scp runs on stapp01 and only if the dump succeeded.

The escaping is right too, and for a reason my notes got slightly wrong. I had written that the backslash makes the filename come out in the right format. What it actually does is choose whose clock names the file: \$ stops the local shell expanding the substitution, so the literal text is sent and date runs on stapp01.

Which is the exact inverse of Day 71, where "$PACKAGE" had to expand locally because the Jenkins parameter only exists on the Jenkins side. Same mechanism, opposite intent. Escape it to evaluate remotely, leave it bare to evaluate locally, and know which one you mean.

Step 3: The job works. The backup does not.

The file arrived on both machines and the build went green. Then I read it:

-- MariaDB dump 10.19  Distrib 10.5.29-MariaDB, for Linux (x86_64)
--
-- Host: localhost    Database: kodekloud_db01
-- ------------------------------------------------------
-- Server version       10.5.29-MariaDB

/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
...
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
...
-- Dump completed on 2026-04-14  7:02:52

A header, session settings, the restore of those session settings, and a completion line. No CREATE TABLE. No INSERT. Nothing whatsoever between the opening SET block and the closing one.

So kodekloud_db01 either has no tables, or kodekloud_roy cannot see the ones it has. Either way this job backs up zero rows, successfully, every ten minutes, and produces a plausible-looking file while doing it.

The task is satisfied. It asked for a backup job and there is one. The backup is worthless, and nothing anywhere in the pipeline says so: valid SQL, reasonable size, green build, file present at the destination.

This is Day 74's lesson with the stakes raised. There, a scheduled job copying no logs succeeds on time forever, and the fix was to confirm the source files existed first. Here the equivalent check is one line:

mysql -u kodekloud_roy -p... -e "SHOW TABLES IN kodekloud_db01"

And the verification that belongs in the job itself asserts on content rather than exit status:

grep -qi 'CREATE TABLE' "$DUMP" || { echo "dump has no tables"; exit 1; }

An empty lab database and a privilege problem look identical from outside, and mysqldump exits zero for both. Only the content tells you which you have.

What the docs recommend

Three things, and the first is the one that would matter in a real environment.

The password is in the job. MySQL does not hedge: "Specifying a password on the command line should be considered insecure. To avoid giving the password on the command line, use an option file." The documentation also explains the odd-looking flag, since "there must be no space between --password= or -p and the password following it."

In a Jenkins job this is worse than at a terminal, because the command is stored rather than typed. The build step lives in the job's config.xml, and anyone with Job/Configure on the job can read it in a browser.

Day 73 granted Job/Configure on a job to two people. The same grant here hands over a database credential, and nothing in the permission grid hints at it. The column says Configure. What it conveys is the contents of every build step.

Jenkins points at the fix. Credentials have their own permission group, with Credentials/View and Credentials/Update kept separate from job permissions, which is Jenkins separating credential access from job access on purpose. And its Pipeline documentation gives the pattern directly: never interpolate a secret into the command text, send a literal command string, and let "the shell read secrets from its environment", which the credentials binding has already populated. Two things improve at once, because the value leaves config.xml and Jenkins masks it in the console output.

Second, there is a latent bug in the date. date is called twice, and the two substitutions are evaluated at different moments. If midnight falls between the dump finishing and the scp starting, mysqldump writes one filename and scp looks for another and fails. On a ten-minute schedule that is one build a day exposed to it, presenting as an occasional failure with no obvious cause.

Computing it once removes the whole class of problem, and lets the quoting get simpler because nothing needs to expand locally any more:

ssh tony@stapp01 '
  D=$(date +%F)
  mysqldump --defaults-file=~/.my.cnf kodekloud_db01 > "/tmp/db_$D.sql" &&
  scp "/tmp/db_$D.sql" natasha@ststor01:/home/natasha/db_backups/
'

Third, the schedule. H/10 * * * * rather than */10 * * * *, for Day 74's reasons: H "should be used wherever possible" and is "a hash of the job name, not a random function". With a log-copy job already on a six-minute cycle, round intervals guarantee the two collide eventually, and H separates them without anyone coordinating. And mysqldump without --single-transaction locks tables while it reads, which costs nothing on an empty database and blocks writes every ten minutes on a real one.

Task 2: A VM in a purpose-built network

The task: a VNet, a subnet, and a VM inside it with a public IP and SSH reachable from the internet.

Step 1: Name things rather than letting the VM name them

--vnet-name devops-pub-vnet --subnet devops-pub-subnet
--public-ip-address devops-pub-vm-pip --public-ip-sku Standard
--nsg devops-pub-vm-nsg

Every earlier VM in this track let az vm create build its own network, which produces names derived from the VM, like <vm>VNET and <vm>VMNic. Here the network came first and the VM was told where to go, so the names come from the design. That starts mattering the moment a second VM shares the network.

Naming the public IP also carries the Day 21 benefit: it survives the VM being deleted and recreated, so the address stays put.

Step 2: What makes the network public, and why my answer has expired

My notes said a VNet has no public or private attribute, and that reachability comes entirely from a public IP on the NIC plus an NSG rule. The first half still holds. The second is now incomplete, and what has changed is outbound rather than inbound.

Subnets do now carry a relevant property, and Microsoft's own word for the result is private: "Azure is changing the default subnet configuration for new virtual networks to require explicit outbound connectivity." With a date attached that has already passed: "For the API version released after March 31, 2026, subnets in new virtual networks have defaultOutboundAccess set to false by default."

So a VNet created today with a current API version gets subnets whose VMs "can't rely on default outbound access". My note's claim that Azure handles the routing and the NSG is the only gate describes a platform behaviour being retired, not a property of Azure networking.

Three things keep that from affecting this task, and the second is the interesting one.

Existing networks are untouched: "Existing virtual networks aren't changed", and deployments on earlier API versions "retain the earlier default behavior", which the lab's CLI may well have done.

And this VM already has an explicit outbound method, without my having chosen one for that purpose. Microsoft lists "Associate a Standard public IP to any of the virtual machine's network interfaces" among the four explicit methods, and --public-ip-sku Standard provided exactly that. A VM built to be reachable from the internet happens to have the thing that also makes its egress explicit.

Of those four methods, Microsoft marks one as preferred: "Associate a NAT gateway to the subnet of your virtual machine. Note this is the recommended method for most scenarios."

Which means the AWS comparison in my notes needs qualifying rather than deleting. Azure still has no internet gateway or route table to configure for basic reachability, which is why Day 40 and Day 45 have no counterpart on this track. What it no longer has is outbound connectivity you can assume. A private subnet needs a NAT gateway, and that is the closest Azure analogue to the work Day 45 did on AWS.

What the docs recommend

Two smaller things, both about not doing the same work twice.

The port 22 rule was created twice. --nsg-rule was not passed to az vm create, and its accepted values are {NONE, RDP, SSH} with Linux defaulting to SSH, which creates a default-allow-ssh rule at priority 1000. The explicit az vm open-port --port 22 --priority 1001 then added a second rule for the same port. Harmless, because both are Allow and the one at 1000 matches first, and still redundant configuration in two places. Either --nsg-rule NONE plus the explicit call, or --nsg-rule SSH and no call, leaves one rule and one place to change it.

And the disk flags, which I have now carried forward three times and explained wrongly twice:

--storage-sku Standard_LRS --os-disk-size-gb 30

Day 21 failed on this, and Days 22, 23 and 26 all pass the flags without being asked to. My explanation, that az vm create defaults the OS disk to Premium on premium-capable sizes, is wrong, as Day 72 covered: --storage-sku has no documented default at all, and the managed-disk path sends no storage account type whatsoever. Something below the CLI chooses, and the rule is not published. The flag is still mandatory in this subscription.

What I learned

Both halves of today were green and both were built on something that does not hold up.

The backup job is the one I would want to find before someone needed it. Every signal said it worked: exit zero, file created, file copied, file identical on both machines, build green, schedule firing. The only thing that said otherwise was reading the file, and what it said was that the job had been faithfully backing up nothing. No monitoring built on exit codes would ever have caught it, because nothing failed.

The Azure half is the same error made against documentation rather than against a file. I wrote "Azure handles the routing" in my notes as a fact about the platform. It was a default, and Microsoft had already announced its retirement with a date that has since passed. The VM survives on a technicality that happens to be sound: the Standard public IP that makes it reachable is also, separately, one of the four documented ways to make its egress explicit.

The pattern underneath both is the same one from Day 75, arriving by a different route. A thing that is true today, recorded as a thing that is true.

So here is the Day 76 question. Which of your backups has anyone actually opened?

Day 76 down. Twenty-four to go.

Day 75: A Java Baseline and a Moving Disk

Nnamdi Felix Ibe — Sat, 31 Oct 2026 09:00:00 +0000

Day 75 of DevOps, Day 25 of Azure. Both tasks today punished something I had written down as a fact when it was really a snapshot.

A Jenkins agent would not start because the Java version on it had been fine until the controller moved past it. And an Azure disk that was sda before the task was sdb after it, with a different disk holding the old name.

One Jenkins task, one Azure task. Attach the app servers to Jenkins as build agents, then expand one disk and add another. The tasks come from the KodeKloud Engineer platform.

Task 1: Three servers as build agents

The task: attach the three app servers to Jenkins as agents. For the first one, name App_server_1, label the same, remote root directory /home/tony/jenkins.

Step 1: The plugin, and what it is called now

The plugin is SSH Build Agents. It "is formerly known as 'SSH Slaves Plugin'" and was renamed in version 1.31.0, and its plugin ID is still ssh-slaves.

So three names for one thing, which is why searching for any of them lands in the same place. The KodeKloud task uses the old vocabulary, and so does the identifier you would put in a configuration file.

Its documented scope is almost nothing: it "Allows to launch agents over SSH", using "a Java implementation of the SSH protocol", and it "Requires Jenkins 2.504.1". That is the whole of what its page says about behaviour.

Step 2: Create the node, and find the fields are undocumented

Name, label, remote root directory, host, credentials, host key verification strategy. Jenkins' Managing Nodes page defines none of them. It does not describe the Remote root directory, the Labels field, the Usage field, or the launch methods, and does not list SSH among them.

That is the third day running. Day 71 was the Execute shell build step, with no page. Day 72 was String and Choice parameters, with no page. Today it is the entire node configuration form.

What Jenkins does document is the requirement and the reason for the whole exercise. "An agent can use any operating system that supports Java", and you should "Ensure that you have a java version available on your agent machine". And then, plainly: "running tasks on the built-in node is discouraged for security, performance, and scalability reasons."

That last line is why this task exists. Days 71 through 74 ran everything on the controller, directly or over ssh from it. This is the day the work moves off the machine Jenkins itself lives on.

Step 3: The agent refuses to start

java.lang.UnsupportedClassVersionError: hudson/remoting/Launcher has been compiled by a
more recent version of the Java Runtime (class file version 61.0), this version of the
Java Runtime only recognizes class file versions up to 55.0
Agent JVM has terminated. Exit code=1

[tony@stapp01 ~]$ java -version
openjdk version "11.0.20.1" 2023-08-24 LTS

Java 11 on the agent. Installing java-17-openjdk and relaunching brought it online.

The fix was right. My explanation of it was wrong in three ways, and the correction is more useful than the fix.

I had written that Jenkins requires Java 17 "because it is the current Long-Term Support (LTS) version, and its components are compiled to run on it, ensuring better stability".

Java 17 is not the current LTS release and was not then. Java 21 is a later one, and Jenkins has moved past both: "Beginning with Jenkins 2.555.1, Jenkins requires Java 21 or Java 25 on both the controller JVM and agent JVMs." The same page adds that "With the release of Jenkins 2.555.1, the Java 17 images have been retired."

The reason Java 17 was required at all is a project decision with a date on it. Jenkins raised its baseline at the 2.463 weekly release in June 2024, announced as "Jenkins requires Java 17 or newer", with 2.462 being "the last weekly release to support Java 11". The requirement covers "agent JVMs (i.e., JVMs running remoting.jar)" explicitly, and the announcement names this exact failure: "Running a Java 11 agent with a Java 17 controller produces an UnsupportedClassVersionError."

And it is not about stability. The agent's JVM cannot load a class file built for a newer JVM than itself, so the agent does not start. Saying Java 17 gives "better stability" implies Java 11 would be worse. Java 11 would be nothing.

The fix was still correct for this Jenkins: Day 68 established the lab runs 2.541.3, the last LTS release supporting Java 17. Written as a general rule for today, "install Java 17 for Jenkins agents" is already out of date. The general rule is to install whatever the controller's version requires, and to upgrade controller and agents together, agents first.

What the docs recommend

Two choices in this node configuration are worth more attention than the Java version.

The host key verification strategy was set to Non verifying, which I justified as skipping manual trust setup in a lab. Practical, and the strategy names turn out to be undocumented: the plugin's page lists no host key verification options at all.

What its page does carry is a security advisory, and it is precisely this one: "Man-in-the-middle vulnerability due to missing host key verification", affecting version 1.14 and earlier.

So the non-verifying strategy is not a missing feature. It is an option to reinstate, per node, the exposure that advisory was issued to fix. Choosing it accepts that anyone who can answer on the agent's address and port can impersonate the agent, and Jenkins will hand over the connection and the credentials used to open it. Which in this task are tony's password.

That is the second choice. The credential here is a username and password, where Days 71 and 74 both set up key-based trust for the same class of connection. The plugin supports SSH keys. A password is a reusable secret sitting in the credential store for no benefit.

To be fair to the credential store, it is a boundary. Jenkins gives credentials their own permission group, with Credentials/View and Credentials/Update separate from job permissions, so credential access is deliberately not the same as job access. A password in the store is at least behind that line.

A password typed into a build step is not. That is tomorrow's problem.

Task 2: Expand one disk, add another

The task: grow the VM's disk from 32 to 64 GiB, then create a 64 GiB standard HDD named devops-disk and mount it at /mnt/devops-disk.

Step 1: The outage belongs to the OS disk, not the task

az vm deallocate ...
az disk update -g $RG -n $OSDISK --size-gb 64
az disk create -g $RG -n devops-disk --size-gb 64 --sku Standard_LRS
az vm disk attach -g $RG --vm-name devops-vm --name devops-disk
az vm start ...

I had written that expanding a managed OS disk requires the VM deallocated. Right conclusion, and Microsoft never says it in those words, so it is worth showing how it follows.

Expanding without downtime is documented, and its scope is explicit: it "Is supported only for data disks." For the path that does not qualify, "the VM must be deallocated." An OS disk is outside the downtime-free feature, so it lands on the deallocation path by elimination rather than by a rule written about OS disks.

Two things follow that I had not drawn out. The data disk alone could have been expanded with the VM running, so the outage was the OS disk's requirement, not the task's. And deallocate is not a synonym for stop: "Stopping the VM with az vm stop doesn't release the compute resources. To release compute resources, use az vm deallocate."

Since the VM had to stop anyway, creating and attaching the data disk in the same window meant one outage covering both halves of the task instead of two.

Step 2: The reboot did the other two layers, and I overstated how

Day 50's lesson was that a disk has three layers and each needs its own action. After the VM started:

sdb1   63.9G   part /
/dev/root  ext4  62G  1.7G  61G  3% /

Already grown, with nothing run inside the guest.

I credited that to cloud-init running growpart on every boot. The documented claim is narrower and says nothing about every boot: "On Ubuntu 16.x and newer, the root partition of the OS disk and filesystems are automatically expanded to use all free contiguous space on the root disk by cloud-init. A small amount of free space must be available for the resize operation."

So cloud-init does it, on Ubuntu 16.x and newer, to the root partition of the OS disk, and it needs a little free space. Whether it checks on every boot or only when something changed is not stated.

The mirror-image reading survives the correction, and it is the part worth keeping:

AWS allowed an online resize, so no reboot happened, so cloud-init never ran, so the partition and filesystem were manual.
Azure required a deallocation, and the restart that followed is what expanded them.

The platform constraint supplied the convenience. Which matters, because a data disk expanded with the VM running gets no reboot, and so still needs the filesystem grown by hand.

Step 3: The letters moved while I was working

Before this task the OS disk was sda. After attaching one data disk:

sda   64G                      <- devops-disk, the NEW disk
sdb   64G   sdb1 /             <- the OS disk
sdc    4G   sdc1 /mnt          <- the temporary disk

The OS disk became sdb, and the new data disk took the name the OS disk used to have.

Day 8 and Day 50 both warned that device letters are assigned in enumeration order and are not stable. This is that warning arriving. A script written as "format /dev/sda1" against yesterday's layout would have reformatted the root volume today.

So the script resolved the device instead of naming it:

DEV=$(readlink -f /dev/disk/azure/scsi1/lun0)

The LUN is the slot on the VM and does not change. The letter is an artefact of enumeration order. And for the persistent mount, the UUID does the same job, because an fstab entry naming /dev/sda1 would, after the next disk change, mount the wrong filesystem or fail the boot.

Which is the Jenkins half of today in different clothing. sda meant one disk and now means another. remoting.jar ran on Java 11 and now will not. Both were true when written down and neither was a stable fact about the system.

What the docs recommend

The mount point the task asked for is the thing I would change in real infrastructure.

/mnt/devops-disk sits inside /mnt, and on an Azure Linux VM /mnt is where the ephemeral temporary disk is mounted. So this is a persistent disk mounted at a path inside a filesystem that gets wiped on deallocation. It works now. After the next stop and start, the directory will not exist on the fresh temporary disk.

nofail keeps that from blocking the boot, and that is exactly the problem: the data disk may simply not be mounted, and nothing complains. Azure's own fstab line for the temporary disk shows the pattern for this, ordering itself after cloud-init, and the equivalent here would be x-systemd.requires-mounts-for=/mnt.

I have not applied it, because the task asked for the mount and the mount is there. But a persistent disk underneath an ephemeral mount point is the wrong shape, and /data or /srv/devops-disk at the root avoids the whole question.

One more number worth knowing before someone asks. The 64 GiB disk reports 63G to the filesystem and 60G available, and both are correct: ext4's metadata takes the first slice and its root reserve takes the second. Three numbers, one disk, nothing wrong.

What I learned

Everything that broke today was something I had recorded as a fact and that was really a reading at a moment in time.

sda was the OS disk. Java 11 was sufficient for the agent. Neither statement was wrong when written. Both became wrong without anything on those machines changing: one because a disk was attached, the other because the Jenkins project moved its baseline, twice, and my note about it was already stale when I wrote it down.

The defence in both cases turned out to be the same, which is what makes it worth noticing. Do not name the thing, resolve it. The disk script asks for LUN 0 and follows the symlink to whatever letter the kernel chose this boot. The agent should take whatever Java the controller's version requires, checked against the controller, rather than the version that worked in a tutorial.

And the two configuration choices nobody made me justify were the ones carrying actual risk: a host key strategy that reinstates a patched man-in-the-middle vulnerability, and a password where a key would have done.

So here is the Day 75 question. Which version number or device name is hardcoded in your scripts because it was true the day you wrote them?

Day 75 down. Twenty-five to go.

Day 74: A Key Each Way and a Schedule

Nnamdi Felix Ibe — Fri, 30 Oct 2026 18:00:00 +0000

Day 74 of DevOps, Day 24 of Azure. Both tasks today were about SSH keys, and specifically about which way round they point.

A Jenkins job needed one server to be able to reach another. An Azure task started by checking whether a key already existed, and the .ssh directory turned out to be full of a file that pointed the other way. Same two files, opposite directions, and getting them mixed up produces a setup that looks finished and cannot connect.

One Jenkins task, one Azure task. Schedule a job that copies Apache logs between servers, then build a VM that only accepts key-based SSH. The tasks come from the KodeKloud Engineer platform.

Task 1: A job on a six-minute schedule

The task: a freestyle job named copy-logs, running every six minutes, copying the Apache logs from stapp03 into /usr/src/finance/ on the storage server.

Step 1: Check the logs are actually there

[banner@stapp03 ~]$ cat /var/log/httpd/access_log
127.0.0.1 - - [-] "GET / HTTP/1.1" 200 -

Boring step, and it is the one that makes everything after it mean something. A scheduled job that copies nothing succeeds, on time, forever. The build goes green, the destination stays empty, and nothing in Jenkins tells you why, because from Jenkins' point of view nothing went wrong.

Step 2: Key the connection between the two servers

[banner@stapp03 ~]$ ssh-keygen -t rsa -b 4096
Enter file in which to save the key (/home/banner/.ssh/id_rsa):

[banner@stapp03 ~]$ ssh-copy-id natasha@ststor01
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
natasha@ststor01's password:
Number of key(s) added: 1

Generated as banner, not as root, which is why ssh-keygen offered /home/banner/.ssh/id_rsa. The identity has to belong to whichever account opens the connection.

Then one detail in the login banner on the way in, which is worth reading rather than scrolling past:

Last failed login: Sun Apr 12 09:41:57 UTC 2026 from 10.244.81.53 on ssh:notty
There were 2 failed login attempts since the last successful login.

Those two failures are ssh-copy-id probing with the new key before it was installed, which is exactly what its "1 key(s) remain to be installed" message was telling me. Benign here. On a server where you had not just run ssh-copy-id, two failed logins from another host is something to go and look at.

Step 3: Let the schedule fire

Six minutes later the build ran on its own, and:

[natasha@ststor01 ~]$ ls /usr/src/finance/
access_log  error_log

What the docs recommend

A six-minute schedule written the obvious way is */6 * * * *. Jenkins documents a better form, and the reason is load.

The H symbol "should be used wherever possible". The example Jenkins gives is a dozen jobs all set to 0 0 * * *, which produces "a large spike at midnight", against H H * * *, which still runs each job once a day but at a time of its own.

The part worth knowing is that H is not random. It is "a hash of the job name, not a random function", so the value "remains stable for any given project". You get spreading without losing reproducibility. For short cycles the documented example is "every fifteen minutes (perhaps at :07, :22, :37, :52)" written as H/15 * * * *.

So H/6 * * * * is the form for this job. Still every six minutes, but offset within each window by a hash of this job's name, so that this job and the next one somebody writes do not fire in the same instant for the rest of time.

Jenkins also warns about exactly this kind of schedule: "short cycles such as */3 or H/3 will not work consistently near the end of most months", because "Hashes are always chosen in the 1-28 range". Worth knowing before a six-minute job becomes load-bearing.

One link in the chain I cannot account for

This job needs two hops. The logs are on stapp03. The destination is ststor01. The build runs as jenkins on the Jenkins controller, which is neither machine.

What I set up and verified is the second hop, stapp03 to ststor01. What I did not set up anywhere in this task is the first, from the controller to stapp03. And the job ran on schedule and the files arrived.

Two explanations fit. The controller may already have been trusted by the app servers from earlier work, since Day 71 keyed jenkins@jenkins to the storage server and the same thing could have been done to stapp03 on a day my notes do not cover. Or the build did not run on the controller.

I cannot tell from what I have, so I am recording it as an open question rather than writing in the step that would make the story tidy. Same gap as Day 71, where the build command in my notes could not have produced the result my notes also recorded. Same cause, too: a screenshot shows you a green build, not the chain of trust that let it go green.

Task 2: A VM that only accepts keys

The task: create an SSH key if one does not already exist, build a VM that trusts it, and prove password-less access works.

Step 1: A non-empty .ssh directory is not a key pair

-rw------- 1 root root  562  authorized_keys

/root/.ssh/ existed and already had a file in it. No id_rsa, no id_rsa.pub.

Those two serve opposite directions:

File	Direction	Purpose
`authorized_keys`	Inbound	Keys allowed to log in to this host
`id_rsa` and `id_rsa.pub`	Outbound	This host's identity for logging in elsewhere

The lab seeds authorized_keys so that I can reach the client machine. It does not give the client machine an identity of its own.

So "check if a key already exists" has to mean checking for the file the next step needs:

if [ -f /root/.ssh/id_rsa ]; then echo "key exists, reusing"; else ssh-keygen ... ; fi

Written as "does ~/.ssh exist", or "is it empty", the check finds files, concludes a key is present, and leaves the VM creation with no public key to install.

Which is the Jenkins half of today from the other end. There, stapp03 needed an outbound identity and ststor01 needed that public key in its inbound list. Two files, two directions. Confuse them and you get a setup that looks complete and cannot connect.

Step 2: The key size was a downgrade and I did not notice

ssh-keygen -t rsa -b 2048 -f /root/.ssh/id_rsa -N '' -q

-b 2048 reads like a sensible conservative default. It is not a default, and it is below the real one.

The ssh-keygen manual gives ed25519 as the default key type, and for RSA, "the minimum size is 1024 bits and the default is 3072 bits". So this command overrode the type to get RSA and then overrode the size downwards, from 3072 to 2048. Omitting -b entirely would have produced a stronger key than asking for 2048 did.

Worth putting beside Day 71, three days ago, where I used -t rsa -b 4096 and called 4096 an upgrade from 3072. Same series, same flag, used in opposite directions, and in neither case because I had checked what the default was. The honest conclusion is to omit -b and take 3072, or omit both flags and take ed25519.

-N '' is the flag that makes this password-less: it sets an empty passphrase, and a passphrase-protected key prompts on every connection unless an agent holds it. Real tradeoff rather than a convenience, though. The private key on disk is now the only thing between anyone who can read that file and the VM.

Step 3: Name the key rather than generating one

--ssh-key-values /root/.ssh/id_rsa.pub

Earlier tasks used --generate-ssh-keys, which is documented to "Generate SSH public and private key files if missing" and store them in ~/.ssh. Convenient, and it hides which of the two branches ran.

Here the first step had already settled the key, so naming it explicitly means the VM gets the key I just checked, with no conditional behaviour in between. Two steps that each do one thing.

Step 4: Password authentication came out disabled

"PasswordAuthDisabled": true

That is the observation that makes the task's title accurate, and it is worth separating from my explanation of it. I had written that creating a Linux VM with a key and no --admin-password makes Azure set disablePasswordAuthentication: true.

The observation is real. The mechanism is not documented in those terms. The ARM reference describes the property as specifying "whether password authentication should be disabled" and gives no default for it, where neighbouring properties do state theirs. What the CLI does document is --authentication-type, which "Defaults to password for Windows and SSH public key for Linux".

So the chain from "Linux VM, key supplied, no password" to "password authentication off" is consistent with the documentation rather than stated by it. Which I would rather say than present an inference as a fact.

The consequence holds regardless: there is no password to guess, so the VM is not exposed to credential stuffing against port 22 even with the NSG allowing SSH from anywhere. Passing --admin-password instead would have left password authentication on with port 22 open to the internet, which is the combination behind most compromised cloud VMs. It also means losing the private key locks you out, and recovery runs through az vm user update or Run Command.

What the docs recommend

The verification is the part I would reuse. Configuration and behaviour are different claims and need separate checks:

PasswordAuthDisabled: true          <- configuration
KeyPaths: [.../authorized_keys]     <- configuration

ssh -o BatchMode=yes ... 'whoami'   -> azureuser     <- behaviour

az vm show proves Azure was asked for the right thing. The SSH proves the guest accepted the key. And BatchMode=yes is what makes the second one mean "password-less" rather than just "worked": without it, a successful login is ambiguous, because ssh could have fallen back to a prompt that got satisfied somehow. With it, a prompt is impossible, so the key is the only explanation.

One last distinction, because it is easy to assume otherwise. This key is not an Azure resource. Day 1 created a Microsoft.Compute/sshPublicKeys object, reusable across VMs by name. Today's key exists only on the client machine and as a copy inside the VM's authorized_keys, and az sshkey list would return nothing. Both give working SSH; only one gives you something you can refer to later.

What I learned

Two files, two directions, and almost every SSH problem I have had is one of them in the wrong place.

The Azure task made it explicit by seeding authorized_keys and nothing else, so that a naive "is there a key here" check would say yes and be wrong. The Jenkins task made the same point by needing an outbound identity on one server and an inbound entry on another, and my own notes then failed to account for one of the two hops the job must have used.

The key size is the smaller finding and the more embarrassing one. I used -b 4096 three days ago and -b 2048 here, in both cases without checking that the RSA default is 3072. One was an upgrade, one was a downgrade, and neither was a decision.

So here is the Day 74 question. In the SSH setups you maintain, which machine holds an identity it no longer needs, and which one is still trusting a key whose owner left?

Day 74 down. Twenty-six to go.

Part 3: Nine Ways an HL7 v2 to FHIR Mapping Goes Quietly Wrong

Nnamdi Felix Ibe — Wed, 28 Oct 2026 19:26:21 +0000

Part 1 built the translation layer. Part 2 built the pipeline that checks it. This part is about the translation itself, which turns out to be where the interesting failures live.

None of what follows threw an exception. Every one produced a FHIR Bundle that a server would accept, containing the wrong thing. That is the particular danger of mapping work: the output is always well-formed, so the only way to find the problems is to run the mapper and look at what comes out.

I found nine. One of them stores a positive culture result as the number zero, and three of the nine I did not find at all: readers of Part 1 did.

The problem

A mapping is a long list of small decisions, and each one looks obviously right while you are making it. PID-5 is the name. PID-7 is the date of birth. You write the line, the test passes, you move on.

What makes it dangerous in a clinical setting is that the failures are silent and plausible. A missing middle name is not an error. A date of birth that vanished is not an error. The Bundle validates, the server returns 200, and the wrong record is now in the system of record with every appearance of being right.

So this week I stopped reading the mapper and started feeding it things.

What I found

Nine behaviours, each checked by running the real mapper on a real message:

behaviour                                before   after
--------------------------------------------------------
middle name kept                          FAIL     pass
partial date YYYYMM kept                  FAIL     pass
dob with tz offset kept                   FAIL     pass
escape sequence decoded                   FAIL     pass
fullUrls are real UUIDs                   FAIL     pass
patient resend updates, not duplicates    FAIL     pass
text result not made numeric              FAIL     pass
no JSON null in the resource              FAIL     pass
sex A maps to other, per HL7              FAIL     pass

A positive culture result became zero

This is the one I would want to know about if it were my trust.

An OBX segment carries a result. OBX-2 says what type the value is: NM for a number, ST for text, CE for a coded entry. The mapper ignored OBX-2 entirely and tried to turn every value into a valueQuantity. When the text would not parse as a number, it did this:

except ValueError:
    return {"value": 0, "unit": units or "", "comparator": None, "_raw": value}

So a microbiology result of POSITIVE came out as:

{ "valueQuantity": { "value": 0, "unit": "", "comparator": null, "_raw": "POSITIVE" } }

A qualitative result became the number zero. Anything downstream reading valueQuantity.value, a chart, a rule, a dashboard, sees a measurement of nought. _raw is not a FHIR element either. It was invented, so a conformant server is free to reject the resource or drop the property, and with it the only surviving copy of the actual result.

The fix is to read OBX-2 and pick the matching value[x]:

if vt in _CODED_VALUE_TYPES:
    return {"valueCodeableConcept": {"text": value}}
if vt in _NUMERIC_VALUE_TYPES or vt == "":
    try:
        quantity = {"value": float(value)}
    except ValueError:
        return {"valueString": value}
    ...
return {"valueString": value}

POSITIVE is now {"valueString": "POSITIVE"}. A segment that claims NM and carries text also becomes a string, because inventing a number is the failure mode I am trying to get away from.

Every admission created a new patient, and the obvious fix is the wrong one

The Bundle posted the Patient like this:

{ "method": "POST", "url": "Patient" }

An unconditional create. Send the same admission twice, get two patients. Resend a day's traffic after an outage, get a duplicate for every message.

This one came from a reader of Part 1. Anjan Kumar Gundaboina asked on LinkedIn whether the mapper was idempotent, or whether a resend produced a duplicate Patient. It produced a duplicate.

The obvious fix is a conditional create. FHIR has Bundle.entry.request.ifNoneExist, defined as "Instruct the server not to perform the create if a specified resource already exists". One match returns 200 and creates nothing, so a resend stops duplicating.

That is not enough, because an A08 is a patient update message. If the patient exists and the message carries a corrected date of birth, a conditional create looks at the one match and does nothing at all. The duplicate is gone and so is the correction.

A conditional update handles both. FHIR defines it as PUT [base]/[type]?[search parameters], where with no matches "The server creates the resource", with one match "The server performs the update against the matching resource", and with several it returns 412.

{
  "method": "PUT",
  "url": "Patient?identifier=https://fhir.nhs.uk/Id/nhs-number|9000000009"
}

When there is no NHS number, there is no condition and it stays a plain create. A condition that matches everything is worse than none: it would merge every patient without an NHS number into one record.

The dates it threw away

PID-7 is a time stamp, not a date, and senders fill in as much as they have. The mapper accepted YYYYMMDD and returned nothing for anything else.

Readers raised this one too. Lee Wilson pointed out that in many EHRs only the year of birth is known, and Anjan named the partial-date fallback as exactly the kind of silent failure he would want a dashboard on.

FHIR is more accommodating than the mapper was. It defines date as "A date, or partial date (e.g. just year or year + month)" in the format "YYYY, YYYY-MM, or YYYY-MM-DD".

So 198001 should be 1980-01. It was becoming nothing at all. So was 19800101120000+0100, a perfectly ordinary full time stamp with a time zone offset, because the offset did not match any of the three formats being tried. A date of birth with a time zone on it was silently dropped.

Both now map. The offset is stripped, because a FHIR date "SHALL be no time zone".

The names it mangled

HL7 reserves five characters as delimiters and defines five escape sequences so they can appear inside a field anyway. \T\ is the subcomponent separator, so O\T\BRIEN is how the wire carries O&Brien. The mapper stored the escape sequence.

One subtlety is worth getting right. \E\ stands for the escape character itself, so decoding has to be a single pass. Replace \F\ first and then \E\, and the sequence \E\F\E\, which means the literal text \F\, becomes a field separator instead. Sequences outside the five, like \H\, are passed through rather than stripped.

The middle name, and the test that approved of losing it

PID-5 is DOE^JOHN^A. Component 3 is the middle name. FHIR puts given and middle names together in given, in order.

The mapper read component 2 and stopped, so A was dropped. That is the least serious of the nine and the most instructive, because of what I found in the test file:

assert patient["name"][0]["given"] == ["JOHN"]

The test asserted the bug. It had passed since the day it was written, because it was written by reading what the mapper did rather than what the segment said. A green suite told me the mapping was right, and what it actually told me was that the mapping had not changed.

The identifiers that were not identifiers

Each Bundle entry had a fullUrl like urn:uuid:patient-1. FHIR's uuid type is "A UUID (aka GUID) represented as a URI", meaning RFC 4122. patient-1 is not one. HAPI accepted them, which is why it survived; a stricter server need not. They are real UUIDs now.

Nulls where FHIR forbids them

When PID-7 would not parse at all, birthDate came out as None and serialised to "birthDate": null. FHIR's JSON rules say properties "never have null values" and to "Omit a property if it is empty".

Testing that turned up a second case: a PID with no name at all produced "family": "". The same page says "String property values can never be empty".

Both are now pruned on the way into the Bundle, along with empty arrays and objects. Zero and false survive the pruning, because a quantity of 0 is a real measurement and that is the mistake this article started with.

The answer I said did not exist

I had written PID-8 up as an open question. HL7 table 0001 carries A for ambiguous and N for not applicable, FHIR's administrative-gender has only male, female, other and unknown, and I concluded there was no clean target and left the mapper sending both to unknown.

Then Dalisay Giffard suggested on LinkedIn that I check the PID-8 mapping, because administrative sex has a different meaning to gender. The field held up: FHIR R4 defines Patient.gender as administrative gender, and that is where HL7's v2-to-FHIR guide puts PID-8. The codes did not. Checking them properly, I found that HL7 publishes a concept map for exactly this, in the v2-to-FHIR implementation guide. It sends A to other and N to other, and rates both "is equivalent to".

Which is obviously right once you see it. unknown says we do not know. The sender did know; the answer was simply neither male nor female. I had been treating a recorded answer as a missing one, and I had reasoned my way to a conclusion the standard had already settled in a table I had not looked for.

The map was not the territory

There is a docs/hl7-fhir-mapping.md in the repo, a careful table of HL7 fields and their FHIR targets. I checked each row against the running mapper. Three described mappings that do not exist: PID-11 to Patient.address, RXE-4 to dosageInstruction, RXE-13 to requester. It also called _raw an extension, which gave an invented property the air of deliberate design.

That is three weeks running where the most useful finding was a document describing a system nobody had checked. Week 2 it was a security scanner with no rules loaded. This week a mapping table describing a mapper I had not written.

The rewritten version says "not mapped" where that is the truth, and every row was produced by running the thing.

What readers found that I have not fixed yet

Part 1 and Part 2 are public, and so is the repo, which means the mapping has had more eyes on it than mine. Several more things came back that are not in this week's change, named here so they do not quietly disappear.

PID-3, raised by Kevin Mayfield, Ian Haywood and Lee Wilson. PID-3 reads only CX-1 and CX-4. The CX-5 identifier type code, NH or MR, is dropped, there is no use, and every non-NHS identifier lands in one local-patient-identifier system with no assigner, so medical record numbers from two different trusts collide. The North West GMSA HL7 v2 guide treats CX-1, CX-4 and CX-5 as mandatory and uses the ODS code as the assigning authority. My fixture uses GENERAL_HOSP, which is not an ODS code. Ian's warning was the practical one: unless the namespace carries through, you'll have fun later when another source uses the same ID. Kevin added that the MRN system should be a URI or an OID, that EPR vendors and trusts have each made their own choice, and that his organisation uses a URI.

Replay ordering, which follows from Anjan's question. Now that writes update rather than create, replaying an older dead-lettered message after a newer one would overwrite newer data with older. The fix I have in mind is to check MSH-7 against what was last applied, at the point of the write. Using the NHS number as an SQS FIFO message group ID orders live traffic but does nothing for a replay out of the dead-letter folder.

Field-level failures are silent, which Anjan asked about directly. Only a whole-message parse failure dead-letters. A single field that fails to map produces no flag, no metric and no log line, which is how all nine of these survived.

No canonical model, which Mitchel Coon asked about. The mapper goes straight from segments to FHIR resources, with no intermediate representation, so every quirk is handled inline. HL7's own v2-to-FHIR guide maps directly too, so this is defensible, but it is why the fixes above are scattered across the file rather than sitting in one place.

Lee also pointed out that the NHS number is not guaranteed, that address fields are often misused, and that phone fields can hold free text like "don't call after 6pm". Without an NHS number the Patient falls back to a plain create, so those patients still duplicate. PID-11 and PID-13 are not mapped at all yet, so the free text is a problem waiting for whoever maps them.

The one thing still open

PID-8 is settled and the duplicates are fixed. One judgement call is left, and I would rather leave it visible than decide it quietly in code.

The Encounter has no identifier and no period, so a resent admission duplicates it exactly the way the Patient used to.

Mustafa Erbay made the right suggestion in the comments on Part 1 here on dev.to: use PV1-19, the visit number, rather than building a composite key out of the fields that happen to be present. It is the natural identity for a visit and it is what the segment is for. The reason it is not in this week's pull request is the case he and I did not settle, which is what to do when PV1-19 is absent, since the standard does not require it. A conditional create with an empty condition matches every Encounter that also lacks a visit number and merges them, which is a worse failure than the duplicates I have now. So the duplicates stay until the absence case has an answer.

Worth stealing

Feed the mapper, don't read it. Every one of these nine produced valid output. Reading the code tells you what it says. Running it tells you what it does, and in mapping work those come apart constantly.

Write the test from the specification, not from the output. The middle-name test passed for months because it was written by running the code and pasting the result. A test written from PID-5's definition would have failed on day one.

Check the mapping document against the mapper. A mapping table is the thing colleagues trust most and verify least, because it reads like a specification when it is actually a description, and descriptions drift.

Look for the concept map before you reason from first principles. I decided PID-8 had no clean FHIR target and wrote that conclusion into a document. HL7 had already published the answer as a table. The standards bodies have done more of this work than it is comfortable to assume.

Let the message decide the shape. OBX-2 exists precisely to tell you what kind of value you are holding. Guessing with float() and falling back to zero is how a culture result becomes a number.

Where AI helps here, and where it doesn't

The fixes were quick to write once I knew what was wrong. The escape decoder, the date precision logic, the OBX-2 branch: all small, well-specified, mechanical.

Finding them was a different activity, and it was not reading. It was constructing awkward messages, running them through, and looking at the JSON. A patient with a middle initial. A date of birth with a time zone. A name with an ampersand. A culture result that says POSITIVE. None of those are exotic, and every one of those defects is invisible in a code review, because each line is individually reasonable.

That is now three weeks where running the thing found what reading it could not. I am going to stop describing it as a surprise.

The other half is less comfortable. Three of the nine I did not find by any method, and would not have: they came from people who read Part 1 and asked questions I had not thought to ask. Publishing the thing in an unfinished state is the only reason those arrived at all, which is an argument for writing the series while the work is still wrong rather than after it is right.

What I'd do differently

I would have written the mapping document and the tests from the HL7 and FHIR specifications first, and the mapper second. I did it the other way round, and ended up with a test suite and a document that both described the implementation back to me. Two sources of confirmation, neither of them independent.

Sources

Part 4, on 4 November, is weekly rather than fortnightly from here, and it is about observability, including an alert in this repo that has never been able to fire.

Which of your mappings has a test that was written by pasting in what the code already did?

This is a personal project. It is not affiliated with or endorsed by NHS England or any NHS organisation. It uses the public PDS FHIR sandbox and synthetic test data only. Views are my own.

Day 73: Six Permissions and a Repeat

Nnamdi Felix Ibe — Wed, 28 Oct 2026 18:00:00 +0000

Day 73 of DevOps, Day 23 of Azure. Both tasks today looked uniform and were not.

Six permissions in one list turned out to come from three different places in Jenkins. And an Azure deployment I had run the day before, with the same script and the same image, behaved differently enough to change which verification I would trust.

One Jenkins task, one Azure task. Give two users different permissions on one job, then repeat yesterday's self-configuring VM from the CLI. The tasks come from the KodeKloud Engineer platform.

Task 1: Two users, one job, different access

The task: grant Rohan build, cancel, configure, read, update and tag on an existing job, and grant Sam build, configure and read.

Step 1: Check, do not create

The task says to grant permissions to users. It does not say to create them, and both already existed.

Worth being deliberate about, because creating a second rohan leaves the grant attached to an account nobody logs into, and the symptom is a user who still cannot do anything. That reads like a permissions problem for as long as you let it.

Step 2: Install the plugin, again

Matrix Authorization Strategy is not part of Jenkins. Its own page describes it as offering "matrix-based security authorization strategies (global and per-project)", and on a stock Jenkins the Security page has no matrix option at all.

Same finding as Day 70, and it is worth repeating because the task's wording assumes it. A task that says "using a project-based matrix authorization strategy" is describing a plugin, and the authorisation half cannot start until it is installed.

Step 3: The step the task does not mention

Grant Overall/Read to authenticated users, globally, before touching the job.

Without it, logging in as Rohan gives an access-denied message naming the missing Overall/Read permission, so you cannot verify anything. The usual shorthand for this is that nothing works without Overall/Read, and that overstates it.

Jenkins is careful here: "While Overall/Read is a prerequisite for more extensive access to Jenkins, some features are available without it." The plugin says the same from the other side: "A user not granted read access to Jenkins in general will not be able to use most of the other permissions."

Most, not all. So the accurate version is that Overall/Read gates almost everything a person would recognise as using Jenkins, and granting it to authenticated users is a real grant rather than a formality. Every logged-in user can then see that the instance exists and enumerate what they have.

If you want to avoid that, Jenkins has a narrower tool. Job/Discover is documented as "only useful if anonymous users have Overall/Read permission, but not Job/Read", which is the shape for hiding specific jobs from users who can still reach Jenkins.

Step 4: The six permissions come from three places

This is the part I would have got wrong by reading the grid's column labels.

Jenkins groups its permissions as Overall, Agent, Job, Run, View and Credentials. The Job group is Build, Cancel, Configure, Create, Delete, Discover, Move, Read and Workspace. Four of the task's six names map straight onto it:

Task name	Documented permission
`build`	Job/Build
`cancel`	Job/Cancel
`configure`	Job/Configure
`read`	Job/Read

The other two do not.

There is no Job/Update. The only documented Update permissions are Run/Update and Credentials/Update, and the one that appears in a per-job grid is Run/Update, which sits in the Run group and concerns individual builds rather than the job's configuration.

And there is no documented permission called Tag anywhere on Jenkins' permissions page. Tag comes from a plugin, and shows up in the grid only when that plugin is installed. I could not work out which plugin supplies it in this lab, so I am leaving that as unknown rather than filling it in.

Which makes the lesson a practical one rather than a pedantic one. A permission grid looks like one decision and is several, and a column header does not tell you whether you are granting something about the job or about its builds. Read the grid by column group, not by column label.

Step 5: Log in as both of them

The grid tells you what was ticked. It does not tell you what works.

Logging in as Sam and confirming the job can be configured and built, then as Rohan and confirming the same plus what he has extra, is the only check that distinguishes a configured grid from a working one. Day 70 made this point for a single read-only user, and it is stronger with two users who are supposed to differ.

What the docs recommend

Granting on the job rather than globally is the whole point, and the documentation explains why in one line. Global grants apply everywhere: "Permission applying to such items or agents that are granted in the global configuration apply to all of them." Per-item grants sit on top of that, additively, under the default mode: "Permissions explicitly granted on individual items or agents will only add to permissions defined globally."

So Rohan's six permissions granted globally would have given him them on every job, present and future. On the job, the scope stays where the task put it, and the global grid carries only the Overall/Read everyone needs.

There is one thing in this task worth more worry than the permission names, though, and it is that both users got Configure.

A freestyle job's configuration is its commands. Job/Configure lets a user change the build steps, which means anything in those build steps is visible to and editable by that user, including a password typed into a shell command. If the job runs on the controller rather than an agent, that command runs where Jenkins itself lives.

That is not a hypothetical in this series. Day 76's job puts a database password directly into its shell step. Anyone with Configure on it can read that password. Granting Configure is closer to granting shell access than the word sounds, and the fix is to keep secrets out of job configuration rather than to be careful about who gets Configure.

Task 2: Yesterday's VM, from the CLI

The task: the same self-configuring nginx VM as Day 22, named datacenter-vm, with the region named explicitly.

Everything from yesterday carried over. The Azure CLI base64-encodes --custom-data. --nsg-rule accepts only {NONE, RDP, SSH}, so port 80 needs az vm open-port. Cloud-init dispatches on the script's first line. And --storage-sku Standard_LRS is mandatory in this subscription, because the policy refuses whatever the undocumented default produces.

Step 1: Pass the region anyway

The resource group is already in eastus, so omitting --location would have worked. I passed it because the task names a region, which is the Day 4 rule: inheriting a default that happens to match is luck, not correctness.

eastus is the CLI name. "East US", as the task writes it, is the display name, and the CLI rejects it.

Step 2: The same commands took a different amount of time

Day 22:  attempt 1: 000    attempt 2: 000    attempt 3: 200
Day 23:  attempt 1: 200

Same script, same image, same size, same region, one day apart. Yesterday nginx needed about 40 seconds after az vm create returned. Today it answered immediately.

Nothing changed on my side. The variance is in apt-get update and apt-get install, which depend on mirror latency and package cache state on the day.

That is the entire argument for polling a condition rather than sleeping an interval. A sleep 45 tuned to yesterday wastes 45 seconds today. A sleep 20 tuned to today fails yesterday.

The sharper version: a single immediate curl would have passed today and failed yesterday. That is the worst kind of check, because it works often enough to earn trust it has not demonstrated. A check that fails loudly is worth more than one that is usually right.

What the docs recommend

The task asked for two things that sound like one: HTTP reachable from the internet, and the nginx service running. Three checks, three different claims:

Check	What it proves
`curl` returns 200	Reachable now, end to end: NSG rule, public IP, listening process, content
`systemctl is-active`	Running now, independent of the network path
`systemctl is-enabled`	Still running after a reboot

curl subsumes "running", since a stopped service cannot answer. What it cannot reach is is-enabled, which reports whether nginx starts at boot. That is invisible over HTTP, and it only becomes visible after the next restart, by which point nobody is looking.

systemctl enable --now in the boot script set both. Only the direct check confirms the enable half took effect, and az vm run-command invoke made that possible without SSH.

What I learned

Two lists that looked uniform and were not.

Rohan's six permissions read as one decision about one job. They are four permissions about the job, one about its builds, and one that Jenkins does not document at all. Nothing in the grid says so. You find out by reading the column groups instead of the column labels, or by granting something and discovering later it did not mean what you thought.

And two runs of the same deployment, with every input identical, differed by about 40 seconds in whether the thing I was checking for existed yet. Which is the more useful of the two findings, because it tells you something about verification rather than about Azure: if a check would have passed today and failed yesterday, it was never a check. It was a coin flip that happened to land well.

So here is the Day 73 question. Which of your verification steps would still pass if the thing it verifies had not finished yet?

Day 73 down. Twenty-seven to go.

Day 72: A Parameter and a Boot Script

Nnamdi Felix Ibe — Mon, 26 Oct 2026 18:07:58 +0000

Day 72 of DevOps, Day 22 of Azure. Both tasks today were about a value crossing from outside into something that runs it, and what happens at the boundary.

A Jenkins parameter crosses into a shell script. A boot script crosses into a brand new VM. In both cases, the interesting part is not the value, it is the handoff, and yesterday's bug turns out to explain today's safety.

One Jenkins task, one Azure task. Add two parameters to a job and use them, then build a VM that configures itself at first boot. The tasks come from the KodeKloud Engineer platform.

Task 1: Two build parameters

The task: a job named parameterised-job, with a String parameter Stage defaulting to Build, and a Choice parameter env offering Development, Staging and Production. Then a shell step that uses them.

Step 1: Add them, and notice the documentation gap

Both go under This project is parameterised. That part is a form.

What is missing is any documentation of it. Jenkins' Pipeline docs describe the parameters directive, show a string parameter, and say parameters "are accessible as members of the params variable". They never mention choice parameters, and they never mention freestyle projects at all. This is the second day running that the freestyle feature a task asks for has no page: Day 71 had the same gap for the Execute shell step.

There is exactly one documented sentence connecting the two halves, and it is the one that matters: "parameterised freestyle projects pass parameter values to Shell and Batch build steps as environment variables."

So nothing in the script declares anything. Jenkins exports Stage and env into the build's environment, and the shell expands them.

Step 2: The Choice parameter is yesterday's recommendation

Day 71's job took a free-text package name and interpolated it into a command on a remote host with passwordless sudo. I said a validated choice list was the fix and had no example to point at. This is the example.

env can only ever be one of three strings. The set of possible build behaviours is finite, and it is visible in the job configuration rather than in whoever typed into the box.

Stage is still free text, and its default value is what makes the job runnable without input. Worth knowing that a String parameter with no default is an empty string rather than an error, so a script that assumes a value will get one, and it will be empty.

Step 3: Why this is safer than Pipeline, by accident

Jenkins documents the injection risk for Pipeline carefully, and the mechanism it names is specific. The hazard is Groovy string interpolation: "Groovy string interpolation can inject rogue commands into command interpreters via special characters", which "can result in problems analogous to SQL injection".

The worked example is good. Give a string parameter the default hello; ls /, then write sh("echo ${STATEMENT}"), and the ls runs. Groovy substitutes the value into the command text before the shell ever sees it, so a user who controls that value "would be able to make the sh step run arbitrary code on the agent".

The documented fix is to stop substituting and let the shell read the value instead. Arguments referencing parameters should "use single quotes to avoid Groovy interpolation", so Jenkins sends the literal command string and the shell picks the value out of its environment.

A freestyle Execute shell step is already in that second form. There is no Groovy layer. Jenkins exports the parameter, and $Stage in the script is a plain shell parameter expansion.

And the shell's own ordering is what makes that safe. Bash performs expansions in a fixed order, "brace expansion, then tilde, parameter and variable, arithmetic, and command substitution (left to right), then word splitting, then filename expansion, and finally quote removal". Of all of those, "Only brace expansion, word splitting, and filename expansion can increase the number of words."

Parameter expansion is not on that list. A value containing ; or && expands into characters sitting inside a word, not into new command separators, because the shell decided where its commands began when it broke the input into words and operators, and that happened before any expansion.

Which is yesterday's bug seen from the other side. On Day 71, ssh host cmd1 && cmd2 split into two commands on two machines because the && was in the script text, tokenised before ssh ran. Today, an && arriving inside a parameter value cannot split anything, for the same reason and in the same order.

Tokenise, then expand. Text you write in the script is structure. Text that arrives in a variable is data. One rule, and it cuts both ways.

What the docs recommend

None of that makes an unquoted expansion fine. Word splitting and filename expansion do still apply, and they are the next two steps in that same list:

echo "Stage is $Stage, environment is $env"

Unquoted, a value with spaces becomes several arguments, and a value containing * gets expanded against the working directory. Those are ordinary bugs rather than remote code execution, and they are still bugs.

One naming note. env is also a standard Unix command, and in Jenkins Pipeline it is the global environment namespace. As a freestyle parameter it collides with nothing, but the same name in a Pipeline would read as though it should. TARGET_ENV says the same thing and costs nobody an afternoon.

Task 2: A VM that configures itself

The task: an Ubuntu VM named nautilus-vm that installs and starts nginx on first boot, reachable on port 80.

Step 1: Pass the disk flags, and a correction about why

--storage-sku Standard_LRS --os-disk-size-gb 30

Day 21 failed on exactly this, and nothing in today's task mentions disks, so the same default would have applied, and the same policy rejection would have followed. First time in this track that a recorded failure has prevented its own repeat rather than being rediscovered.

My explanation of the default was wrong, though, and it is worth correcting because it is in two of my notes. I had written that az vm create defaults the OS disk to Premium_LRS on premium-capable sizes.

The CLI reference lists allowed values for --storage-sku and gives no default at all. The only hardcoded Premium_LRS in the CLI's own validation applies to the old unmanaged disk profiles, and the managed-disk path, which is the default, sends no storage account type whatsoever. Something below the CLI chooses, and the rule it uses is not written down anywhere I can read.

The finding survives, the mechanism does not: omit --storage-sku in this subscription and you get a Premium disk and a policy refusal. Specify it.

Step 2: The CLI encodes the script for you

--custom-data /root/cloud-init.sh

Microsoft puts the requirement and the exemption in one sentence: "To use custom data, you must Base64-encode the contents before passing the data to the API--unless you're using a CLI tool that does the conversion for you, such as the Azure CLI. The size can't exceed 64 KB."

Worth setting beside the AWS equivalent, because the asymmetry bit me on Day 44:

Call	Encoding
`aws ec2 run-instances --user-data file://...`	Automatic
`aws ec2 create-launch-template` UserData	Manual. Raw text succeeds and never runs
`az vm create --custom-data`	Automatic

The launch template case is the dangerous one, because nothing fails. The template is created, the instance launches, the script simply never executes.

Step 3: Custom data or user data

Azure has both flags and they are not the same field. My notes hedged, saying which one a given image acts on varies by image and cloud-init version. That hedge was unnecessary and I could not find anything supporting it.

Microsoft draws the line by purpose: "Unlike custom data, user data is retrievable from the Azure Instance Metadata Service (IMDS), is persistent for the lifetime of the VM, and can be updated without stopping or reimaging the VM. Use user data when you need post-provision access or updates; use custom data when you need a payload processed by the provisioning agent at first boot."

So custom data is the one the provisioning agent runs at first boot, which is what this task wanted. Not the safer of two uncertain options, just the right one.

One warning attaches to custom data and not to user data: "We advise not to store sensitive data in custom data."

Step 4: Port 80 needs its own command

--nsg-rule cannot do it. Its accepted values are {NONE, RDP, SSH}, full stop, so an arbitrary port goes through a second call:

az vm open-port -g $RG -n nautilus-vm --port 80 --priority 900

open-port-80       80   900   Allow
default-allow-ssh  22   1000  Allow

Priority 900 sits ahead of the SSH rule deliberately. Nothing conflicts today, so order does not matter today. It matters in general, because NSG rules are evaluated in priority order and the first match wins, so a Deny landing between two Allows changes everything. Leaving the low numbers free for rules that need to win is the habit worth having.

My notes also claimed az vm open-port defaults to priority 900 and to * for the port. I could not verify either. Passing both flags explicitly makes the question moot, which is reason enough to pass them.

What the docs recommend

The script is four lines, and its first line is the one that decides whether any of it happens:

#!/bin/bash
apt-get update -y
apt-get install -y nginx
systemctl enable --now nginx

Cloud-init dispatches on that first line. #! means run it as a script, #cloud-config means parse it as YAML, and neither means it is ignored. Which is the same silent failure as unencoded AWS user data, arriving by a different route.

Then the thing that actually caught me:

attempt 1: 000
attempt 2: 000
attempt 3: 200

az vm create reported VM running straight away. The VM did not serve HTTP for about another 40 seconds, while cloud-init ran apt-get update and installed nginx. A 000 from curl means no TCP connection at all, so for those 40 seconds the port was open at the NSG with nothing listening behind it.

This is Day 38's lesson in its mildest form. There an ECS task reported RUNNING and the service was never reachable, and I failed the task because verification stopped at the status field. Here the gap is 40 seconds rather than forever, and the shape is identical: the control plane reports the resource, not the application.

The retry loop is the right instrument, and not because it is more patient than a sleep. It makes the gap visible rather than hiding it, and it tells "not ready yet" apart from "broken" by showing progress. Had it stayed at 000, cloud-init status separates the three possibilities that matter: still running, finished, or finished with an error.

What I learned

Both halves came down to the same question: at what point does a value stop being text and start being structure?

In the Jenkins job, the answer is that the shell decides where commands begin before it expands any variable. That is why Day 71's && broke a command in two, and it is why an && inside today's parameter cannot. Same rule, opposite outcomes, and knowing the order is what lets you predict which one you get.

On Azure, the answer is that the first line of a boot script decides whether the rest of it is a script at all, and the CLI decides whether the payload arrives in a form the agent can read. Get either wrong and nothing errors. You get a running VM that quietly did none of what you asked.

The disk SKU is the third version of the same thing, and the one I had explained wrongly. A value I never wrote became part of the request, something I cannot read chose it, and the policy refused it.

So here is the Day 72 question. In your build scripts, which values are you writing into the command text that should be arriving through the environment instead?

Day 72 down. Twenty-eight to go.

Day 71: One Command, Two Machines

Nnamdi Felix Ibe — Sun, 25 Oct 2026 09:00:00 +0000

Day 71 of DevOps, Day 21 of Azure. Both tasks today turned on the same mistake: treating one line of a command as one unit of work.

A Jenkins build step I wrote as a single remote command was actually two commands on two different machines, and only one of them reached the server I meant. An az vm create I thought of as one command turned out to be a deployment of six resources, so when it failed, five of them stayed behind.

One Jenkins task, one Azure task. Build a parameterised job that installs a package on a remote host, then gives a VM a static public IP. The tasks come from the KodeKloud Engineer platform.

Task 1: A Jenkins job that installs a package

The task: a freestyle job named install-package, with a string parameter PACKAGE, which installs the named package on the storage server ststor01.

Step 1: Find out what the target runs

cat /etc/os-release

NAME="CentOS Stream"
VERSION="9"
ID="centos"
PLATFORM_ID="platform:el9"

CentOS Stream 9, so dnf. The Jenkins node, from Day 68, is Ubuntu 24.04, so apt. Two package managers in one job, which is the detail that makes the next step go wrong.

Step 2: The parameter, and how it reaches the shell

A String parameter named PACKAGE, then a shell build step using $PACKAGE.

That works because of something Jenkins documents directly: "parameterised freestyle projects pass parameter values to Shell and Batch build steps as environment variables." The parameter is not substituted into the script by Jenkins. It is exported, and the shell expands it.

Worth noting what is not documented. Jenkins' own site has no page describing the freestyle "Execute shell" build step, its shell, or its exit-status behaviour. The documented equivalent is Pipeline's sh step, which "Runs a Bourne shell script", uses "the system default shell ... using the -xe flags", and fails the build when "a script which exits with a nonzero status code" runs. Freestyle behaves the same way in practice. I could not cite that from Jenkins.

Step 3: A key for the storage server

Jenkins has no credentials for ststor01, so:

ssh jenkins@jenkins
ssh-keygen -t rsa -b 4096
ssh-copy-id natasha@ststor01

The key has to belong to the jenkins user, because that is who the build runs as. Logging in as root and generating a key in /root/.ssh produces a key the build cannot see. ssh-keygen offering /var/lib/jenkins/.ssh/id_rsa is how you know you are the right user, since JENKINS_HOME is /var/lib/jenkins.

-t rsa -b 4096 also overrides two defaults, and not the ones usually assumed. The ssh-keygen manual gives the default key type as ed25519, and for RSA, "the minimum size is 1024 bits and the default is 3072 bits". So 4096 is an upgrade from 3072, not from 2048.

sudo then works over that connection without a password because natasha has a passwordless rule, and it has to, because there is nowhere to type one. The ssh manual: ssh "by default will only request a pseudo-terminal (pty) for interactive sessions when the client has one". Supplying a command is not an interactive session, so there is no terminal for a prompt to appear on.

I had attributed that to sudo's requiretty. It is a different thing. Requiretty "will only run when the user is logged in to a real tty", it "is off by default", and it governs whether sudo runs at all, not whether it can prompt.

Step 4: The build step was wrong

This is what went into the job:

ssh natasha@ststor01 sudo dnf makecache && sudo dnf install -y $PACKAGE

Read as one remote command, it makes sense. It is not one remote command.

&& belongs to the local shell. Bash's manual: "A list is a sequence of one or more pipelines separated by one of the operators ';', '&', '&&', or '||'", and the shell "Breaks the input into words and operators" before it runs anything. So ssh never receives && as an argument at all. By the time ssh starts, the line has already been split.

ssh builds its remote command only from the arguments it does receive. Its synopsis is destination [command [argument ...]], and "If a command is specified, it will be executed on the remote host instead of a login shell."

Which means the line runs on two machines:

sudo dnf makecache              -> on ststor01
sudo dnf install -y $PACKAGE    -> on the Jenkins node

And the Jenkins node is Ubuntu. It has no dnf. The install half never ran anywhere it could work.

The fix is to quote the whole remote command:

ssh natasha@ststor01 "sudo dnf makecache && sudo dnf install -y $PACKAGE"

Double quotes, deliberately, not single. The local shell expands $PACKAGE before sending, which is what is wanted here, because the Jenkins parameter only exists on the Jenkins side. Single quotes would send the literal $PACKAGE, and the remote shell would expand it to nothing.

One thing I cannot reconcile, and would rather say than quietly tidy up. My own notes record the build succeeding, and vim-enhanced.x86_64 2:8.2.2637-27.el9 present on ststor01 afterwards. The command as I wrote it down cannot produce that. The likeliest explanation is that the build step in Jenkins was not identical to the line I copied into my notes, which is the gap a screenshot-driven write-up leaves. The corrected form above is the one to use either way.

What the docs recommend

The risk in this job is not the package manager. It is that $PACKAGE is free text that ends up inside a command line on a remote host, with passwordless sudo at the other end.

Jenkins warns about exactly this shape. Using a Windows example: "A variable value that contains valid Batch commands may result in the execution of these commands by Jenkins", and "special care may need to be taken to filter or sanitise environment variable values". Then the general case: "Any user who can add environment variables with a name they choose may be able to modify the behaviour of builds this way."

So anyone who can trigger this build can run an arbitrary command on the storage server as a user with passwordless sudo. A validated choice parameter instead of a free string closes most of it. And as on Day 69 and Day 70, the job's definition is worth living in a file where that constraint is reviewable, rather than in a form where it is one edit away from gone.

Task 2: A static public IP for a VM

The task: datacenter-vm, Ubuntu, Standard_B1s, with an SSH key and a static public IP named datacenter-pip, reachable over SSH.

It took four attempts, and the sequence is the interesting part.

Step 1: The default OS disk is banned

RequestDisallowedByPolicy: Resource 'datacenter-vm_OsDisk_...' was disallowed by policy.
Reasons: 'Global: This storage configuration is not allowed. Ensure the disk size is
128 GB or less and the SKU is not Premium for Compute disks.'

The fix was to stop leaving the disk at the default:

--storage-sku Standard_LRS --os-disk-size-gb 30

Day 2 and Day 3 both passed --storage-sku Standard_LRS and both worked, not because I knew about the policy but because those tasks specified a disk type. This one said nothing about disks, so the default applied, and the default is the one thing this subscription refuses.

My explanation of which default was wrong, and it was specific enough to be worth correcting. I had written that az vm create picks Premium_LRS whenever the VM size supports premium storage, which Standard_B1s does.

The CLI does not do that. --storage-sku has no documented default, and the only hardcoded Premium_LRS in the CLI's validation applies to the old unmanaged disk profiles rather than to managed disks. On the managed-disk path, which is the default, the CLI sends no storage account type at all and leaves the Compute service to choose. What it chooses is not stated in the CLI reference.

Which makes the lesson stronger, not weaker: something downstream picked a Premium disk, and I cannot read the rule it used. In a policy-constrained subscription, the values you do not specify are the ones most likely to be rejected, and the mechanism behind them may not be documented anywhere you can check.

Step 2: az vm create does not roll back

The failed attempt left this:

datacenter-pip      Microsoft.Network/publicIPAddresses
datacenter-vmVNET   Microsoft.Network/virtualNetworks
datacenter-vmNSG    Microsoft.Network/networkSecurityGroups
datacenter-vmVMNic  Microsoft.Network/networkInterfaces
datacenter-vm       Microsoft.Compute/virtualMachines    <- provisioningState: Failed

az vm create is one command and six resources. When the VM fails, the five that already exist stay, and the VM itself remains as a resource in Failed state.

That there is no rollback is not documented for the command. The surrounding behaviour is the same, and it points the same way: ARM's undo is opt-in. --rollback-on-error is "The name of a deployment to roll back to on error, or use as a flag to roll back to the last successful deployment", with a warning that it "rollbacks a failed deployment by redeploying an earlier deployment" and the result "may be different than what you would expect from undoing the failed deployment". ARM replays. It does not reverse. And it does nothing unless asked.

This is Day 70's finding showing its consequences. An ARM deployment is a record of an operation, not an owner of resources, so nothing owns the half-built output and nothing cleans it up.

CloudFormation is the other way round. Day 47 failed on an IAM permission and rolled itself back, because OnFailure defaults to ROLLBACK. The price there was that the stack had to be deleted before retrying. The price here is cleaning up by hand.

Step 3: My own cleanup caused the next two failures

OperationNotAllowed: Operation 'Update VM' is not allowed on VM 'datacenter-vm'
since the VM is marked for deletion.

Caused by this:

az vm delete -g $RG -n datacenter-vm --yes 2>/dev/null

Two mistakes on one line. 2>/dev/null hid whether it worked at all. And Azure's control-plane operations are asynchronous, so success means started, not finished: "Some Azure REST operations run asynchronously because the operation can't be completed quickly", returning a status code and a header to poll. I then ran az vm create against a VM that was still being deleted.

The obvious waiter did not help:

az vm wait -g $RG -n datacenter-vm --deleted
The operation failed

It reported failure while the VM was still listed. What --deleted actually polls is not documented; the help says "Wait until deleted." and gives an interval and a timeout. What I can say is what it did here, which is give up on a resource that was already in a failed state.

What worked was testing the only thing that matters:

az vm delete -g $RG -n datacenter-vm --yes; echo "delete exit=$?"
until ! az vm show -g $RG -n datacenter-vm >/dev/null 2>&1; do sleep 10; done

until ! az vm show asks whether the resource still exists, rather than asking a waiter for its opinion about state.

Step 4: The IP, which was the point

Created as its own resource first, then referenced by name:

az network public-ip create -g $RG -n datacenter-pip --sku Standard --allocation-method Static
az vm create ... --public-ip-address datacenter-pip

The address did not change across four attempts:

Attempt 1 (policy failure):  20.119.41.127
Attempt 4 (success):         20.119.41.127

Because deleting a VM does not delete a public IP. Microsoft: "By default, disks, NICs, and Public IPs associated with a VM are persisted when the VM is deleted." And the CLI documents the dual behaviour of the flag: --public-ip-address is the "Name of the public IP address when creating one (default) or referencing an existing one".

Had I let az vm create generate the IP, each failed attempt would have produced a different address, and every retry would have invalidated whatever DNS record or firewall allowlist pointed at the last one. Which is the problem the task exists to solve.

--allocation-method Static is redundant with --sku Standard, since Standard's allocation method is Static with no alternative. I passed it anyway, so the requirement is visible in the command rather than implied by a SKU.

What the docs recommend

Three resources survived my cleanup on purpose:

datacenter-pip      the whole point, keep it
datacenter-vmVNET   az vm create reuses a VNet of this name
datacenter-vmNSG    az vm create reuses an NSG of this name

az vm create generates names from the VM name, {vm}VNET, {vm}Subnet, {vm}NSG, {vm}PublicIP and {vm}VMNic, and for most of them it will reuse an existing resource of that name instead of failing. The reference says so for the NSG: "The name to use when creating a new Network Security Group (default) or referencing an existing one."

The NIC is the exception, in the CLI's source and in practice. It holds the private IP reservation and the public IP association, so a new VM cannot adopt one already sitting there under the generated name. That is why only the VM, its NIC and its disk had to be removed between attempts.

Two more small things that only show up on a retry. --generate-ssh-keys printed its message on the first attempt, the one that failed, and the later attempts reused the keys, because it is documented to "Generate SSH public and private key files if missing" and stores them in ~/.ssh. And azureuser rather than the local username, because the local user here is root, and root is on Azure's list of disallowed Linux admin usernames. The CLI anticipates that: "Default value is current username of OS. If the default value is system reserved, then default value will be set to azureuser."

Verification, as on Day 13:

ssh -o BatchMode=yes azureuser@20.119.41.127 'whoami; hostname'

azureuser
datacenter-vm

BatchMode=yes makes a password prompt impossible, so that login could only have come from the key.

What I learned

One line is not one unit of work, and both halves of today punished that assumption in different ways.

The ssh command looked like one instruction to one machine and was two instructions to two machines, because && is handled by the shell in front of ssh, not by ssh. Nothing errored loudly. Half the work simply happened on the wrong host, where the tool it needed does not exist.

az vm create looked like one instruction and was a deployment of six resources, so failure is not a single event. Five resources existed, one sat in Failed state, and the command meant to clean up started an operation it did not wait for, with its error output discarded.

The static IP is the counterweight. By creating the address as its own resource and referring to it by name, the one thing that had to survive four rebuilds did, while everything around it was destroyed and recreated repeatedly.

So here is the Day 71 question. In your scripts, which single line is actually doing its work on more than one machine, or on more than one resource, than you would say if asked?

Day 71 down. Twenty-nine to go.

Day 70: A Read-Only User and a Template

Nnamdi Felix Ibe — Sat, 24 Oct 2026 08:00:00 +0000

Day 70 of DevOps, Day 20 of Azure. Both tasks today were about what a system does to the things you did not mention. In Jenkins, a permission you do not tick is denied, which is the easy half. In ARM, a property you leave out of a template is reset to its default on the next deployment, which is not what the word incremental led me to expect.

One Jenkins task, one Azure task. Create a user and give them read-only access, then edit and deploy an ARM template. The tasks come from the KodeKloud Engineer platform.

Task 1: A read-only Jenkins user

The task: create kirsty with full name Kirsty and password BruCStnMT5, then grant read-only access using a project-based matrix authorisation strategy.

Step 1: Create the user

Manage Jenkins, Users, Create User, with the three values the task gave. That part is a form.

Step 2: Install the strategy, because it is not in Jenkins

This is the step the task's wording assumes you will work out. Matrix Authorisation Strategy is a plugin, not core, described on its own page as offering "matrix-based security authorisation strategies (global and per-project)". On a Jenkins where nobody has installed it, the Security page has no matrix option at all, so the authorisation half of the task cannot start until it is installed.

Jenkins names the two strategies the plugin adds as "Matrix-based security" and "Project-based Matrix Authorisation Strategy", listing them alongside the core choices "Anyone can do anything", "Legacy mode" and "Logged in users can do anything". The plugin's own page calls the same things global and project-based matrix authorisation. Two vocabularies, one feature.

The difference is whether per-item permissions exist at all. Matrix-based security is a single global grid. Project-based adds a second grid on every job and agent, which is what the requirement means by project-based.

Step 3: Grant Overall/Read globally

Manage Jenkins, Security, switch the strategy, add kirsty, tick Overall/Read.

This one goes first for a documented reason, with a caveat about how hard it is to state. The plugin says, "A user not granted read access to Jenkins in general will not be able to use most of the other permissions." Jenkins' permissions reference is more careful: "While Overall/Read is a prerequisite for more extensive access to Jenkins, some features are available without it."

So "nothing works without Overall/Read" is too strong, and "most of it does not" is right.

Step 4: Grant Read on the job

Open the job, Configure, enable project-based security, add kirsty, tick Read.

Item-level grants add to the global ones, and that is a default rather than a property of the strategy. The plugin: "Permissions explicitly granted on individual items or agents will only add to permissions defined globally." Two other inheritance modes exist, "Inherit global configuration only" and "Do not inherit permissions", and under the last of those nothing is additive.

Step 5: Log in as the user, because the grid tells you nothing

A matrix of ticks looks correct whether or not it is. The only check worth doing is logging out and back in as kirsty.

The dashboard comes up with no Manage Jenkins entry. The job opens and offers nothing beyond reading it. That is the requirement expressed as behaviour rather than as configuration, and it is the difference between having set the permissions and having set the right ones.

What the docs recommend

Know the way out before you go in. Jenkins documents the lockout case plainly: administrators "may accidentally set up a security realm or authorisation strategy" such that they can no longer administer Jenkins. The recovery is to stop Jenkins, edit config.xml in the Jenkins home directory, "Look for the elements <securityRealm> and <authorizationStrategy> and remove them", set <useSecurity> to false, and restart.

With a warning worth reading twice: "Jenkins will be in an entirely unsecured mode after it starts, allowing anyone full access." So the recovery is a window during which the controller is open to everyone, which is an argument for having a second administrator account rather than a recovery procedure.

Put the matrix in a file. This is Day 69's Configuration as Code point arriving with a sharper reason. A security grid built by clicking has no diff and no history, so the state you locked yourself out of is also the state nobody can describe. In a configuration file, it is reviewable, and for a Configuration as Code installation, the recovery is editing the authorizationStrategy directive rather than deleting elements from config.xml.

Task 2: Edit and deploy an ARM template

The task: change a template's VNet name, address space and tags, then deploy it with the CLI. The first infrastructure-as-code task on the Azure track, after CloudFormation on Days 47 and 48.

Step 1: Edit by path, not by pattern

cp vnet-deployment-template.json vnet-deployment-template.json.bak
jq '.resources[0].name = "arm-vnet-devops"
  | .resources[0].tags.displayName = "arm-vnet-devops"
  | .resources[0].tags.Environment = "KKE-devops"
  | .resources[0].properties.addressSpace.addressPrefixes = ["192.168.0.0/16"]' \
  vnet-deployment-template.json.bak > vnet-deployment-template.json

sed would have worked on this file and breaks the moment whitespace, key ordering or a repeated string differs. "virtualNetwork1" appears twice here, as the name and as the displayName, and both needed the same new value, so a blind substitution would have been right by luck rather than by intent.

Two details in that command matter. jq file > file empties the file, because the shell truncates a redirect target before the command runs: "If the file does not exist it is created; if it does exist it is truncated to zero size." Reading from the backup gives the read and the write separate paths. And assigning tags.Environment creates the key while leaving displayName alone, whereas assigning .tags = {...} would have replaced both.

Step 2: Validate, then preview

az deployment group validate -g $RG --template-file vnet-deployment-template.json

{ "Error": null, "Prov": "Succeeded" }

This is more than a syntax check. The command takes a --validation-level, and "With a validation level of 'provider', ARM will perform full validation and check that you have sufficient permission to deploy all resources in the template". Only the template level skips "preflight and permissions checks".

Which is a real difference from the AWS equivalent. On Day 47, aws cloudformation validate-template returned clean, and the stack then rolled back on an IAM permission. The limitation is documented, though not where you would look first: the API reference does not mention it, and the User Guide says, "The validate-template command is designed to check only the syntax of your template. It does not ensure that the property values that you have specified for a resource are valid for that resource."

There is a better pre-flight check still:

az deployment group what-if -g $RG --template-file vnet-deployment-template.json

"The what-if operation doesn't make any changes to existing resources. Instead, it predicts the changes if the specified template is deployed." So validate answers whether this will run, and what-if answers what it will do.

Step 3: Deploy

az deployment group create -g $RG -n arm-vnet-deploy --template-file vnet-deployment-template.json

"Mode": "Incremental", "provisioningState": "Succeeded"

The word incremental does not mean what I assumed

I had been carrying a comfortable version of this: Incremental mode leaves unrelated resources alone, and how much of an existing resource's state survives a redeploy varies by type.

Microsoft aims the warning directly at that assumption. "A common misunderstanding is to think properties that aren't specified in the template are left unchanged." What happens instead: "When redeploying an existing resource in incremental mode, all properties are reapplied. The properties aren't incrementally added", and "Properties that aren't included in the template are reset to the default values."

So Incremental is incremental about resources, not properties. A resource the template never mentions is left alone. A resource the template does mention is rewritten to the template's definition, and everything the template omits goes back to its default.

The type-specific caveat is real, and it is a short list of exceptions rather than a general hedge: "Two common examples are subnets on virtual networks and site configuration values for web apps." For subnets, "As long as the subnets are defined on the virtual network, you can redeploy the virtual network and not lose the subnets."

Which turns a vague rule of thumb into a concrete one. Add a tag to this VNet later with az tag update, as on Day 12, then redeploy this template unchanged, and the tag goes. Not maybe. Run what-if first, and it will tell you.

What the docs recommend

Keep Complete mode away from a shared resource group, and know it is on its way out. Complete mode is where "Resource Manager deletes resources that exist in the resource group but aren't specified in the template", so this single-VNet template would empty a populated group. Microsoft now steers away from it entirely: "Use deployment stacks to perform resource deletions when using ARM templates or Bicep files, as the complete mode will be gradually deprecated."

Do not expect a deployment to own anything. This is the core difference from CloudFormation and the reason Day 48's one-command teardown has no equivalent here:

	CloudFormation stack	ARM deployment
Deleting it	"CloudFormation deletes all resources in that stack"	"Deleting a template deployment does not affect the state of the resource group."
On a failed create	`OnFailure` defaults to `ROLLBACK`	No automatic undo
Teardown	`delete-stack`	Delete the resources, or the resource group

Deployment Stacks are the newer answer and do add ownership: a stack "enables you to manage a group of Azure resources as a single, cohesive unit", and a resource removed from the template "can be detached or deleted based on the specified actionOnUnmanage behaviour". The default is still cautious, since stacks "detach and don't delete unmanaged resources".

Parameterise it. The template's parameters and variables are both empty, with the name, address space and tags hardcoded, which is why this task was a text edit. It is also why the file is single-use. Three parameters and "name": "[parameters('vnetName')]" turn the editing exercise into:

az deployment group create -g $RG --template-file ... --parameters vnetName=arm-vnet-devops

Keep the two good habits the template already has. "location": "[resourceGroup().location]" makes region inheritance explicit rather than an unstated CLI default, which is why this VNet landed in eastus without anyone naming a region. And "apiVersion": "2023-11-01" pins the provider API, which the best-practice guidance wants hard-coded: "Set the apiVersion property to a hard-coded API version for the resource type", and specifically "Don't use a parameter for the API version."

Finally, watch the history limit, because it fails rather than grooms. A scope "can store up to 800 deployments in its history", grooming starts above 700 and trims to 600, and "If your scope is already at the 800 limit, your next deployment fails with an error." The grooming runs as the deploying user and needs delete permission, and a CanNotDelete lock stops it. So a locked resource group with a busy pipeline can hit 800 and start failing deployments for a reason that has nothing to do with the deployment. Note that what-if and validate runs count too.

What I learned

Jenkins is explicit about absence. If you don't grant a permission, it's denied, the grid shows every box you didn't tick, and the only surprise is that the strategy itself arrives as a plugin.

ARM is the opposite, and the mode's name is part of the problem. Incremental sounds like it adds to what is there. For a resource the template names, it replaces, and the properties you left out go back to defaults. My notes had a hedged version of this, and the hedge was doing the work of a warning.

So here is the Day 70 question. In your infrastructure code, what is currently true about a resource that the code itself does not say?

Day 70 down. Thirty to go.

Day 69: Two Plugins and One Setting

Nnamdi Felix Ibe — Fri, 23 Oct 2026 17:00:00 +0000

Day 69 of DevOps, Day 19 of Azure. Both tasks today were about reaching for the narrow instrument when a broader one was sitting right next to it. Two named Jenkins plugins rather than whatever else looked useful, and one container closed rather than the whole storage account.

The broader tool would have passed both tasks. That is what makes it worth writing down.

One Jenkins task, one Azure task. Install the Git and GitLab plugins, then convert a public blob container to private without touching its neighbour. The tasks come from the KodeKloud Engineer platform.

Task 1: Install two plugins

The task: log in as admin, install the Git and GitLab plugins, restart Jenkins.

There are no commands in this one. Manage Jenkins, then Plugins, then the Available tab, search for each, tick, install, restart. Which makes the interesting part everything the four clicks did on my behalf.

What ticking two boxes actually downloaded

The progress list that follows is much longer than two entries, and the docs explain why: "Plugins can be automatically downloaded, with their dependencies, from the Update Centre."

Git pulls in the Git client plugin, SCM API and several others. GitLab depends on Git, so the order I picked them in did not matter. It would matter if a dependency were unavailable, because the Update Centre resolves against what it is offering now rather than against what is already installed.

That is the first reason to install two named plugins rather than a handful that look handy. Every plugin is code running with the controller's privileges, and the dependency tree means you install more of it than you chose.

The restart checkbox, and a sourcing problem

The screen offers "Restart Jenkins when installation is complete and no jobs are running", and I ticked it.

Going to cite that, I could not. The string is on the screen and is not in Jenkins' documentation anywhere. The labels the docs do use are "Install without restart" and "Download now and install after restart". So the checkbox is real, and quoting it as documentation would be putting words in Jenkins' mouth.

A restart is not always required either. The docs: "Most plugins can be installed and used immediately by checking the box adjacent to the plugin." Where it is genuinely required is the manual route, dropping a .hpi into JENKINS_HOME/plugins, after which "The controller must be restarted before the plugin is loaded and made available in the Jenkins environment."

The useful half of that checkbox is the second clause. A restart during a build loses the build. On a controller with anything scheduled, install without restart and take the restart in a quiet window.

What the docs recommend

Make the plugin set reproducible. Clicking works once; it does not survive a rebuild. Jenkins documents a CLI route:

java -jar jenkins-cli.jar -s http://localhost:8080/ install-plugin SOURCE ... [-deploy] [-name VAL] [-restart]

described as "Installs a plugin either from a file, an URL, or from the update centre", with -restart to "Restart Jenkins upon successful installation".

There is a second tool worth knowing about with a caveat attached. jenkins-plugin-cli, the Plugin Installation Manager Tool, is what most images use:

RUN jenkins-plugin-cli --plugins "docker-workflow json-path-api"

That line is from Jenkins' Docker installation page, and that page is the only place in Jenkins' documentation it appears. The plugin management chapter does not describe the tool at all. Its own repository says it downloads "plugins and their dependencies into a folder" and uses "update center data to get the latest information about a plugin's dependencies".

For the whole configuration rather than just plugins, Configuration as Code is the declarative answer: "an opinionated way to configure Jenkins based on human-readable declarative configuration files", which its page frames as "just translating into code a configuration process one is used to executing in the web UI". Worth noting it says declarative configuration files rather than YAML, so YAML is accurate in substance and is not a quotation.

One honest gap. I wanted to cite a Jenkins instruction to keep plugins updated for security, and the plugin management chapter has none. What exists is the advisory machinery: "Security advisories are the primary way to publicly inform Jenkins users", and "Jenkins administrators are informed about published security issues directly in Jenkins". The advice to act on those is mine, not theirs.

Task 2: Close one container

The task: convert devops-container-111929086 to private, and leave devops-priv-111929086 alone.

Step 1: Establish the negative first

curl -s -o /dev/null -w "%{http_code}\n" \
  "https://$SA.blob.core.windows.net/$PUB?restype=container&comp=list"
curl -s -o /dev/null -w "%{http_code}\n" \
  "https://$SA.blob.core.windows.net/$PRIV?restype=container&comp=list"

PUB before:  200
PRIV before: 404

This is the step that makes everything after it mean something. A 404 at the end could be a wrong container name, a typo in the URL or an account that does not exist. A 404 that was a 200 five minutes ago, from the identical command, is the change itself.

The second line earns its place too. It confirms the already-private container behaves as expected, and gives a known-good reference for what private looks like over HTTP from this client.

Step 2: Pick the narrow control

az storage container set-permission --name "$PUB" --account-name "$SA" \
  --account-key "$KEY" --public-access off

The other option was one command away:

az storage account update -g "$RG" -n "$SA" --allow-blob-public-access false

That one closes every container in the account, now and in future, and Microsoft is explicit about the relationship: "Disallowing anonymous access for a storage account overrides the access settings for all containers in that storage account", and "If you disallow anonymous access for the account, you don't also need to disable anonymous access for individual containers."

It would also have passed the grader. The task said to leave the other container unchanged; it is already private, so nothing visible would change about it. The account flag still alters the conditions it operates under, and the requirement named one container.

The principle is the one from the Jenkins half: match the scope of the change to the scope of the requirement. A control that also fixes things you were not asked about is not a better fix.

A claim I had to drop

I had a second argument for the narrow control, and it does not hold up. I had written that closing at the account level leaves publicAccess: container on the container itself, so the container's own property would still read public and the requirement would not be met.

Microsoft describes the override entirely in terms of effective access, and never says whether the stored container property is retained or cleared. The nearest indirect evidence is that the same article still offers a per-container audit command, which only makes sense if the property persists independently of the account flag. That is an inference. It is not a documented behaviour and I should not have written it as one.

The scope argument stands on its own and did not need the second one.

Step 3: Prove it, and watch a column vanish

PUB after:   404

Then the table:

Name                        PublicAccess
--------------------------  --------------
devops-container-111929086  container
devops-priv-111929086

became:

Name
--------------------------
devops-container-111929086
devops-priv-111929086

The PublicAccess column did not go blank. It disappeared. The CLI's table renderer only adds a key to a row when its value is not null, so once both containers were private, every value in that column was null and the column stopped existing.

Which is uncomfortable, because the disappearance is the result. Both containers being private is exactly what was wanted, and the output announcing it looks like a broken query.

az storage container list --account-name $SA --account-key $KEY \
  --query "[].{Name:name,PublicAccess:properties.publicAccess}" -o json

That returns an explicit "PublicAccess": null for each. Use -o json when the expected answer is an absence, and -o table only for scanning values you expect to be present.

What the docs recommend

Use the conditional headers this operation actually supports. I had recommended --if-match to avoid racing another process adjusting permissions, and that header is not available here. The conditional headers table lists Set Container ACL with If-Modified-Since and If-Unmodified-Since only, and the CLI matches it:

az storage container set-permission --name "$PUB" --account-name "$SA" --account-key "$KEY" \
  --public-access off --if-unmodified-since 2026-09-28T00:00:00Z

Do not read a null in the CLI output as an absence in the API. My response showed "LastModified": null, and the REST reference lists both ETag and Last-Modified among Set Container ACL's response headers, with a sample response carrying both. The last-modified time genuinely advances here: "Any operation that modifies the container or its properties or metadata updates the last-modified time, including setting the container's permissions."

Do not quote a propagation time for this. I had written that the change is immediate. Microsoft documents no propagation figure for a container-level public access change. There are two nearby 30-second figures, one for the account-level anonymous access setting and one for establishing a stored access policy, and neither is this. Borrowing one would be inventing a number.

Plan for what breaks, because nothing warns you. Every blob URL that worked a moment ago now fails for anonymous callers: pages embedding images, documentation links, a partner integration, a CDN with a stale origin. Azure has no record of who was using those URLs, so there is no list to notify.

The scoped replacement is a shared access signature:

az storage blob generate-sas --account-name "$SA" --container-name "$PUB" --name <blob> \
  --permissions r --expiry 2026-09-29T00:00:00Z --account-key "$KEY"

--expiry "Specifies the UTC datetime (Y-m-d'T'H:M'Z') at which the SAS becomes invalid", which is the property a public container can never have.

What I learned

Both halves had a bigger lever within reach, and in both cases pulling it would have passed. Installing a comfortable handful of plugins works. Disabling anonymous access account-wide works. Neither is what was asked for, and the difference only shows up later, when somebody wonders why a container they never touched behaves differently or why a controller is running code nobody chose.

The part I did not expect was having to give up an argument I liked. My second reason for preferring the narrow Azure control turned out to be undocumented, which left the first reason doing all the work. It was enough on its own, which is the useful thing to find out before publishing rather than after.

So here is the Day 69 question. The last time you fixed something, did you fix the thing you were asked about, or everything adjacent to it that looked wrong on the way past?

Day 69 down. Thirty-one to go.

Day 68: Installing Jenkins and Copying a File

Nnamdi Felix Ibe — Wed, 21 Oct 2026 17:00:00 +0000

Day 68 of DevOps, Day 18 of Azure. Kubernetes closed yesterday at Day 67, and Jenkins opens here and runs to Day 81. Both of today's tasks ended the same way: the tool said it had succeeded, and the check that settled it was something else entirely.

One Jenkins task, one Azure task. Install Jenkins and get through the setup wizard, then copy a single file into a blob container. The tasks come from the KodeKloud Engineer platform.

Task 1: Install Jenkins

The task: install Jenkins on the jenkins server, start it with the service command, and create a named first administrator.

Step 1: Java, because nothing else works without it

ssh root@jenkins
java -version

-bash: java: command not found

sudo apt update
sudo apt install fontconfig openjdk-21-jre
java -version

openjdk version "21.0.10" 2026-01-20
OpenJDK Runtime Environment (build 21.0.10+7-Ubuntu-124.04)

The installation page lists "Java 21 or later" and gives that exact install line, fontconfig openjdk-21-jre.

The version is worth a moment rather than a copy and paste. Jenkins' Java support policy lists Java 17 under unsupported versions, and names its final LTS release as 2.541.3. That is the version this lab installed. So this server is sitting on the last LTS that would still have accepted Java 17, and from the next one Java 21 is not a recommendation, it is the floor.

Step 2: The repository and the package

sudo wget -O /etc/apt/keyrings/jenkins-keyring.asc \
  https://pkg.jenkins.io/debian-stable/jenkins.io-2026.key

echo "deb [signed-by=/etc/apt/keyrings/jenkins-keyring.asc]" \
  https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null

sudo apt update
sudo apt install jenkins

Get:1 https://pkg.jenkins.io/debian-stable binary/ jenkins 2.541.3 [95.8 MB]

Two things in those four commands are easy to get wrong from an old tutorial. debian-stable is the Jenkins LTS apt repository, not anything belonging to Debian or Ubuntu, so none of this comes from the distribution archives. And the key filename carries a year, jenkins.io-2026.key, which is exactly the sort of value a copied script keeps long after it has moved on.

Step 3: Read the install output before scrolling past it

invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Created symlink /etc/systemd/system/multi-user.target.wants/jenkins.service -> /usr/lib/systemd/system/jenkins.service

Those first two lines say the package tried to start the service and the environment refused. That is a container convention rather than a failure, and it is why the next command reported Jenkins not running immediately after a clean install.

service jenkins status

 * jenkins is not running

service jenkins start
service jenkins status

 * Starting Jenkins Automation Server jenkins    Setting up max open files limit to 8192  [ OK ]
 * jenkins is running

The task asked for service. The documentation does not use it anywhere, and says why: "Beginning with Jenkins 2.335 and Jenkins 2.332.1, the package is configured with systemd rather than the older System V init." service jenkins status works through systemd's compatibility shim. systemctl status jenkins is what the docs would have me type.

Step 4: Unlock, and notice where the password lives

cat /var/lib/jenkins/secrets/initialAdminPassword

Jenkins listens on 8080 and refuses to do anything until that value is pasted into the browser. The design is deliberate: the only person who can finish the setup is someone who can already read the server's filesystem. Which is also the reason not to leave a half-configured Jenkins reachable while you go and find the file.

Then "Install suggested plugins", described in the docs as "the recommended set of plugins, which are based on most common use cases", and the first administrator user. Day 69 is about what happens when the suggested set does not contain what you need.

What the docs recommend

Use systemctl. The package has been a systemd unit since 2.332.1, and systemctl cat jenkins shows you the unit the package installed, which is where the port and the Java options actually live.

Know where JENKINS_HOME is before you need it. "On Ubuntu by default, this is set to /var/lib/jenkins", and it is "the root of the disk directory structure that Jenkins uses to perform builds and store archives". Everything that matters is in there: jobs, credentials, plugins, the lot. It is the backup target, and on Day 71 it is also where an SSH key generated by the jenkins user ends up.

Treat the suggested plugin set as a starting point rather than a decision. It is built for common cases, and the next two days are both about reaching past it.

Task 2: Copy a file into a blob container

The task: copy /tmp/xfusion.txt into the existing container xfusion-blob-398826849.

Step 1: Upload

az storage blob upload --account-name $SA --account-key $KEY \
  --container-name $CONTAINER --name xfusion.txt --file /tmp/xfusion.txt

Step 2: Do not stop at the listing

The listing showed the right name and 33 bytes, matching the local file. That is good evidence and it is not proof. A file can be the right length with different bytes in it, and text tools hide trailing whitespace, line-ending changes and encoding shifts.

az storage blob download --account-name $SA --account-key $KEY \
  --container-name $CONTAINER --name xfusion.txt --file /tmp/roundtrip.txt

cmp -s /tmp/xfusion.txt /tmp/roundtrip.txt && echo "MATCH: byte-identical"

MATCH: byte-identical

cmp compares bytes, which is the only check that means anything when the claim is that nothing changed. Cheap here at 33 bytes. For a large upload the equivalent is comparing checksums rather than re-downloading.

The checksum that was not where I looked

--query "{ETag:etag,Md5:contentMd5}"

{ "ETag": "\"0x8DF1C7E15C4FC83\"", "Md5": null }

So I read it off the blob instead, which worked:

az storage blob show ... --query "properties.contentSettings.contentMd5" -o tsv

My conclusion at the time was that contentMd5 is not in the upload response. Checking it, that is wrong. The Put Blob REST reference lists Content-MD5 among its response headers, "Returned for a block blob so that the client can check the integrity of the message content", and adds that "In version 2012-02-12 and later, this header is returned even when the request doesn't include Content-MD5 or x-ms-blob-content-md5 headers".

So the API did return it, and the CLI did not surface it under the name I queried. That makes the finding sharper rather than weaker: what a command prints is the command's own shape, and it is not a reliable guide to what the service sent back. The habit that follows is unchanged, and now rests on better ground:

az <command> -o json | grep -i <fragment>

Hex and base64 are the same digest

md5sum /tmp/xfusion.txt

2eeef38a56ad6c682ecd2cf611c9f921

md5sum prints hex. Azure wants Base64, and the requirement is explicit in the error table: InvalidMd5 means "The MD5 value specified in the request is invalid. The MD5 value must be 128 bits and Base64-encoded." The neighbouring error, Md5Mismatch, is what a well-formed but wrong digest gets.

openssl md5 -binary /tmp/xfusion.txt | base64

Worth knowing before deciding that a checksum mismatch means corruption.

What the docs recommend

Use conditional headers rather than an overwrite flag. The ETag is the concurrency token: "Each version of the blob has a unique tag, called an ETag, that you can use with access conditions to assure you only change a specific instance of the blob."

--if-match '"0x8DF1C7E15C4FC83"'   # only write if unchanged since I read it
--if-none-match '*'                # only write if the blob does not exist

The wildcard is documented directly: "Specify the wildcard character (*) to perform the operation only if the resource doesn't exist, and fail the operation if it does exist." An unmet condition returns 412.

That beats relying on --overwrite, whose default is not uniform. On az storage blob upload it defaults to false and "the operation will fail with ResourceExistsError"; on az storage blob download the docs give the default as true. Per command, again.

Pick the blob type deliberately, because you only get one chance. "You specify the blob type when you create the blob. Once the blob has been created, its type cannot be changed." Block blobs for files, append blobs for logs, page blobs for virtual machine disks. A log sink created as a block blob cannot later be appended to efficiently.

Remember the namespace is flat. "By default, the Blob service is based on a flat storage scheme, not a hierarchical scheme. However, you may specify a character or string delimiter within a blob name to create a virtual hierarchy." A slash in a blob name is a character. There is no empty folder, deleting the last blob under a prefix makes the folder disappear, and renaming a folder means copying everything under it.

If you need real directories, the upgrade path now exists, which corrects something I had written down as fixed. Hierarchical namespace used to be a creation-time choice only. Microsoft now documents az storage account hns-migration start --type upgrade, with the warning that "An upgrade is one-way. There's no way to revert your account once you've performed the upgrade", and preconditions: no page blobs in the account, and snapshots, encryption scopes, immutable storage and soft delete disabled first.

And know where this command stops. For a directory there is az storage blob upload-batch. For anything large, there is AzCopy, where resume is documented: azcopy jobs resume "Resumes the existing job with the given job ID", and on resume it "attempts to transfer all of the files that are listed in the plan file and weren't already transferred".

What I learned

Both halves handed me a success message I could not use. service jenkins status said it was running, which is true and says nothing about whether the setup wizard will load. The blob listing said 33 bytes, which is true and says nothing about the bytes.

The second one also cost me a conclusion. I had written down that the upload response does not carry the checksum, when what actually happened is that the response carried it and the CLI printed something else. Those look identical in the terminal, and they are not the same fact.

So here is the Day 68 question. When a command prints a field as null, do you check whether the service sent it, or do you just look somewhere else and move on?

Day 68 down. Thirty-two to go.

Part 2: From Docker Compose up to Terraform on AWS, and the CI That Checks It

Nnamdi Felix Ibe — Wed, 21 Oct 2026 12:00:00 +0000

Part 1 ended with a pipeline that turns HL7 v2 into FHIR R4 and runs on a laptop with make demo. This part is about the distance between that and something a trust could run, and about the checks that tell you whether you have crossed it.

The short version: the distance is larger than the Terraform makes it look, and the first thing I did this week was build the pipeline that proves it.

The problem

docker compose up gives you a working system and almost no evidence. It tells you the code runs. It tells you nothing about whether the database is reachable from the internet, whether a credential is sitting in the history, or whether the thing you deploy is the thing you tested.

In most settings that gap is a nuisance. In an NHS setting, it is the whole job. The data is patient-identifiable, auditors ask questions about provenance, and "it worked on my laptop" doesn't answer any of them.

So the move from compose to AWS is not really a move from one runtime to another. It is a move from a system you have seen working to a system you can show is working. That needs two things: infrastructure described as code, and a pipeline that checks it on every change.

I had the first. I did not have the second, and this week showed me what that was costing.

What I built

The repo describes an AWS deployment in Terraform: a VPC with public, private and database subnets, an EKS cluster with a managed node group, an RDS Postgres instance for the HAPI FHIR server, an S3 bucket for the dead-letter path, and KMS keys. terraform plan reports 46 resources.

Two honest notes before anything else.

The provider block in infra/terraform/versions.tf is this:

provider "aws" {
  region = var.aws_region

  access_key = "mock"
  secret_key = "mock"

  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
}

That is how the plan runs in CI without an AWS account. It hasn't always been like that. In April I ran apply against my own AWS account to see the stack come up, and that is where the EKS add-on version conflicts in docs/lessons-learnt.md came from. Nothing from that run is still up, so the code you can clone today has been planned, not deployed. The mock credentials keep it free for anyone else to run.

The second note is worse, and the pipeline found it.

The pipeline, and what it found

Until this week the repo had a .gitlab-ci.yml describing five stages, and nothing had ever run it. Porting it to GitHub Actions took an afternoon. Getting it green took longer, because it kept being right. Some of what it found turned up while I was porting it, and the rest arrived on its first real run.

The secret scan had never scanned anything

The pipeline runs Gitleaks. The repo has a .gitleaks.toml. The file held exactly this:

# Gitleaks config - ignore false positives in vendored terraform modules

One comment. No rules, and no [extend] block. Gitleaks' documentation describes two options for a custom config: "define your own configuration, default rules do not apply", or extend one, where "useDefault will extend the default gitleaks config built into the binary".

Mine was the first kind. A configuration with no rules, replacing roughly a hundred built-in ones.

I checked it rather than assuming. A directory with a planted Slack bot token, scanned three ways with gitleaks 8.28.0:

no config file                  leaks found: 1    exit 1
the repo's .gitleaks.toml       no leaks found    exit 0
[extend] useDefault = true      leaks found: 1    exit 1

The middle line is the one that matters. For months, a secret scan ran, found nothing, and the nothing meant nothing. The repo also carries docs/verification/gitleaks.json, a stored run showing zero findings, which I had been treating as evidence that the repo was clean. It is evidence that the scanner was switched off.

When I first wrote this section, the next line said I had no reason to think there was a secret in there. Then the fixed scanner ran over the git history and found one.

On 26 April I committed a PDS API key to .env.example while getting a demo working. The next commit blanked the value. The file looked clean from then on, and the key sat in the public history for six months. A scanner pointed at the current files would never have seen it. gitleaks git reads every commit, and it did.

The key is revoked now. It stays in the history because rewriting a public history breaks every clone and tag. Revoking it is what makes it harmless. The repo's .gitleaksignore now holds that one finding's fingerprint, with a comment saying when the key was committed and when it was revoked. Any other secret still fails the build.

So the stored report with zero findings wasn't just meaningless. It was wrong, and I had been treating it as evidence.

The test job could not have passed

The GitLab test job runs:

pytest -q --junitxml=junit.xml --cov=. --cov-report=xml

pytest-cov appears in none of the three requirements-dev.txt files. I ran that command against each service in a clean virtualenv built from its pinned requirements, and got the same thing three times:

error: unrecognised arguments: --cov=. --cov-report=xml

The tests themselves passed. Sixteen of them pass as soon as you drop the flags. It was the job around them that had never worked, and because nothing ran it, nothing said so.

With pytest-cov added, the numbers are ingest 98 per cent, transform 83 per cent, pds-client 93 per cent.

The database could call the internet

The new pipeline runs Trivy against the Terraform. Its first finding was critical, in the security group in front of the FHIR database:

resource "aws_security_group" "rds" {
  description = "HAPI FHIR Postgres, ingress from EKS nodes only"

  ingress {
    description     = "Postgres from EKS"
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [module.eks.node_security_group_id]
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Look at the care in the ingress rule and the shrug in the egress one. Inbound is locked to the node group's security group, port 5432, TCP only. Outbound is every port, every protocol, every address on the internet.

The description even says "ingress from EKS nodes only", which is true and beside the point. I wrote a careful inbound rule, felt careful, and pasted the default outbound rule underneath it.

The fix restricts egress to the VPC.

The cluster API was open to the internet

My first count of Trivy findings was wrong. I ran it on a copy of the repo where it never downloaded the community EKS module, so it only scanned my own files. A clean checkout in CI downloads the module and scans what my inputs do to it. That found three more critical findings.

Two were the same problem. The EKS API endpoint was public and open to 0.0.0.0/0:

  # API endpoint is private by default; grant access to the CI runner via
  # the public endpoint + CIDR allow-list when needed.
  cluster_endpoint_public_access       = true
  cluster_endpoint_public_access_cidrs = ["0.0.0.0/0"] # tighten in prod

The comment says private. The code says public, to everyone. Nothing outside the VPC needs that endpoint, because Argo CD pulls from inside the cluster, so it is now private.

The third was node egress to any address, the same rule as the database. This one I kept. The nodes need outbound internet to pull images and to call the PDS API. It sits in a .trivyignore.yaml, scoped to the one module file and with the reason written next to it. I checked the scope by putting the database rule back the way it was: the build failed.

A dependency I never chose

The image scan failed on all three services for the same reason. fastapi==0.115.0 pulls in starlette 0.38.6, which has CVE-2024-47874, rated high. I had pinned FastAPI and never looked at what it brought with it.

Bumping FastAPI alone wasn't enough. The newer Starlette releases had high findings of their own, fixed only in 1.x, and prometheus-fastapi-instrumentator 7.0.0 capped Starlette below 1.0. Both had to move. All sixteen tests still pass, and coverage hasn't changed.

What is left on the Terraform is four findings, none above medium: IAM authentication and deletion protection on the database, a customer-managed key for Performance Insights, and access logging on the bucket. Those are real, and they are on the list for Part 5. Every run prints them, along with the accepted one, so they don't live only in my head.

What the pipeline actually checks

Seven jobs run on every pull request into master:

lint, ruff over each service.
test, pytest with coverage over each service.
secrets, Gitleaks over both the working tree and the git history.
sast, Semgrep with the p/ci, p/python and p/security-audit rulesets.
terraform, fmt -check, then init -backend=false and validate.
iac-scan, Trivy against the Terraform.
image, a Docker build per service, then a Trivy image scan and a Syft SBOM.

A few decisions inside that are worth more than the list.

Every version is pinned, down to the scanner binaries. The gitleaks problem was a security tool that silently stopped checking, and a floating latest tag is the same failure wearing a different hat. If a tool changes what it catches, I want that to arrive as a version bump I can read in a diff.

terraform validate runs with -backend=false, which initialises the providers without touching remote state, so the job needs no AWS credentials at all. Validation is a question about syntax and types, not about an account.

The IaC scan reports everything and fails only on high and critical. A gate you have not triaged is a gate someone turns off in a week. Report first, then promote findings to blocking as you fix them.

The image is built and scanned but not pushed. There is nowhere to push it to yet, and inventing a registry to make the diagram look complete would be the same instinct that produced the egress rule.

And one job did not come across at all. The GitLab pipeline ended with a deploy stage that bumped a Helm values file and let Argo CD sync it. Its rule was if: $CI_COMMIT_BRANCH == "main". This repo's branch is master. That job could never have run on any commit, which is a tidy summary of the whole week: the pipeline described a system nobody had checked.

Best practices worth stealing

Commit the Terraform lock file. HashiCorp's documentation says "You should include this file in your version control repository", because Terraform "remembers the decisions it made in a dependency lock file" so that "it can (by default) make the same decisions again in future". Mine was in .gitignore. CI is exactly the second machine that needs to make the same decision.

Make the local command and the CI command the same one. The repo's Makefile already had make tf-validate, running init -backend=false, validate and fmt -check. The CI job runs the same three. If it passes on your machine, it passes in CI, and you stop pushing commits to find out.

Scan the history, not just the tree. gitleaks dir checks what is there now. gitleaks git checks what was ever there. A secret removed in a later commit is still published. While you are at it: detect is deprecated, and the README says "v8.19.0 introduced a change that deprecated detect and protect". The modes are git, dir and stdin now.

Make the tool fail loudly or not at all. Semgrep prints its findings and exits 0 unless you pass --error. Trivy does the same unless you pass --exit-code 1, and it is in two of my jobs. So three of the seven would have been decorative by default: they would have printed the problem in a log nobody opens and gone green.

Gitleaks is the exception, and I had it wrong until I tested it. Its help text reads --exit-code int exit code when leaks have been encountered (default 1), which I first skimmed as requiring the flag. It is not: gitleaks exits 1 on a finding whether you ask or not. I had already written the opposite into a draft of this article. Checking the three tools took one command each.

Read why a gate went red. My first iac-scan job passed --no-progress to trivy config. That flag works on trivy image and doesn't exist on trivy config, so the job failed before it scanned a single file. Red, for a reason that had nothing to do with Terraform.

Fix the name, not the rule. Semgrep flagged requests.append({... "http://snomed.info/sct" ...}) as an unencrypted HTTP request. It was a Python list called requests, holding FHIR MedicationRequests, and the URL is a SNOMED CT system identifier that FHIR requires in that exact form. Renaming the list to medication_requests cleared it without switching anything off.

Where AI helps here, and where it doesn't

The port was a good use of AI. Translating a GitLab pipeline into Actions is mechanical, well-documented work, and the shape of the answer is known before you start.

What it did not do is notice. The gitleaks config, the key in the history, the missing pytest-cov, the egress rule and the open endpoint were all found by running things and reading the output, not by looking at the code and thinking about it. Every one of those files reads as correct. A config file with an allowlist comment looks like a config file. A requirements file without pytest-cov looks like a requirements file. An egress rule set to 0.0.0.0/0 is the most common four characters in any Terraform repo.

The same thing happened to me twice in two weeks. In Part 1 it was a retry wrapper raising its own exception type, so the dead-letter path never ran. Here it is a scanner with its rules switched off. Both passed review, mine included, because reading code tells you what it says and running it tells you what it does.

Running things has a catch of its own. The first Trivy count in this article came from an AI-assisted run on a copy of the repo that never downloaded the EKS module. It ran, it printed numbers, and the numbers were wrong. CI's clean checkout was the run that counted.

So the rule from Part 1 survives contact with Part 2: AI drafts, and the specification, the tests and the running system decide. I would add a line to it now. The output of a check is only evidence if you have seen the check fail, in the same place it will run for real.

What I'd do differently

I would have built the pipeline first, before the Terraform, before the Helm charts, probably before the third service. Not because CI is more important than the code, but because for eight weeks I had five stages of YAML that described a quality bar and enforced none of it. That is worse than having no pipeline, because it reads like one.

I would also have protected master before the first workflow landed. The pull request that added the pipeline merged with its own checks red, because nothing required them to pass. A required status check is one setting.

The other thing is smaller and more annoying. docs/lessons-learnt.md and docs/security.md both say every AWS resource carries a DataClass=patient-identifiable tag through Terraform's provider-level default_tags, and describe the audit query that tag makes possible. The default_tags block in versions.tf is commented out. The documentation is describing a feature of the system I meant to build. Fixing that properly belongs with Part 5, on security and compliance, and it is going to be the theme of that article.

Sources

Part 3, on 28 October, is about the mapping itself: HL7 v2 ADT messages to FHIR R4, and the places it goes quietly wrong, several of them found by readers of Part 1.

What does your pipeline check that you have never watched fail?

This is a personal project. It is not affiliated with or endorsed by NHS England or any NHS organisation. It uses the public PDS FHIR sandbox and, earlier in the project, the PDS integration test environment, with synthetic test data only. Views are my own.

DEV Community: Nnamdi Felix Ibe

Day 77: A Pipeline and a Private Subnet

Task 1: A first declarative pipeline

Step 1: The agent and a stale instruction

Step 2: The pipeline

What the docs recommend

Task 2: A VM reachable only from inside

Step 1: One flag decides the whole task

Step 2: Two layers, and only one of them is about inbound

Step 3: This is the VM that loses its way out

What the docs recommend

What I learned

Day 76: A Backup of Nothing and a Retiring Default

Task 1: A database backup on a ten-minute schedule

Step 1: Two hops, two keys

Step 2: The build step, and one thing it gets right

Step 3: The job works. The backup does not.

What the docs recommend

Task 2: A VM in a purpose-built network

Step 1: Name things rather than letting the VM name them

Step 2: What makes the network public, and why my answer has expired

What the docs recommend

What I learned

Day 75: A Java Baseline and a Moving Disk

Task 1: Three servers as build agents

Step 1: The plugin, and what it is called now

Step 2: Create the node, and find the fields are undocumented

Step 3: The agent refuses to start

What the docs recommend

Task 2: Expand one disk, add another

Step 1: The outage belongs to the OS disk, not the task

Step 2: The reboot did the other two layers, and I overstated how

Step 3: The letters moved while I was working

What the docs recommend

What I learned

Day 74: A Key Each Way and a Schedule

Task 1: A job on a six-minute schedule

Step 1: Check the logs are actually there

Step 2: Key the connection between the two servers

Step 3: Let the schedule fire

What the docs recommend

One link in the chain I cannot account for

Task 2: A VM that only accepts keys

Step 1: A non-empty .ssh directory is not a key pair

Step 2: The key size was a downgrade and I did not notice

Step 3: Name the key rather than generating one

Step 4: Password authentication came out disabled

What the docs recommend

What I learned

Part 3: Nine Ways an HL7 v2 to FHIR Mapping Goes Quietly Wrong

The problem

What I found

A positive culture result became zero

Every admission created a new patient, and the obvious fix is the wrong one

The dates it threw away

The names it mangled

The middle name, and the test that approved of losing it

The identifiers that were not identifiers

Nulls where FHIR forbids them

The answer I said did not exist

The map was not the territory

What readers found that I have not fixed yet

The one thing still open

Worth stealing

Where AI helps here, and where it doesn't

What I'd do differently

Sources

Next

Day 73: Six Permissions and a Repeat

Task 1: Two users, one job, different access

Step 1: Check, do not create

Step 2: Install the plugin, again

Step 3: The step the task does not mention

Step 4: The six permissions come from three places

Step 5: Log in as both of them

What the docs recommend

Task 2: Yesterday's VM, from the CLI

Step 1: Pass the region anyway

Step 2: The same commands took a different amount of time

What the docs recommend