The failure pattern
The self-hosted Azure DevOps runner fleet depended on an image-generation pipeline derived from Microsoft’s Azure Pipelines image-generation project. Over time, the local version had fallen behind upstream while manual patches accumulated around it.
Toolchain dependencies aged at different rates, GPU images added CUDA and driver constraints, and a successful build did not make an image safe for every active runner pool.
Recurring instability was the symptom. The deeper problem was an incomplete path from source change to validated fleet rollout.
Separating slow and fast change
Rebuilding everything daily was too expensive; updating only after failures kept runners stale. I split the image by change rate.
A weekly Packer build refreshed the underlying runner foundation. It absorbed operating-system updates and heavier dependencies, including the layers needed by GPU workers. A daily organization-specific refresh applied the faster-moving tools and configuration used by engineering workloads.
This kept dependencies current without repeating the expensive base build every day. It also separated base-image failures from organization-layer failures.
Treating images as promoted artifacts
Pulumi managed gallery definitions, VM scale sets, identities, and Azure DevOps pool infrastructure. Only a validated image advanced to an active scale-set reference.
CPU and GPU runners remained separate because their images, costs, and workload expectations differed. They shared the lifecycle, not one image or scaling policy.
The build encoded real fleet constraints: Azure-compatible NVIDIA drivers, authentication tooling used by platform jobs, and only the processor architecture the fleet used.
If validation failed, scale sets stayed on their known image while the candidate was corrected.
Capacity for real working patterns
Image reliability solved only half the platform problem. A perfectly built runner that starts too late still leaves engineers waiting.
An Azure Pipeline adjusted VM-scale-set capacity on a schedule, complementing normal scaling with warm runners during active hours. CPU and GPU pools used different capacity profiles.
Runner rebuilds became predictable, dependencies stayed fresher, emergency patching became less common, and useful capacity was available as engineering activity increased.
Trade-offs
Weekly and daily responsibilities had to stay distinct, or fast-changing tooling would leak into the expensive base. Pre-warming also accepted some idle cost for reliable working-hour capacity, especially for GPU pools.
What lasted
The durable improvement was not one golden image. It was a controlled path from upstream source to active CPU and GPU fleets, able to absorb changing software and workloads.
Related
Part of: Building Azure Landing Zones, Then Expanding Across Clouds
Related work: