Reducing GPU scheduling waits from days to minutes
A GKE capacity-engineering system that generated workload-specific GPU fallback policies across compatible hardware and provisioning modes.
A valid workload could still wait for days
My contribution: I designed the workload-specific fallback policy and built the generator that produced named GKE ComputeClasses from validated GPU and provisioning options.
Valid GPU workloads could still wait because requests were tied too closely to one unavailable hardware family or provisioning mode.
The platform needed to preserve genuine constraints without turning a preference into a scheduling dead end.
A short retryable job and a long non-preemptible workload should not follow the same fallback path merely because both need a GPU.
Scheduling policy starts with workload intent
Fallback order came from workload type and GPU needs, maximum runtime, and tolerance for interruption. These inputs determined compatible hardware and the order of Flex Start, Spot, and on-demand capacity. No global sequence fit every workload.
A retryable job could prioritize cheaper interruptible capacity; an expensive restart could move stable capacity earlier. Compatible GPU types could broaden before accepting a long wait.
Generating named ComputeClasses
A generator turned supported GPU profiles, sizes, compatibility groups, and provisioning chains into named GKE ComputeClasses, avoiding repetitive YAML and inconsistent fallback behavior.
A workload could select an exact profile, a broader family, or a cross-family class when requirements allowed. Each class carried ordered provisioning priorities.
Class names exposed an understood capacity strategy at the workload boundary instead of hiding ad hoc node-pool logic in manifests.
Broadening capacity without weakening requirements
Fallback did not send any job to any GPU. The generator combined only supported profiles and modes; workloads selected a class broad enough for their tolerance.
This prevented narrow requests from ignoring usable capacity and broad requests from landing on unsuitable hardware or interruption modes. Platform engineers maintained validated policies; workloads selected the matching intent.
Observed impact
In observed production cases, workloads that had waited for days moved toward scheduling in minutes once compatible fallback capacity became available. This was an observation, not a formal benchmark or fleet-wide service-level claim.
The scheduler traversed deliberate hardware and provisioning alternatives instead of waiting indefinitely on one source. Cost choices also became explicit per workload tolerance.
Trade-offs
More fallbacks increase policy surface. Generated configuration still needs review, validation, and clear source-profile ownership. Broader compatibility can hide application assumptions, so requirements and interruption tolerance remain hard inputs.
What lasted
The durable capability was policy generation, not one fixed set of GPU pools. Hardware profiles and fallback chains could evolve while workloads kept selecting stable, named intents.
Related
Part of: Building Azure Landing Zones, Then Expanding Across Clouds
Related work: