Building Azure Landing Zones, Then Expanding Across Clouds: Obeyda Djeffal TUI

Overview

This work began in Azure, before the platform spanned multiple clouds. Subscriptions and a few resources created by an inherited Pulumi prototype existed, but those resources needed rebuilding. The prototype appeared to have been generated from a generic infrastructure tutorial; its code was unreliable and did not fit our needs. I kept the intended resource list, rewrote the implementation, and built the Azure landing zones across development, staging, and production.

That Azure foundation became the base for a larger platform. Growing computational biology and machine learning workloads ran into Azure GPU quota and capacity bottlenecks, so we added Google Cloud Platform (GCP). Later, cloud credits made Amazon Web Services (AWS) useful for additional compute. We retained Azure as the network hub and kept workloads in each cloud independent at runtime.

Evolution & Design Decisions

1. Building the Azure Landing Zones

Starting point

The subscriptions existed, along with some resources created by the inherited Pulumi prototype. Those resources needed recreation; the prototype was not a usable foundation.
The generated code described intended infrastructure, but its structure and assumptions did not fit our requirements. I used its resource list as input and rewrote the implementation.

Work

Built a deployable Pulumi foundation across the existing subscriptions, starting with a development environment and then adding staging and production.
Created virtual networks, peering, VPN gateways, private connections, and DNS for shared services and workload connectivity.
Added identity and role assignments, Azure Policy governance, monitoring, and security controls.
Provisioned application infrastructure including AKS clusters, PostgreSQL databases, Redis caches, and storage accounts.
Organized foundation and workload resources into Pulumi stacks with explicit dependencies.

Result

The first development environment was usable in 10 days, giving developers a place to work and deploy their applications. I continued building staging and production and hardening the Azure foundation afterward. That substantial Azure phase created the base for delivery systems and, later, additional clouds. The Azure foundation story covers the build in more detail.

2. Expanding Beyond Azure

Context & Constraints

The Azure foundation grew to support core applications on AKS, primary databases, Entra ID (Azure AD) identity, and a centralized container registry (ACR). A Site-to-Site (S2S) link connected the on-premises laboratory for local compute and sensor data capture.
Computational biology and machine learning workloads required increasing GPU capacity.
In Azure, turnaround times on GPU quota requests slowed, and obtaining the required GPU allocations became an operational bottleneck.

Decisions

GCP for ML & GPU compute: We introduced GCP and GKE to run GPU workloads, utilizing Container-Optimized OS (COS) node pools and GCP Committed Use Discounts (CUDs) for sustained compute.
AWS for credit-backed compute: We added AWS as a secondary compute environment to utilize allocated cloud credits for batch and GPU pipelines.
Azure retained as primary hub: We kept Azure as the operational core for existing applications, primary databases, CI build infrastructure, and corporate identity.

3. Network Topology: Hub-and-Spoke with Dynamic Routing

Problem Interconnecting four network boundaries (Azure, GCP, AWS, on-prem) with direct point-to-point connections would require 6 bidirectional tunnel configurations ($4 \times 3 / 2 = 6$), increasing routing complexity and operational overhead.

**Network topology**Azure remains the routed hub for GCP, AWS, and on-premises spokes while each cloud keeps its runtime services local.

Decision

Maintained a hub-and-spoke topology centered on Azure.
Configured BGP dynamic routing across the Site-to-Site VPN connections between the Azure Virtual Network Gateway and the spoke gateways in GCP, AWS, and on-premises.
Implemented a modular CIDR plan with non-overlapping blocks across all four environments, allowing expansion without Network Address Translation (NAT).
Provided engineering access via an Azure Point-to-Site (P2S) VPN with split-tunneling and Entra ID authentication, enabling private access to all environments through the hub while public internet traffic remained local.

Trade-off Spoke-to-spoke network transit (e.g., GCP to AWS) incurs an extra network hop through Azure. This was an acceptable trade-off because workloads in GCP and AWS ran independent compute tasks and communicated back to central services rather than with each other.

Result Operating 3 site-to-site VPN connections into Azure kept network administration predictable, while BGP handled route propagation automatically when new cluster subnets were provisioned.

4. Container Registries: Centralized vs. Cloud-Local

Problem Initially, clusters in GCP and AWS pulled images from a centralized Azure Container Registry (ACR) over the inter-cloud VPN tunnels. This created bandwidth contention on the VPN links, incurred cross-cloud data transfer costs during routine image pulls, and made node autoscaling in remote clouds dependent on Azure connectivity.

Decision Provisioned dedicated container registries in each cloud provider (ACR in Azure, Artifact Registry in GCP, ECR in AWS) managed via Pulumi. CI pipelines published images directly to the registry of the target deployment environment.

Trade-off Managing three container registries required additional CI pipeline publishing logic and separate access credentials.

Result Workload image pulls remained local to each cloud provider, removing image-pull data transfer from the inter-cloud VPN links and isolating cluster autoscaling from inter-cloud connectivity issues.

5. Platform Management & Secrets

Problem Operating Kubernetes across three clouds required administrative tooling (Argo CD for GitOps and HashiCorp Vault for application secrets) without managing multi-cloud data replication.

Decision

Decoupled application secrets from cloud-provider secret stores and deployed a self-hosted HashiCorp Vault instance along with Argo CD on a dedicated GKE cluster (the Platform Management cluster).
Retained human user and corporate identity strictly in Azure (Entra ID).
Workloads in remote clusters consumed secrets through the External Secrets Operator (ESO) and Vault sidecars, materializing them as native Kubernetes secrets in each cluster.

Trade-off Centralizing GitOps and application secret generation in a single cluster created an administrative dependency on GCP for control-plane operations across other clouds. We accepted this risk to avoid the operational complexity of running active-active multi-cloud Vault clusters and data synchronization engines.

Result A single control plane managed GitOps deployments and secret policies, while running workloads remained resilient against control-plane disruptions because secrets were cached locally as in-cluster Kubernetes secrets.

6. Centralized CI/CD Runners

Problem Running builds and integration tests required compute resources without duplicating complex runner management across every cloud provider.

Decision Centralized Azure Pipelines self-hosted agent pools in Azure. Agents ran on Azure Virtual Machine Scale Sets (VMSS) with images built using Packer, maintaining separate pools for standard CPU builds and specialized GPU testing.

Trade-off Images targeting GCP and AWS were built on Azure runners and pushed to remote registries across external network paths during CI.

Result Centralizing runners in Azure leveraged existing, mature build automation and avoided the maintenance overhead of managing duplicate VMSS/runner pools across multiple cloud providers.

The runners were one part of the delivery path. Reusable Azure Pipelines and GitHub Actions workflows handled repeatable build and validation work. Pull requests showed Pulumi previews for infrastructure changes; Trivy scanned container images before release. Container images were published to the target cloud’s registry, and production deployment required an approval gate. Delivery then followed the workload’s ownership model: earlier pipeline-managed Helm releases or, after handoff, Git changes reconciled by Argo CD. Keeping artifact production separate from cluster reconciliation made that handoff possible without stopping teams’ existing builds.

7. Ingress Controller Migration

Context Ingress across the Kubernetes environments was initially standardized on Ingress NGINX. After its retirement was announced in 2025, the platform began an intentional migration to Traefik and Istio, adopting Kubernetes Gateway API patterns and service-mesh capabilities where needed.

The ingress layer was part of cluster operations, not a checkbox after provisioning. Routing and TLS depended on ingress controllers, cert-manager, environment-specific configuration, and the health of the workloads behind each route. Service-mesh integration was introduced where workload traffic needed it; it was not a blanket requirement for every cluster service.

Landing Zone Hierarchy

The Azure landing zones established environment and shared-service boundaries. As the platform expanded, we used corresponding separation tiers across providers:

GCP: Dedicated Projects (dev, test, staging, prod, shared-network, shared-data).
Azure: Dedicated Subscriptions mirroring the environment tiers.
AWS: Dedicated Accounts under AWS Organizations mirroring the environment tiers.

Network transit and peering were managed in the shared network scopes, while persistent shared storage and container registries were housed in the shared data scopes.

Runtime Autonomy & Failure Modes

A central architectural priority was limiting the effect of inter-cloud network failures on running workloads:

**Inter-cloud network interruption**A network interruption leaves local runtime dependencies available, but pauses cross-cloud delivery and private administration.

Inter-Cloud VPN Disruption:
- Behavior: Each cloud environment was self-contained at runtime, using local database instances, caches, and storage. Workloads in GCP or AWS did not make synchronous cross-cloud calls to serve application requests.
- Impact: Running applications could remain available while their local dependencies were healthy. Administrative operations were affected: CI pipelines could not trigger new deployments or push images to remote registries, and engineers could not access remote cluster APIs through the P2S VPN until connectivity was restored.
Platform Management Cluster Outage (Vault / Argo CD):
- Behavior: Because the External Secrets Operator materializes secrets as native Kubernetes secrets in target namespaces, running pods did not query Vault in real time.
- Impact: Running pods continued functioning. Only newly scheduled pods requiring un-cached secrets or deployments requiring active Argo CD synchronization were delayed until the management cluster recovered.

Infrastructure as Code: Pulumi with Python

Infrastructure automation began with the Azure landing-zone rebuild using Pulumi in Python. As GCP and AWS joined the platform, the code was organized across providers:

Per-Cloud Repository Structure: Code was separated into dedicated repositories per cloud provider. This isolated provider SDK versions and ensured that changes to one provider’s infrastructure had no blast radius on the others.
Layered Stack Decomposition: Stacks within each cloud were structured by infrastructure lifecycle:
Infrastructure lifecycle
100%
Scroll to zoom · drag to pan · arrow keys to pan · Escape to close
Foundation stacks export network outputs to data stacks, which in turn provide services to compute stacks.
Infrastructure lifecycleFoundation stacks export network outputs to data stacks, which in turn provide services to compute stacks.
Pulumi Stack References: Higher-level compute stacks consumed outputs (such as VPC IDs, subnets, and registry endpoints) exported by lower-level foundation stacks via dynamic stack references, eliminating hardcoded environment values.
Why Python: Python provided a concise, readable language for infrastructure definitions that was familiar to both platform engineers and data science teams, while supporting standard unit-testing frameworks.

Retrospective & Hindsight

What Worked Well

Cloud-Local Registries: Moving away from a single centralized registry removed routine image pulls from the inter-cloud network path and their associated transfer cost.
Azure Foundation: Rebuilding the inherited Pulumi code into landing zones gave later platform work clear subscription, network, access, and governance boundaries.
Runtime Autonomy: Designing workloads without synchronous cross-cloud runtime dependencies prevented inter-cloud network disruptions from impacting running applications.
Python for Pulumi: Using Python for IaC kept stack definitions approachable and allowed direct reuse of configuration and validation logic.

What I Would Approach Differently

Decentralizing the Platform Control Plane:
- Then: Centralizing Vault and Argo CD in a single GKE cluster was a pragmatic decision that avoided the overhead of multi-cloud data replication.
- Now: In hindsight, that single cluster remained an administrative dependency for deployments in Azure and AWS. If designing this today, I would consider deploying lightweight, local GitOps reconciliation agents and localized secret synchronization within each cloud to decouple administrative operations from a single provider.
Standardizing Multi-Cloud Abstractions Upfront:
- Then: The platform grew incrementally from Azure into GCP and then AWS, leading to per-cloud repositories with slightly different patterns.
- Now: Knowing upfront that the platform would span three clouds, I would invest earlier in a shared, versioned library of Pulumi component modules with standardized interface contracts to enforce consistency across providers.