Enterprise EKS Landing Zone

This design was pressure-tested through many revisions until only the decisions that could defend themselves were left. Some patterns that looked mandatory turned out to be habits. Others survived every challenge and stayed. What follows is the reasoning behind each one, not a tour of the icons.

A landing zone is a set of boundaries: between accounts, between network segments, between inbound and outbound traffic, between what a pipeline can touch and what only the cluster can. The AWS services are the vocabulary. The boundaries are the point.

Account structure

The account layout is the first and most expensive decision, made before any workload exists.

The Management account runs Control Tower, Organizations, and IAM Identity Center, and nothing else. It holds org-wide control, so it stays minimal to keep the target small. Everything operational lives one level down in Organizational Units.

Seven OUs:

Security OU. Two accounts. Log Archive is a write-once store for CloudTrail, access logs, and central logs. Audit runs the detection stack as delegated admin: Security Hub, GuardDuty, Detective, Inspector, Macie, Config, Access Analyzer, Prowler. These aggregate findings from every account without logging into them. Findings route to Slack.

Infrastructure OU. The Network account (shared connectivity) and a separate Shared Services account (CI/CD, ECR, Terraform state, KMS). These stay out of the Workloads OU because SCPs that restrict a production workload will break network plumbing and build pipelines if applied to the same scope.

Workloads OU. Splits into Production and Non-Production sub-OUs. This is the boundary an SCP uses to grant production-only permissions and deny them everywhere pre-production, applied automatically to accounts that don't exist yet.

Backup OU. A separate account so a full compromise of production (ransomware, leaked credentials, over-privileged automation) can't reach the backups. Vault Lock in compliance mode means no one, including org admins and root, can delete a recovery point before its retention expires. Resilience Hub scores recoverability from here.

Sandbox OU. Experimentation, no production data.

Policy Staging OU. One empty account. SCPs apply instantly with no dry-run, so a malformed policy breaks production before you know it's malformed. Attach it here first, test against an account with nothing to lose, then promote it.

Suspended OU. Quarantine for decommissioned accounts. Closure takes time and you may need the account for audit evidence, so it moves here under a deny-all-but-billing SCP. A stale account with live credentials is attack surface; this removes it without deleting anything prematurely.

Connectivity

Hub-and-spoke. Every workload account attaches to one hub for egress, inspection, and on-prem reachability instead of solving those per account. That is what the Network account is for.

The default choice is Transit Gateway: cheaper for a single region, well documented. It breaks down at multi-region. TGW is regional, so going global means a gateway per region, peered manually, with route tables kept in sync by hand.

This uses Cloud WAN. Segments are defined once as policy (production reaches shared-services, sandbox reaches nothing, hybrid is isolated) and propagate automatically. Each region gets a Core Network Edge, a regional attachment point on one global core network. Adding a region is adding an edge, not building a second hub.

The honest tradeoff: at current scale (one heavy workload account, a few light ones) a single TGW would do the same job for less. Cloud WAN is not paying for itself yet. It's justified only by the multi-region requirement on the roadmap, which is exactly where manual TGW peering stops scaling. Build it now so expansion is additive.

Two things inside the hub VPC:

NAT Gateways and Firewall Endpoints are per-AZ. Both are AZ-scoped. One shared instance means an AZ failure takes egress down for every private subnet, including healthy AZs, plus cross-AZ data transfer on every packet. One per AZ, each AZ routing to its own, is the only correct layout.

Twingate provides private access to internal endpoints. Engineers reach the private EKS API server (kubectl), internal dashboards, and admin interfaces through it, with no public cluster endpoint exposed. The connector is outbound-only: it dials out to establish the tunnel and accepts no inbound. Runs as an ECS service with two tasks, one per AZ.

Ingress and egress are separate planes

The common mistake is treating a firewall as an inbound perimeter device. That model comes from data centers with servers exposed at the edge. This architecture has no server exposed at the edge, which changes where the firewall goes.

Ingress (public web): User to Route 53 to WAF to CloudFront to private ALB to pods. WAF inspects at layer 7 (SQLi, XSS, bots, rate limiting) at the edge. This is the right tool for HTTP ingress and more capable at it than a network firewall. This traffic never touches Cloud WAN or Network Firewall; it lands privately via CloudFront.

Egress (pods to internet): pod to Cloud WAN to Network hub to Firewall Endpoint to NAT to IGW. This is what Network Firewall is for, and it's the direction that matters more. A compromised pod's next move is outbound: C2, exfiltration, pulling tooling. Outbound filtering (domain allowlists, blocking known-bad destinations) contains it. That's layer 3-4 to arbitrary destinations, which WAF cannot see.

So Network Firewall is outbound-only by design, not by limitation. Inbound is covered by the better tool, and there is no raw internet-facing resource for an inbound rule to protect. Add a public load balancer outside CloudFront later and you route its inbound through the same firewall. Not today.

The load balancer chain that got deleted

CloudFront needs to reach an ALB in a private subnet in the Production account. CloudFront historically required the origin load balancer to be in the same account as the distribution.

That constraint forced a relay: an ALB in the hub account, forwarding to an NLB in Production (for a stable target address as pods churn), forwarding to Production's real ALB, then to pods. Three load balancers to satisfy a same-account rule.

That chain was in the diagram, and it was correct for the constraint.

AWS then shipped cross-account VPC origins. CloudFront now points directly at an ALB in another account, shared through RAM. The relay collapsed to nothing. Two load balancers removed.

The mechanics: CloudFront provisions a managed ENI inside Production's private subnet next to the ALB and connects over the AWS backbone. User to CloudFront is public; CloudFront to ALB is private. The ALB has no public IP and no internet exposure. The only path to it is through CloudFront, which means through WAF.

The point is not the mistake. It's that "standard practice" expires. The relay was standard until it wasn't, and the only way to catch that is to re-check instead of trusting the last time you solved it.

The cluster

Production spans two AZs, each with three subnet tiers: transit (Cloud WAN attachment only), application (the cluster), database.

One cluster across AZs, not one per AZ. The EKS control plane is already multi-AZ. Run one cluster with node groups spread across zones and let the scheduler handle placement and failover. A cluster per AZ doubles operational surface and fights the platform.

EC2 with Bottlerocket, not Fargate. Fargate cannot use the EBS CSI driver. EC2 nodes unlock EBS for block storage, and Bottlerocket (minimal, immutable, container-only) cuts the host attack surface. Cost: no SSH, so node debugging goes through SSM. Deliberate trade.

Namespaces map to blast radius, not tidiness. Each namespace isolates a distinct privilege, credential, or failure domain. Karpenter (creates and terminates EC2) is separated from the load balancer controller (opens network paths), separated from External Secrets (reads the secret store).

Networking runs VPC CNI for IPAM and Cilium in chaining mode on top. Pods keep routable VPC addresses (which the ALB target registration, security groups, and endpoint model all depend on), while Cilium handles eBPF-based NetworkPolicy enforcement and feeds Tetragon for runtime security. VPC-native addressing plus Cilium's policy layer, without giving up either.

Application code lives in the Apps repo; GitHub Actions orchestrates the build, running on CodeBuild-hosted runners (AWS-managed GitHub Actions runners, so the compute sits inside the account rather than on GitHub's infrastructure), and pushes the image to ECR. Platform config lives in a separate Manifest repo that ArgoCD watches and reconciles into the cluster. The pipeline never reaches into the cluster; the cluster pulls. ArgoCD gets its own namespace because it's the only component that can deploy anywhere, and its compromise is the one you contain hardest. "Who can change production" has one answer: ArgoCD, from Git.

KEDA scales on queue depth, not CPU. It watches SQS depth and sets the consumer replica count. KEDA does not read messages; it reads a number and turns it into a replica count. The pods consume. It's a controller, not a data-path component.

Security and observability are in from the start. Dynatrace runs application monitoring via a read-only CSI driver (Bottlerocket blocks host-level agents). Trivy scans images. Kyverno enforces admission policy. GuardDuty runs at the account level.

Data stays out of the cluster: Aurora with a cross-AZ read replica, RDS Proxy for pooling and IAM auth, ElastiCache for Valkey as cache.

Region two: pilot light

Most multi-region diagrams lie. There's a second region drawn, it has never taken a failover, and it wouldn't work if it had to.

Region two here is a pilot light: not active-active, not warm standby, just the components that must be continuously current running live, everything else defined as code and dormant until a declared failover.

Lit, because it cannot be created after the fact:

• Aurora Global Database secondary, replicating from region one, promotable on failover. This is the flame. The data has to already be there.
• RDS Proxy in front of it. Minutes to create, cheap to keep, so creating it mid-incident only adds recovery time.
• ECR and S3 cross-region replication, so spun-up compute has images and assets.
• Cross-region backup copies.

Dark, spun up on failover:

• EKS node groups. This is the compute pilot light turns off. It's the whole model.
• ElastiCache, if it's a cache rather than a system of record. It rebuilds from the data tier.

Provisioned but empty, because empty is free and building under pressure is not:

• VPC, subnets, route tables, security groups.
• A second Core Network Edge on the same core network. Extending is adding an edge; the segmentation policy propagates.

Not duplicated, because global: CloudFront, WAF, Route 53. Region two's load balancer becomes a failover origin on the same distribution. Management, OUs, Log Archive, Audit are org-wide.

The catch: this only works if the dormant half is genuinely defined as code and tested. "Spin up on disaster" becomes "find out during the disaster that the apply fails" unless you've run the drill. An untested pilot light is not DR. The real deliverable is a rehearsed runbook, not the diagram.

Roadmap

Cost governance is the next build. A dedicated billing/FinOps account receives Cost and Usage data from Management, and the AWS FinOps agent runs against it to surface anomalies, idle spend, and rightsizing opportunities in plain language, so finance and engineering both get answers without access to the Management account itself.

AI-assisted operations is the direction after that, and it builds on what's already there. Dynatrace's Davis engine already does detection and root-cause; the missing piece is remediation, moving from "here's what broke and why" to "here's the fix." The AWS DevOps Agent correlates across CloudTrail, config changes, and deploys, and an in-cluster MCP server lets you investigate the cluster in plain English over Bedrock. The goal is faster recovery on routine failures. The guardrail stays the same: the agent recommends, ArgoCD is still the only thing that writes to production.

Summary

An enterprise landing zone is mostly about where the boundaries go: between accounts, segments, ingress and egress, pipeline and cluster. Re-check the patterns you're most sure of. The relay chain that looked like standard practice was a workaround for a constraint that no longer existed.