The problem, at pattern level
A large enterprise's cloud footprint is not one application. It is hundreds of projects owned by dozens of teams, some of which have to reach on-premises systems over private links, some of which must never touch the internet, and some of which produce services that other projects consume. The default outcome, left alone, is a mesh of peerings that nobody can draw and that fails in ways nobody predicted.
The design goal was the opposite: a topology that a new engineer can draw from memory, that a new workload can join without human negotiation, and that a security reviewer can reason about from the Terraform alone.
The shape
- Hubs, not peerings. Network Connectivity Center hubs, one per connectivity domain, with spokes attached by declaration. Six hubs was the number that made the domains legible without making the routing tables absurd. A spoke is a Terraform module call, not a conversation.
- Shared VPC as the tenancy boundary. Host projects own the network; service projects own the compute. Teams get subnets, not networks. The people who can change routing are a short list.
- Hybrid as a first-class path. Dedicated interconnect and VPN attachments terminate on Cloud Routers in the hub domain. On-premises prefixes are advertised once and learned everywhere they should be, and nowhere they should not.
- Producers and consumers. Services that other projects need are published over Private Service Connect. Consumers get an endpoint in their own subnet; the producer's network is never exposed. This is what let the topology stay hub-and-spoke instead of decaying into a mesh.
- Egress by policy. Internet egress goes through a controlled path with inspection. A workload that needs a new destination adds it to an allow-list in code, with a reviewer, with a diff.
What made it hold
Three things, none of them clever.
- Terraform modules with opinions. The spoke module does one thing and refuses to do anything else. Its inputs are the questions a network engineer would ask; its outputs are the facts a security reviewer would want. Drift shows up in plan output, not in an incident.
- A living topology model. Every hub, spoke, attachment, route advertisement, and firewall rule is exported nightly into a graph that can be queried and drawn. Exposure paths are questions against the model, not archaeology. The diagrams on the architecture wiki are generated from the same model, so they are never stale.
- Runbooks written before go-live. For every failure mode we could name (an attachment down, a route leak, a hub saturating), a runbook that a person who did not design the system could execute at three in the morning.
What I would say at a conference
Connectivity is a governance problem wearing a networking costume. The routing is the easy part. The hard part is designing the system so that the correct thing is also the path of least resistance for the team that just wants to ship, and so that the incorrect thing is visible in a plan before it is real.