Engineering / Enterprise scale, now

Governed landing pads for workloads

Kubernetes and serverless containers as the two places a workload is allowed to land, each wrapped in Terraform modules that make the secure configuration the default, with observability and runbooks attached before the first deploy.

GCPKubernetesCloud RunTerraformCI/CDobservabilityreliability

The problem

A large organization does not have one platform team's worth of workloads. It has dozens of teams with dozens of opinions about how a container should run. Left to themselves, they converge on nothing, and the security and reliability posture of the estate becomes the sum of everyone's least careful day.

The alternative is not a single mandated stack. It is a small number of well-understood landing pads, each with a paved road, where the paved road is faster than going off it.

Two landing pads

Kubernetes, for the things that need it. Clusters are private, with workload identity instead of node keys, network policy on by default, image provenance checked at admission, and a namespace-per-team model where the team gets a quota, a service account, and a set of policies it cannot remove. The cluster is a Terraform module; a new cluster is a pull request.

Serverless containers, for most things. Request-driven services land on a managed container runtime with private ingress, service-to-service authentication, and egress through the controlled path. It scales to zero, it has no nodes to patch, and for most workloads it is simply the right answer. Making it the easy answer was the work.

The paved road

  • Modules with opinions. A workload module asks the questions a reviewer would ask (what does it talk to, what identity does it run as, where does it log) and refuses configurations that fail them. Exceptions exist, and they are visible in the plan.
  • Pipelines that know the rules. Policy checks run in the delivery pipeline, against the plan, before anything is real. Findings appear as review comments, not as tickets three weeks later.
  • Observability as part of the module. Logs, metrics, traces, and a default dashboard come with the workload. A team does not have to remember to add them, because they cannot forget.
  • Runbooks before go-live. The module output includes the runbook skeleton: how to roll back, how to scale, what the alerts mean, who owns it at three in the morning.

What I learned

The interesting work in platform engineering is deciding what to make impossible. Everything you make impossible is a class of incident that never happens and a review that never needs to occur. The teams who complained about the guardrails in month one were the teams who shipped fastest by month six, because they stopped having to think about the things the platform had already decided.