Adoption

Pilot runbook

A structured evaluation for one cluster, designed so each stage is independently reversible and produces a number you can judge. Nothing in this plan requires changing application code or manifests beyond adding PriorityClasses.

Before you start

  • One non-production (or tolerant) cluster on a supported platform (platform support): GKE Standard on COS (GKE ≥ 1.36), or EKS on AL2023/Bottlerocket.
  • Run the preflight on the target node pool and keep the output: every node should report READY (see operations for the one-shot manifest).
  • Install with the production preset (values-production.yaml) so auth, network policy, and the observation layer are on from day one.

Days 0–7 — observe only

Install the chart with no PriorityClasses assigned to workloads. Nothing about scheduling changes; the agent attaches, enforces a configuration equivalent to the default arbitration for unclassified pods, and the observation layer starts publishing per-thread placement, PSI, and utilization. Verify the kill switch once on day one: annotate a node with temper.codes/safe-mode-requested=true, watch it revert to stock CFS, remove the annotation.

Exit numbers: dashboard Savings tab shows an identified $/month (rightsizing slack + idle headroom) for this cluster; zero scheduler restarts on the health metrics.

Days 7–30 — tiers on one node pool

Assign PriorityClasses to the workloads on one node pool: the latency-critical service gets temper-critical, its load generators or secondary services temper-high, everything else defaults. Add one best-effort batch workload at temper-background — this is the tier that converts idle cycles into work. Watch the p99 of the critical service in your own APM against the previous weeks.

Exit numbers: critical-service p99 unchanged (your APM), measured Background CPU consumption > 0 on the Savings tab (realized $/month goes non-zero), no sustained temper_guidance_violation or lint alerts.

Days 30–60 — density

Raise Background replica counts stepwise on the pool (the benchmark methodology is the same experiment, formalized) until node utilization reaches your comfort band, watching the critical p99 at each step. Optionally enable the overcommit webhook on a labeled namespace to reclaim requested-but-unused CPU at admission (density & overcommit).

Exit numbers: utilization at the operating point vs. your fleet baseline; p99 at that operating point vs. day-7.

Days 60–90 — consolidation dry-run, then one shrink

Enable the controller and a TemperPolicy with consolidation: {enabled: true, dryRun: true}. The dashboard surfaces plans (victim, predicted survivor state, feasibility record) beside the savings numbers; nothing executes without an explicit, audit-logged approval. When a plan matches what you'd have done by hand, approve one: tier-ordered PDB-honoring drain under the live SLO guard, and your existing autoscaler removes the empty node. Worst case, by design, is an automatic rollback that leaves the cluster exactly as it was.

Exit numbers: nodes before/after, critical p99 through the drain window, and the realized $/month delta — the three numbers a renewal decision is made on.

What a successful pilot looks like

Concretely: p99 flat at every stage, a realized-savings figure sourced from measured consumption (not projections), one automated node removal with the audit trail to show for it, and zero incidents where the fail-safe was needed — or, equally acceptable, one incident where it fired and the node reverted to stock CFS on its own. Both outcomes are the product working.