Adoption
Pilot runbook
A structured evaluation for one cluster, designed so each stage is independently reversible and produces a number you can judge. Nothing in this plan requires changing application code or manifests beyond adding PriorityClasses.
Before you start
- One non-production (or tolerant) cluster on a supported platform (platform support): GKE Standard on COS (GKE ≥ 1.36), or EKS on AL2023/Bottlerocket.
- Run the preflight on the target node pool and keep the output:
every node should report
READY(see operations for the one-shot manifest). - Install with the production preset
(
values-production.yaml) so auth, network policy, and the observation layer are on from day one.
Days 0–7 — observe only
Install the chart with no PriorityClasses assigned to workloads. Nothing
about scheduling changes; the agent attaches, enforces a configuration
equivalent to the default arbitration for unclassified pods, and the
observation layer starts publishing per-thread placement, PSI, and
utilization. Verify the kill switch once on day one: annotate a node with
temper.codes/safe-mode-requested=true, watch it revert to stock
CFS, remove the annotation.
Exit numbers: dashboard Savings tab shows an identified $/month (rightsizing slack + idle headroom) for this cluster; zero scheduler restarts on the health metrics.
Days 7–30 — tiers on one node pool
Assign PriorityClasses to the workloads on one node pool: the
latency-critical service gets temper-critical, its load
generators or secondary services temper-high, everything
else defaults. Add one best-effort batch workload at
temper-background — this is the tier that converts idle
cycles into work. Watch the p99 of the critical service in your own
APM against the previous weeks.
Exit numbers: critical-service p99 unchanged (your APM), measured
Background CPU consumption > 0 on the Savings tab (realized
$/month goes non-zero), no sustained
temper_guidance_violation or lint alerts.
Days 30–60 — density
Raise Background replica counts stepwise on the pool (the benchmark methodology is the same experiment, formalized) until node utilization reaches your comfort band, watching the critical p99 at each step. Optionally enable the overcommit webhook on a labeled namespace to reclaim requested-but-unused CPU at admission (density & overcommit).
Exit numbers: utilization at the operating point vs. your fleet baseline; p99 at that operating point vs. day-7.
Days 60–90 — consolidation dry-run, then one shrink
Enable the controller and a TemperPolicy with
consolidation: {enabled: true, dryRun: true}. The dashboard
surfaces plans (victim, predicted survivor state, feasibility record)
beside the savings numbers; nothing executes without an explicit,
audit-logged approval. When a plan matches what you'd have done by hand,
approve one: tier-ordered PDB-honoring drain under the live SLO guard, and
your existing autoscaler removes the empty node. Worst case, by design, is
an automatic rollback that leaves the cluster exactly as it was.
Exit numbers: nodes before/after, critical p99 through the drain window, and the realized $/month delta — the three numbers a renewal decision is made on.
What a successful pilot looks like
Concretely: p99 flat at every stage, a realized-savings figure sourced from measured consumption (not projections), one automated node removal with the audit trail to show for it, and zero incidents where the fail-safe was needed — or, equally acceptable, one incident where it fired and the node reverted to stock CFS on its own. Both outcomes are the product working.