Designing canary metrics that survive skeptical reviewers

Canary metrics fail when they optimize for engineering vanity instead of operator trust. We start every cohort by asking which customer-visible symptom would justify a halt, then derive queries backward from that symptom rather than forward from whatever Prometheus metric is convenient. The second paragraph of practice focuses on pairing windows: multi-burn alerts only make sense when reviewers agree on which window maps to which risk appetite. We document those pairings beside the rollout manifest so night shifts do not improvise thresholds. In the third paragraph, teams rehearse false positives. We inject latency into staging traces until a canary promotion would have failed for the wrong reason, then rewrite the query until the failure mode matches reality. That exercise is deliberately uncomfortable. Finally, we archive the approved queries beside the retro template so post-incident reviews reference the same numbers operators saw during promotion. Consistency beats cleverness when bridges are on fire.