Sylvics Apply for the beta
Incident and rollback playbookReference
REFERENCE

Incident and rollback playbook

๐Ÿ“Œ TL;DR / Quick Summary <div style="background: rgba(0,0,0,0.3); border: 1px dashed rgba(255,255,255,0.2); padding: 15px; border-radius: 8px; color: rgba(255,255,255,0.7); margin-bottom: 30px;"> What the team agreed at the โ˜• Launch retrospective on 19 March, a week after the night it was needed. Priya Raman drafted it; I kept the customer's side in. Who decides, who writes, and when to roll back without a meeting. At 02:00 on 12 March nobody knew the first of those, and it took forty minutes to find out it was me. </div>

๐Ÿ› ๏ธ Instructions / Core Material

  1. Every launch plan names one rollback decider. By name, not by role, with a telephone number and a deputy. For ๐Ÿ“‚ Exceptions Inbox general availability it was me and it was written nowhere.
  2. The decider decides alone. Others advise. If the decider cannot be reached in ten minutes, the deputy has the full authority and does not need to apologise for using it.
  3. Roll back without a meeting when all three are true: customers are seeing wrong data, not merely slow data; the cause is inside what was launched; a rollback path exists and was tested that week.
  4. Roll back the smallest thing. One feature, not the release. See ๐Ÿ“œ Roll back auto-triage on launch night: triage off, everything else left alone.
  5. One writer. From the first minute somebody who is not fixing anything keeps the timeline: time, what was seen, what was done. No names in the sentences, only in the roster.
  6. Customers hear from us first. Customer Success and Support get a plain message before the first dispatcher starts at 06:00, even if it says only what is switched off.
  7. Write-up within five working days, blameless, in the form Priya uses: what happened, what we saw, what we changed.
  8. The feature comes back on a calm afternoon, announced the day before, with the decider at a desk.

โš ๏ธ Key Warnings / Constraints

  • A rollback path that has not been run this week is a diagram.
  • Patch forward at three in the morning is nearly always the wrong answer, and it is always proposed by the most tired person on the call.
  • Carried over unchanged when the teams merged in August. Lucรญa Ortega owns it now, and the decider on the one roadmap is named per item.
  • The playbook is for the night. The reason the night happened was a load test that was too gentle, and no playbook covers that.

๐Ÿ”— Related Entities & Resources

  • โ˜• Launch retrospective: the meeting on 19 March where this was agreed
  • ๐Ÿ“œ Roll back auto-triage on launch night: the decision the playbook is modelled on
  • ๐Ÿ“‚ Exceptions Inbox general availability: the launch it was first needed for
  • Launch readiness checklist: the list that now names the rollback decider as line fifteen