LECTURE HUB

🧪Lecture 9 - A-B tests and their traps

🚦 Status🌟 Mastered

🧠 Mastery🦅 Adept

📅 Last Review8 Jun 2025

🧩 Randomisation is what lets you say because. Without it you have two groups that were already different.
Decide the sample size and the stopping date before you start, write them down, and do not look for a winner early.
Randomise the unit that actually experiences the change. If dispatchers share a screen or a customer, they are not independent.
One primary metric, chosen in advance. Twenty metrics will hand you one false win by luck alone.
Report the result as an interval. It worked is not a result.

an experiment is the only time the data owes you an answer (Hale, roughly)
randomise → the two groups differ only by chance and by the thing you did. that is the whole trick
he listed traps and I have personally met every one
peeking: checking daily and stopping when it looks good. I did this in 2021 with the onboarding wizard
what gets randomised matters: packets or shifts. for us: dispatchers or customers
novelty: people behave differently in week one of anything
quiz at the start: 9 of 10. fine. it is fine
our triage test went live this morning. I sized it last week. feeling smug and slightly ill

Check the triage test against tonight's list of traps
Write down the stopping date and tell Bao I am not allowed to move it
Start 📌 Statistics I - Assignment 2 - An A-B test from start to finish with the triage test as the case
Write 🧩 Randomisation as a concept note
Ask whether dispatchers in the two arms sit next to each other

📝LECTURE NOTES