The Underpowered Retention Test Mistake

We see teams, even top Pubs making this mistake

Buy 500 installs for D1 Retention Test.
Don't realize this few installs = 8-9% margin of error!
Make poor decisions like 'our new build's D1 is worse!'

In the age of hybrid-casual, high-vol prototype testing, it's essential to power tests correctly.

Last year I worked with a team who would buy 500 installs for retention tests, trying to prove D1 Retention and D0 playtime improvements with each build.

They were looking for 3%-4% improvement per build, but getting D1 retention reads with 8% margins of error!

So a build that looks 4% better could truly be 4% worse!

Or vice versa.

Imagine a team feels confident in their current direction, doubles-down, then 'sees' D1 drop 8% in next release.

This whiplash / uncertainty is truly maddening for teams.

Have you ever had this experience?

Below is a menu of viable options.

Assumptions behind the math

"True" D1 of 35%. (for 40%, just add 5.5% to cost)

IPM = 5. (For IPM = 2.5, 2x cost. If 10, 50% cost!)

CPM per tier:

US: $30. Tier-1A (UK+CA): $22 Tier-1B (Netherlands, Sweden, Denmark): $16 Tier-3 (Thailand, Philippines, Malaysia): $5

All calculations to 95% confidence interval

Acquire responsibly, my friends.

May the odds be ever in your favor.