Loyalty Builders’ mathematician Bill Vorias offers a hypothetical example of how a lowered confidence-level requirement can work.
Given a list of 2,000 e-mail addresses, you send two different messages to two segments of 1,000 each. Response runs 7% and 10%, respectively.
The question: Is the 10% side of the split a true winner or just a random error? According to Vorias, 72% of the time this result represents a significant difference. Not 95%, but not bad. To get to 90% confidence, a sample size of roughly 1,700 for each group (or 3,400 for the campaign) would be needed. Looking at it from the other direction, if there’s only 2,500 e-mail addresses to play with, you’d have to be satisfied with an 80% confidence that the difference is meaningful.
For anyone who wants to reproduce these numbers, Vorias says you need to know that the test was run at an alpha level of 0.05 and was a one-sided (vs. two-sided) trial.
Network
