Find Sample Sizes - dummies

By Jeff Sauro

Part of Customer Analytics For Dummies Cheat Sheet

With customer analytics, collecting data from a sample of customers costs a lot less and takes a lot less time than measuring every customer. The level of precision you get from even a small sample is usually sufficient to make decisions from the data.

If you have a stand-alone survey or study (no comparisons), here’s the margin of error you will have for each sample size (based on a proportion of .50 and 95% confidence).

95% Margin of Error (+/-) Sample Size
24% 13
20% 21
15% 39
14% 46
13% 53
12% 63
11% 76
10% 93
9% 115
8% 147
7% 193
6% 263
5% 381
4% 597
3% 1,064
2% 2,398

If you’re conducting a comparison study (using surveys, usability studies, or findability studies), here are the sample sizes needed to be able to detect a difference (based on a proportion of .50, 90% confidence, and 80% power). The Sample Size within Subjects column represents same participants on each version, and the Sample Size between Subjects column represents different participants on each version.

Difference to Detect Sample Size within Subjects Sample Size between Subjects
50% 17 22
40% 20 34
30% 29 64
20% 50 150
12% 93 426
10% 115 614
9% 130 760
8% 148 962
7% 171 1,258
6% 202 1,714
5% 246 2,468
4% 312 3,860
3% 421 6,866
2% 640 15,452
1% 1,297 61,822

If you’re conducting an A/B test to compare conversion rates, here are the sample sizes you need to detect differences from design A to design B for differences of .1% to 50% (assumes 90% confidence and 80% power).

Difference Each Group Total Design A Conversion Rate Design B Conversion Rate
0.1% 592,905 1,185,810 5% 5.1%
0.5% 24,604 49,208 5% 5.5%
1.0% 6,428 12,856 5% 6.0%
5.0% 344 688 5% 10.0%
10.0% 112 224 5% 15.0%
20.0% 40 80 5% 25.0%
30.0% 23 46 5% 35.0%
40.0% 15 30 5% 45.0%
50.0% 11 22 5% 55.0%