How to Determine the Correct Sample Size
If you aren’t measuring every transaction or surveying every customer, you’ll have to deal with the uncertainties that come with sampling a portion of your customer population. Even if you sample everyone, you’ll likely want to make estimates about future customers or future transactions, and to do so, you’ll still have to deal with the uncertainty.
In general, it costs more money and takes more time to either sample or analyze data from a large database, so you must put some thought into how large of a sample size you need.
In general, you should consider two important concepts when planning your sample sizes:
You need larger sample sizes to detect smaller differences.
A new design, promotion, or feature may improve customers’ attitudes or sales, but if the increase is small (something like a 5% increase), unless your sample size is large enough, that difference won’t be distinguishable from random fluctuations in the data.
For very large sample sizes, almost all differences will be statistically significant.
Statistically significant essentially means that the differences are not likely due to sampling error. However, statistically significant does not mean the findings are noteworthy or important. Will customers notice a one-second reduction in the time it takes to rent a car online? Probably not. Although it’s a good idea to drive increases in positive attitudes and sales, watch out for spending a lot of effort for little return.