As described here, statistical significance measures how much confidence you can have that a result from a study (e.g., a survey, observation, or experiment) represents the real world. That is, how likely are you to find the same results as those obtained in that survey, observation, or experiment if you were to go out and replicate the study yourself?
This applies to comparing conversion rates between BDRs, form-fill rates in an A/B test, sales cycle lengths for different solutions, or virtually anything else you might like to measure or compare.
Factors Influencing Statistical Significance
Whether a sample of any size can return a statistically significant result depends on a wide variety of factors, principally sample size, variability, and effect size.
Sample Size
The closer in size your sample is to the population or full set of things you are studying, the more likely you are to be able to find statistical significance.
Variability
Another critical factor is variability. If there is a large range in the measures, scores, or responses of the thing you are measuring, then finding statistically significant patterns is harder. For example, let’s say that your company’s BDRs produce anywhere from 10 to 100 meetings per month, with the fluctuations obeying no clear pattern. Your company implements a new cadence for prospecting accounts and wants to know if the results it sees after the first month are due to the new cadence. In this high variability situation, it will be hard to know in just a month – or even several months – if any productivity difference is due to the new cadence or just happened by chance. However, if the team produces between 95 and 100 meetings per month regularly, with months rarely going below or above that small range, spotting differences due to new innovations will be much easier – both with and without statistics.
Effect size
If you are trying to find subtle changes or differences, it is likely that you will need a larger sample than if the differences you are looking for are larger ones. In the example of implementing a new BDR cadence on a team where productivity varies by plus or minus 5 opportunities per month, an effect size of five or fewer opportunities will be difficult to spot and require many months of data.
Control of the Environment
When researchers conduct controlled laboratory experiments where all or nearly all of the factors that might influence the experimental outcome are tightly controlled, very small samples are often sufficient to identify a statistically significant effect (or to rule one out). Put another way, if you are testing whether A affects B, and the only thing that can affect B is A, it doesn’t take many samples or experimental trials to see whether A does in fact affect B.
In our BDR example, there will always be many factors influencing BDR productivity in a given month, and many factors beyond your control will influence the number of opportunities produced. Finding a statistically meaningful difference in such an environment will require a substantial sample.




