About the Episode:
In this episode of Statistically Speaking, Kerry Cunningham and Sara Boostani shift focus from hypothesis testing to the practical question of how to pull a good sample in the first place. Because even the most rigorous statistical testing is only as sound as the sample behind it. Kerry and Sara walk through the key principles of good sampling — from defining your population clearly, to accounting for the very human tendency of survey respondents to tell you what they think you want to hear.
Topics Covered:
- Defining your population
- Before you collect a single data point, you need a clear definition of who you are studying — and that definition sets the boundaries of every generalization you can make
- In their marketer compensation study, Kerry and Sara sampled B2B marketers in North America — which means their findings can only be generalized to that specific group, not to all marketers globally
- Being specific about your population isn’t a limitation, it’s what makes your findings defensible
- Adequate sample size
- Sampling too few people from a large population produces unreliable results — no amount of statistical testing can fix an undersized sample
- The right sample size depends on your population, your research questions, and how much confidence you want in your results
- Tools like SurveyMonkey’s sample size calculator allow you to input your estimated population size and desired confidence level to get a recommended sample size — and this is worth doing before you run a study.
- When reading other people’s research, always ask: how many people did they sample, and is that enough to generalize to the population they’re claiming to represent?
- Alignment between sample and population
- A good sample isn’t just large enough — it needs to reflect the population you’re studying across the dimensions that matter: industry, company size, seniority, region, and so on
- Understanding which variables matter and which don’t helps researchers know when they can speak broadly and when they need to be more specific
- Response bias and the observer effect
- People don’t always answer surveys honestly — even anonymously, respondents often try to impress whoever they think is asking
- When respondents know who is conducting the research, their answers can shift systematically — particularly on questions about performance, compensation, or anything tied to how they want to be perceived
- Researchers can build scales specifically designed to detect whether a group of respondents is inflating their answers — and when bias is detected, it can be measured and adjusted for mathematically
- Consistency in measurement
- If you’re running an annual study, changes in how you ask questions or structure your response scales can create the illusion of change where none exists
- Consistency in measurement is also something to look for when consuming other people’s research
Key Takeaways
- A good sample starts with a precise definition of who you are studying — that definition determines what you can and cannot claim
- Always ask whether a study sampled enough people to reliably represent the population it claims to speak for
- Response bias is real and measurable — good researchers test for it and account for it
- When running longitudinal research, consistency in how you measure matters as much as consistency in who you measure
- As a reader of research, these are the questions to bring to every benchmark report you encounter: who did they study, how many, and did they measure them the same way every time?
Related Resources
