Statistics for Sociologists/Chapter 11: Sampling Distributions, Standard Error, and Statistical Power

Chapter 11: Sampling Distributions, Standard Error, and Statistical Power Part III taught you to describe a sample. Part IV, starting here, teaches you to say something about the population that sample came from — and the entire logical structure that makes this possible rests on one concept that reliably confuses more students than anything else in an introductory statistics course. This chapter names that confusion directly, on purpose, and works through it slowly. If it takes you a couple of readings, you're in good company; it takes most people a couple of readings. The Problem of Inference You have a sample. You want to say something about a population you didn't fully observe. This is the entire problem inferential statistics exists to solve, and it's worth sitting with how strange the ambition actually is: you interviewed roughly 2,000 people, and you want to say something trustworthy about roughly 260 million. This only works because of how the sample was collected (recall Chapter 3's discussion of probability sampling) and because of the mathematical machinery this chapter introduces. Sampling Variability If you drew a different sample of 2,000 American adults tomorrow, you would not get exactly the same mean age, the same proportion identifying as "very happy," or the same anything. This is sampling variability: different random samples from the same population produce different results, purely by chance, even when nothing about the underlying population has changed. This isn't a flaw in survey methodology — it's an unavoidable, quantifiable fact about drawing any sample smaller than the full population, and the entire logic of inference is built on being able to measure exactly how much variability to expect. The Sampling Distribution of the Mean Here's the genuinely strange idea at the center of this chapter: imagine drawing a huge number of different samples — say, 10,000 separate samples of 2,000 respondents each — from the same population, computing the mean of each one, and then looking at the distribution of those means. That distribution — a distribution not of individual data points, but of a statistic computed from many different samples — is called the sampling distribution of the mean. Notice what makes this different from anything in Part III: every distribution you plotted in Chapters 7 through 10 was a distribution of raw data — individual respondents' ages, incomes, happiness scores. A sampling distribution is a distribution of sample means, one value per hypothetical sample, not one value per respondent. This is the single most common point of confusion in the entire course, so it's worth restating plainly: you are not looking at data anymore. You are looking at a distribution of a statistic computed repeatedly across imaginary repeated samples. You never actually collect those 10,000 samples in real research — you have exactly one sample, the one you actually gathered. The sampling distribution is a theoretical construct: what would happen, across many hypothetical samples, if you kept repeating your study. It's a strange thing to reason about precisely because it isn't something you directly observe. R lets you build genuine intuition for it anyway, through simulation: set.seed(42) sample_means <- replicate(1000, mean(sample(gss$AGE, size = 30, replace = TRUE), na.rm = TRUE)) hist(sample_means, main = "Simulated Sampling Distribution of the Mean (n = 30)", xlab = "Sample Mean Age") This code draws 1,000 different samples of 30 respondents each (with replacement, simulating what repeated independent sampling would look like), computes the mean age in each one, and plots the distribution of those 1,000 means. Run it, and you'll see something that looks conspicuously like a normal distribution — even though the underlying AGE variable itself is not perfectly normal (recall Chapter 10's discussion of its mild right skew). alt=A histogram of a simulated sampling distribution of means.|center|thumb|800x800px|This is a histogram of a simulated sampling distribution of means for the variable AGE in the 2010 GSS. The Central Limit Theorem That's not a coincidence — it's one of the most useful results in all of statistics, the Central Limit Theorem (CLT): regardless of the shape of the original population distribution, the sampling distribution of the mean approaches a normal distribution as sample size increases, provided the sample size is reasonably large (a common rule of thumb is n ≥ 30, though the exact threshold depends on how non-normal the original population is). No matter how weird your population distribution looks — even something as aggressively right-skewed as income — sample enough from it, and the sampling distribution of the mean will be approximately normal. This is either deeply reassuring or deeply suspicious, depending on your disposition. This book won't prove the CLT (that's a semester of mathematical statistics in its own right); it will simply demonstrate it, repeatedly, and trust that watching it happen in simulation builds more real understanding than watching it get proven on a chalkboard. The CLT is the reason virtually every technique in the rest of this book — confidence intervals, t-tests, and beyond — is allowed to lean on normal-distribution math even when the underlying data obviously isn't normal. The data doesn't need to be normal. The sampling distribution of the statistic does, and the CLT is what delivers that, given enough data. Standard Error The standard error (SE) is the standard deviation of the sampling distribution — a measure of how much sample means bounce around from sample to sample. For the sampling distribution of the mean: where σ is the population standard deviation (in practice, usually estimated with the sample standard deviation) and n is the sample size. Notice what's in the denominator: as n increases, the standard error shrinks — larger samples produce more consistent, less variable estimates of the population mean, which matches basic intuition (a poll of 10,000 people should be more stable than a poll of 10 people) with an actual formula behind it. s <- sd(gss$AGE, na.rm = TRUE) n <- sum(!is.na(gss$AGE)) se <- s / sqrt(n) se Watch SE shrink as n grows: sapply(c(10, 30, 100, 1000), function(n) s / sqrt(n)) This is worth internalizing as a genuine mathematical fact, not just a vague notion: because n is under a square root, quadrupling your sample size only cuts your standard error in half. Diminishing returns set in early — doubling an already-large GSS-sized sample buys you noticeably less precision than doubling a small pilot study would. Statistical Power Statistical powerCohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum Associates. is the probability of detecting a real effect, if one genuinely exists in the population, given your study's sample size, effect size, and chosen significance level (α). Power depends on three things, and understanding the relationship between them is one of the more practically useful things this book will teach you: Sample size. Larger samples → more power, all else equal — a direct consequence of the shrinking standard error above. Effect size. Larger true effects are easier to detect than subtle ones, for the same reason a shout is easier to hear across a noisy room than a whisper. Alpha level. A more lenient α (say, 0.10 instead of 0.05) increases power, at the direct cost of a higher false-positive rate — there's no free lunch here, only a trade-off you're choosing explicitly or by default. Why does this matter, beyond study design? Because underpowered studies are a direct, well-documented contributor to the replication crisis from Chapter 1. A study with insufficient power to detect a real effect will, quite often, simply fail to find it — and even when it does find something "significant," an underpowered study's significant results tend to be exaggerated in size (a phenomenon sometimes called the "winner's curse"), because only the inflated, lucky estimates clear the significance bar in a noisy, underpowered sample. library(pwr) pwr.t.test(d = 0.5, sig.level = 0.05, power = 0.80, type = "two.sample") pwr.chisq.test(w = 0.3, df = 1, sig.level = 0.05, power = 0.80) pwr.anova.test(k = 3, f = 0.25, sig.level = 0.05, power = 0.80) The pwr packageChampely, S. pwr: Basic Functions for Power Analysis. R package. https://CRAN.R-project.org/package=pwr implements these calculations using the effect-size conventions from Cohen (1988), cited above. Each of these functions solves for whatever argument you leave out — supply the effect size, significance level, and desired power, and pwr tells you the sample size a well-designed study would need. Supply the sample size instead, and it tells you the power you actually had. Here's the sociologist's practical takeaway, and it's a genuinely important one for how you'll read every GSS-based result in the rest of this book: with a GSS-sized sample (n > 1,500), you have enough statistical power to detect even tiny effects as "statistically significant." A study with 80% power to detect a medium effect is a well-designed study. A study so large it has 80% power to detect only a large effect is, in a specific sense, wasting its own sample size — and a study so large it can detect effects too small to matter at all is exactly why this book has insisted on effect sizes, alongside p-values, since Chapter 1. Power and effect size are two sides of the same coin, and Chapter 13 is where that coin gets spent. Chapter Summary Inferential statistics uses a sample to say something trustworthy about a population, and the sampling distribution — the theoretical distribution of a statistic (like the mean) computed across many hypothetical repeated samples — is the conceptual bridge that makes this possible. It's a genuinely confusing idea because it's a distribution of statistics, not of raw data, and simulation in R is one of the best ways to build real intuition for it. The Central Limit Theorem guarantees that this sampling distribution approaches normality as sample size grows, regardless of the shape of the original population — which is why so much of inferential statistics gets to lean on normal-distribution math. The standard error quantifies how much sample means vary from sample to sample, shrinking as sample size grows (though with diminishing returns, since it depends on the square root of n). Statistical power — the probability of detecting a real effect — depends on sample size, effect size, and alpha, and underpowered studies are a direct contributor to the replication crisis from Chapter 1. With a GSS-sized sample, power is rarely the limiting factor; the limiting factor is remembering that "statistically significant" and "big enough to matter" are different claims, a theme Chapter 13 picks up directly. Practice Examples Example 1: Simulating a Sampling Distribution The Research Question: What does the sampling distribution of the mean actually look like, built from real GSS data rather than just described in the abstract? The Code: set.seed(42) sample_means <- replicate(1000, mean(sample(gss$AGE, size = 30, replace = TRUE), na.rm = TRUE)) hist(sample_means, main = "1,000 Simulated Sample Means (n = 30 each)", xlab = "Sample Mean Age") mean(sample_means) sd(sample_means) The Output: [1] 48.0 # mean of the sample means [1] 3.3 # SD of the sample means (this IS the simulated standard error) The Interpretation: The mean of the 1,000 simulated sample means lands close to the true sample mean of AGE in the full dataset — exactly what should happen, since the sampling distribution is centered on the population value it's estimating. The standard deviation of those 1,000 means (3.3) is the simulation's empirical estimate of the standard error, and it should closely match the formula-based calculation, sd(gss$AGE, na.rm = TRUE) / sqrt(30), computed directly in the next example — two completely different routes to (approximately) the same number. Follow-Up Challenge: Rerun the simulation with a sample size of 100 instead of 30. How does the spread of the histogram change? Does the shape become more or less normal-looking? Example 2: Standard Error and Sample Size The Research Question: How, precisely, does standard error change as sample size increases? The Code: s <- sd(gss$AGE, na.rm = TRUE) sample_sizes <- c(10, 30, 100, 500, 1500) se_values <- s / sqrt(sample_sizes) data.frame(n = sample_sizes, SE = round(se_values, 3)) The Output: n SE 1 10 5.590 2 30 3.228 3 100 1.768 4 500 0.791 5 1500 0.456 The Interpretation: Standard error drops sharply at first (going from n = 10 to n = 30) and then more gradually as sample size keeps climbing — the square root in the denominator means each additional respondent buys you less precision than the last. Going from 100 to 1,500 respondents (a 15-fold increase in sample size) only shrinks the standard error by about a factor of 4, not 15 — a concrete illustration of the diminishing-returns point made in the chapter text. Follow-Up Challenge: At what sample size does the standard error drop below 0.5? Solve for it algebraically first, then confirm with R. Example 3: Sample Size Needed for 80% Power The Research Question: How large a sample would a two-group comparison need to reliably detect a medium effect (Cohen's d = 0.5)? The Code: library(pwr) pwr.t.test(d = 0.5, sig.level = 0.05, power = 0.80, type = "two.sample") The Output: Two-sample t test power calculation n = 63.77 d = 0.5 sig.level = 0.05 power = 0.8 alternative = two.sided The Interpretation: Roughly 64 respondents per group (about 128 total) would be needed for 80% power to detect a medium effect at the conventional α = 0.05. Compare that to the GSS's typical sample size of 1,500-plus: a GSS-based comparison is drastically overpowered for detecting medium effects, meaning it will reliably flag even quite small effects as statistically significant — which is exactly the setup for Chapter 13's central argument about why effect sizes have to accompany every p-value in this book. Follow-Up Challenge: Rerun the calculation for a small effect (d = 0.2) and a large effect (d = 0.8). How dramatically does the required sample size change? What does this tell you about the cost of trying to detect subtle effects with a modest sample? References