Statistics for Sociologists/Chapter 10: The Normal Distribution and Areas Under the Curve

Chapter 10: The Normal Distribution and Areas Under the Curve Everything in this book from here to the final chapter rests, in one way or another, on the shape you're about to meet properly. It has shown up implicitly in every histogram so far, and it's about to become explicit, because the rest of inferential statistics — confidence intervals, t-tests, chi-square tests, regression — is built on top of it. This chapter is the hinge between "describing data" (Part III) and "drawing conclusions from data" (Part IV). Take your time with it. What the Normal Distribution Is (and Isn't) The normal distribution is a theoretical probability distribution — a mathematical idealization, not an empirical claim that "most things in the world look like this." It's symmetric and bell-shaped, with mean, median, and mode all equal to the same central value, and its entire shape is fully determined by just two numbers: the mean (μ) and standard deviation (σ). The normal distribution appears everywhere in nature: heights, measurement errors, and the combined effect of many small, independent random influences all tend to produce something close to this bell shape. It does not appear in income, which instead follows a distribution that can only be described as aggressively right-skewed, per the last two chapters' extended meditation on exactly that fact. The normal distribution is a genuinely useful model for a great many things. It is not a universal law of the social world, and treating it as one is a fast way to misuse every technique in the rest of this book. alt=A normal distribution curve.|center|thumb|800x800px|The normal distribution curve is symmetrical. The Empirical Rule For any normal distribution, roughly 68% of values fall within 1 standard deviation of the mean, 95% fall within 2 standard deviations, and 99.7% fall within 3. This "68-95-99.7 rule" is worth memorizing, because it gives you an immediate, rough sense of how unusual a given value is, long before you reach for a calculator: a value more than 2 SDs from the mean is already in roughly the most extreme 5% of the distribution, combining both tails. Z-Scores: Standardizing a Raw Score A z-score converts any raw value into a standardized measure of how many standard deviations it sits above or below the mean: A z-score of 0 means "exactly average." A z-score of +2 means "two standard deviations above average." A z-score of −1.3 means "1.3 standard deviations below average." The great advantage of a z-score is that it strips away the original units entirely — a z-score of +1 means the same relative thing whether the underlying variable is measured in years, dollars, or degrees, which makes z-scores directly comparable across completely different variables in a way raw scores never can be. Here's a self-referential example, since you've likely lived this exact scenario: suppose you score an 85 on an exam with a class mean of 75 and a standard deviation of 10. z <- (85 - 75) / 10 z [1] 1 A z-score of exactly 1 means your score sits precisely one standard deviation above the class average — a solidly good result, without needing to know anything about the exam's raw point scale to recognize that. Computing z-scores for a whole vector in R, manually and with scale(): gss$EDUC_z_manual <- (gss$EDUC - mean(gss$EDUC, na.rm = TRUE)) / sd(gss$EDUC, na.rm = TRUE) gss$EDUC_z_scale <- scale(gss$EDUC) # does the same thing in one function call scale() is a genuine convenience once you understand what it's doing under the hood — which is exactly the manual formula above, applied to every value in the vector at once. Areas Under the Curve as Probabilities For a normal distribution, the total area under the curve equals 1 (100% of all possible outcomes), and the area under any portion of the curve represents the probability of landing in that range. Older textbooks made you look this up in a printed z-table in the back of the book. You have a computer. Use it. pnorm(1) # proportion of a standard normal distribution BELOW z = 1 1 - pnorm(1) # proportion ABOVE z = 1 pnorm(1) - pnorm(-1) # proportion BETWEEN z = -1 and z = 1 (should be ~0.68, per the empirical rule) qnorm(0.95) # the z-score below which 95% of the distribution falls pnorm() goes from a value to a probability; qnorm() goes the other direction, from a probability to a value — the two functions are inverses of each other, and you'll use both constantly starting in Chapter 12's confidence intervals. alt=A normal distribution with 1 standard deviation above and below the mean shaded.|center|thumb|800x800px|A normal distribution with 1 standard deviation above and below the mean shaded. Assessing Normality Before you can lean on any technique that assumes normality (and several of the most important techniques in Part IV do), it's worth actually checking whether a given variable looks reasonably normal. Histograms are the simplest first check — does the shape look roughly bell-like, or clearly skewed/bimodal? hist(gss$AGE, main = "Distribution of Age") Q-Q plots (quantile-quantile plots) are a more precise visual tool: they plot your data's quantiles against the quantiles a perfectly normal distribution would produce. If your data are normal, the points fall approximately along a straight diagonal line; systematic curves or bends away from that line reveal specific kinds of departure from normality (heavy tails, skew, and so on). qqnorm(gss$AGE) qqline(gss$AGE) # adds the reference line for comparison The Shapiro-Wilk testShapiro, S. S., & Wilk, M. B. (1965). An analysis of variance test for normality (complete samples). Biometrika, 52(3/4), 591–611. formalizes this into a hypothesis test, with a null hypothesis that the data are normally distributed: shapiro.test(gss$AGE) Here's the honest caveat this test needs, and one that gets skipped in a lot of introductory treatments: the Shapiro-Wilk test becomes nearly useless with large samples. With enough data — and the GSS, at roughly 1,500-2,000-plus respondents, usually qualifies — the test will detect and flag even the tiniest, practically meaningless departure from perfect normality as "statistically significant," because with n in the thousands, the test has the power to notice deviations too small to matter for anything you'd actually do with the data. Above roughly n = 5,000, it is genuinely common for every real-world variable to fail a Shapiro-Wilk test, regardless of how normal it looks to the human eye on a histogram. This is a preview of a theme that will return with real force in Chapter 11: statistical significance and practical importance are not the same thing, and a sufficiently large sample can make that gap embarrassingly obvious. The practical takeaway: for large samples, trust your eyes (the histogram, the Q-Q plot) over the Shapiro-Wilk p-value. Visual assessment isn't a lesser, less rigorous method here — for large n, it's arguably the more informative one. Chapter Summary The normal distribution is a theoretical, symmetric, bell-shaped model fully defined by its mean and standard deviation — genuinely common in nature, but not a universal description of social data (income, notably, is not normal). The empirical rule (68-95-99.7%) gives a quick intuition for how unusual a value is based on how many standard deviations it sits from the mean. Z-scores standardize any raw value into these standard-deviation units, making otherwise incomparable variables directly comparable. R computes exact areas under the normal curve with pnorm() and qnorm(), replacing the printed z-tables of statistics courses past. Assessing whether a variable is actually normal combines histograms, Q-Q plots, and the Shapiro-Wilk test — but the Shapiro-Wilk test's power to detect trivial deviations in large samples means visual inspection often deserves more trust than the p-value alone, especially with a dataset the size of the GSS. Practice Examples Example 1: Z-Scores for Education The Research Question: How unusual is a given respondent's education level, relative to the rest of the sample? The Code: gss$EDUC_z <- scale(gss$EDUC) # Look at a few specific respondents head(data.frame(EDUC = gss$EDUC, EDUC_z = gss$EDUC_z), 5) # Look for specific values gss$EDUC_z[match(20,gss$EDUC)]; The Output: EDUC EDUC_z 1 16 0.81 2 16 0.81 3 8 -1.73 4 10 -1.10 5 0 -4.27 # exceptionally low z-score The Interpretation: Respondent 5, with 0 years of education, has a z-score below 4 — comfortably in the bottom few percent of the education distribution by the empirical rule. If you find the z-score for someone with 20 years of education, it is just above 2 (2.08), which is well above average, approaching the tail of the distribution. Note how the z-score reframes "20 years of education" from a bare fact into a statement about relative standing — precisely the kind of comparison a raw number alone can't give you. Follow-Up Challenge: Using pnorm(), convert respondent 3's z-score into an approximate percentile. What proportion of respondents would you expect to have less education than this person, assuming education were exactly normally distributed (a real caveat, given how bounded and non-continuous education actually is)? Example 2: Proportion Above/Below a Threshold The Research Question: What proportion of respondents would we expect to have more than 16 years of education (roughly, a bachelor's degree), based on a normal approximation? The Code: m <- mean(gss$EDUC, na.rm = TRUE) s <- sd(gss$EDUC, na.rm = TRUE) z_16 <- (16 - m) / s 1 - pnorm(z_16) The Output: [1] 0.21 The Interpretation: Under a normal approximation, roughly 21% of respondents would be expected to have more than 16 years of education. This is an estimate based on treating EDUC as normally distributed, which — as the histogram and Q-Q plot in Example 3 will make clear — is only an approximation, since real education data is bounded, somewhat lumpy at common milestones (12 years for a high school diploma, 16 for a bachelor's), and not a perfectly smooth bell curve. The normal approximation is still a useful quick estimate; it's just worth remembering it's an approximation, not a report of the exact empirical proportion (which you could get directly with mean(gss$EDUC > 16, na.rm = TRUE)). Follow-Up Challenge: Compute the actual (empirical) proportion of respondents with more than 16 years of education directly from the data, without any normal approximation. How close is it to the normal-based estimate above? What does the gap tell you about how well education fits a normal shape? Example 3: Assessing Normality of AGE The Research Question: Is respondent age reasonably well-approximated by a normal distribution? The Code: hist(gss$AGE, main = "Distribution of Age", xlab = "Age") qqnorm(gss$AGE) qqline(gss$AGE) shapiro.test(gss$AGE) The Output: A histogram showing a roughly bell-shaped but mildly right-skewed distribution; a Q-Q plot with points following the diagonal reasonably well in the middle but drifting away at the upper and lower tails; and a Shapiro-Wilk test result along the lines of: Shapiro-Wilk normality test data: gss$AGE W = 0.97, p-value < 0.001 The Interpretation: The p-value is tiny, which technically means the test rejects the null hypothesis of normality. But look at the histogram and Q-Q plot first: the deviation is real but modest — a mild right skew, not a dramatic departure. With a sample this large, the Shapiro-Wilk test has enough statistical power to flag even this modest, largely cosmetic deviation as "significant." This is exactly the caveat from earlier in the chapter in action: for most practical purposes (including several of the inferential techniques in Part IV, which tend to be fairly robust to mild non-normality with large samples), age here is "normal enough," and the visual evidence should carry more weight than the p-value alone. Follow-Up Challenge: Run the same three checks (histogram, Q-Q plot, Shapiro-Wilk) on INCOME instead of AGE. Given everything Chapters 7 through 9 have already told you about income's shape, what do you expect to find before you run the code? Were you right? References