Statistics for Sociologists/Chapter 9: Measures of Variability

Chapter 9: Measures of Variability Two distributions can have exactly the same mean and look nothing alike. Imagine one classroom where every student scored between 78 and 82 on an exam, and another where scores ranged from 40 to 100 — both could easily average out to 80. Reporting only the mean for each classroom would tell you they're identical. They are not. One is a room full of students performing consistently; the other is a room full of students having wildly different experiences with the material. Central tendency alone cannot tell these two stories apart. This chapter teaches the tools that can. Why Variability Matters If Chapter 8 answered "what's typical?", this chapter answers the equally important follow-up: "how much do people actually differ from that typical value?" A measure of central tendency without a measure of variability is half a description — technically not wrong, but genuinely incomplete in a way that can hide the most sociologically interesting part of the story. Two neighborhoods with the same average income can have very different levels of inequality. Two demographic groups with the same average happiness score can differ enormously in how consistently happy their members are. Range The range — the difference between the maximum and minimum values — is the simplest possible measure of spread, and its simplicity is also its weakness: it depends entirely on two values, and says nothing about how the other however-many-thousand respondents are distributed in between. One unusual outlier can make the range enormous even if everyone else in the dataset is clustered tightly together. range(gss$AGE, na.rm = TRUE) diff(range(gss$AGE, na.rm = TRUE)) # the range as a single number Interquartile Range (IQR) The interquartile range fixes the range's biggest weakness by ignoring the extremes entirely: it's the range of just the middle 50% of the data, from the 25th percentile (first quartile) to the 75th percentile (third quartile). Because it's built around the median's position in the ordered data rather than the literal minimum and maximum, it's considerably more robust to outliers than the range is. IQR(gss$EDUC, na.rm = TRUE) quantile(gss$EDUC, probs = c(0.25, 0.75), na.rm = TRUE) Variance Variance measures the average squared distance of each value from the mean. Why squared, rather than just averaging the plain distances? Because distances above the mean are positive and distances below it are negative, and if you don't square them first, they cancel each other out almost perfectly — you'd get an average deviation suspiciously close to zero for every dataset, which tells you nothing. Squaring makes every deviation positive before averaging, so spread in either direction contributes to a larger number rather than canceling out. For a population, variance is defined as: For a sample — which is what you'll actually compute nearly every time in this book — it's defined very slightly differently: That n − 1 in the denominator, instead of a plain n, is not a typo, and it's important enough that it gets its own explanation two sections from now, once standard deviation is on the table too. var(gss$EDUC, na.rm = TRUE) The cost of this fix is that variance ends up in squared units — "years of education, squared" is not a unit any human has ever needed outside of a statistics class, and it makes variance nearly impossible to interpret directly. Standard Deviation The standard deviation is variance's more interpretable cousin: take the square root of the variance, and you're back in the original units. Variance is in squared units, which are a unit no human has ever used naturally; standard deviation corrects this by taking the square root, which is the statistical equivalent of apologizing. sd(gss$EDUC, na.rm = TRUE) A standard deviation of, say, 3.15 years for EDUC means that a "typical" deviation from the mean education level (in the 2010 GSS, the mean level of education is 13.46 years, or about 1.5 years of college beyond high school) is about 3.15 years — some respondents deviate more, some less, but 3.15 years captures roughly how spread out the data are, in a unit (years) that actually means something to a human reader. Degrees of Freedom: What Sudoku Can Teach You About Statistics That n − 1 from the variance and standard deviation formulas above is your first encounter with a concept called degrees of freedom (often abbreviated df), and it is going to follow you for the rest of this book — every t-test, every chi-square test, every ANOVA, and every regression model you run from here forward will report a degrees-of-freedom value in its output. Most introductory textbooks introduce the term, attach a formula to it, and move on without ever really explaining what it means. This book is going to take one detour to actually explain it, using a tool that has nothing to do with statistics on its surface: Sudoku. If you've never played, the rules are simple: fill a 9×9 grid with the digits 1 through 9, such that every row, every column, and every 3×3 box contains each digit exactly once. alt=A sudoku puzzle that has lots of degrees of freedom|center|thumb|800x800px|Sudoku illustrates degrees of freedom very clearly. The highlighted row has 7 degrees of freedom. The highlighted column has 6 degrees of freedom. And the highlighted box has 8 degrees of freedom. Here's the part that matters for our purposes: imagine a single row, and imagine eight of its nine cells are already correctly filled in. Is the ninth cell a free choice? No. It's completely determined — there's exactly one digit missing from 1 through 9, and it has to go in that last cell. That cell has zero freedom left. It looks like a blank waiting to be filled, but the moment the other eight cells were set, its value was already decided. alt=A more complete Sudoku puzzle.|center|thumb|802x802px|This Sudoku puzzle has more squares completed. The highlighted row has just one missing value. With just one missing value, the number is predetermined. Can you solve that row? Degrees of freedom, in statistics, means exactly this: the number of values in a calculation that are still genuinely free to vary, once you've imposed some constraint on them. And the constraint, in the case of sample variance, is one you've already used without necessarily noticing it: before you can compute how far each value deviates from the mean, you have to calculate the mean first — and by definition, all of the deviations from that mean have to sum to exactly zero (the points above the mean and the points below it balance out perfectly, every time, for any set of numbers and their own mean). That's a real mathematical constraint, exactly like a Sudoku row that has to contain each digit exactly once. So: if you have n data points and you've already used them to calculate the mean, only n − 1 of those points are still free to be anything at all. The last one is locked in — the same way the last cell in that Sudoku row was locked in — because it has to be whatever value makes all the deviations sum to zero. You've spent one degree of freedom calculating the mean, and you have n − 1 left over to spend on estimating the spread around it. That's why sample variance and sample standard deviation divide by n − 1 rather than n — dividing by the number of values that were actually still free to vary, rather than the total count, produces a better (specifically, unbiased) estimate of the population value. This is a genuinely useful mental model to carry forward, because "degrees of freedom" keeps reappearing throughout the rest of this book with the exact same underlying logic, just applied to different constraints: A one-sample t-test (Chapter 13) has df = n − 1 — one degree of freedom spent estimating the sample mean, exactly as above. An independent-samples t-test has roughly df = n₁ + n₂ − 2 — one degree spent estimating each group's own mean. A chi-square test of independence (Chapter 14) has df = (rows − 1) × (columns − 1) — think of a cross-tabulation as a much bigger Sudoku grid, where the "clues" are the row and column totals rather than the digits 1 through 9; once you know all the row and column totals and all but the last row and column of interior cells, every remaining cell is fully determined by those totals, exactly like that last Sudoku cell. ANOVA (Chapter 15) has separate degrees of freedom for its between-groups and within-groups variance components, following the same logic: one degree of freedom spent per group mean being estimated. If you've ever filled in the last cell of a Sudoku row without feeling like you were making a real decision, you have already, whether you realized it or not, personally experienced the loss of a degree of freedom. dplyr, computing several spread statistics at once: library(dplyr) gss %>% summarize( sd_educ = sd(EDUC, na.rm = TRUE), var_educ = var(EDUC, na.rm = TRUE), iqr_educ = IQR(EDUC, na.rm = TRUE) ) Box Plots and the Five-Number Summary A box plotTukey, J. W. (1977). Exploratory Data Analysis. Addison-Wesley. visualizes the five-number summary — minimum, first quartile, median, third quartile, and maximum — in a single compact shape: a box spanning the IQR, a line at the median, and "whiskers" extending to the most extreme non-outlier values. summary(gss$AGE) # the five-number summary as text Base R: boxplot(gss$AGE ~ gss$SEX, main = "Age by Sex", xlab = "Sex", ylab = "Age") ggplot2: library(ggplot2) ggplot(gss, aes(x = factor(SEX), y = AGE)) + geom_boxplot() + labs(title = "Age by Sex", x = "Sex", y = "Age") Comparing variability across groups — not just comparing means across groups — is where this chapter gets genuinely sociological. Two groups can have nearly identical average happiness while one group is far more internally consistent than the other; that difference in spread can be as substantively interesting as a difference in the average itself. Income variability by race, happiness variability by marital status, education variability by region: these are real questions, and a side-by-side box plot answers them at a glance in a way that two separate means never could. alt=a box plot comparing age by sex|center|thumb|800x800px|This boxplot compares age by sex of respondent in the 2010 GSS. Outliers and the 1.5 × IQR Rule A common, defensible rule of thumb for flagging outliers: any value more than 1.5 times the IQR below the first quartile or above the third quartile. Q1 <- quantile(gss$CONRINC, 0.25, na.rm = TRUE) Q3 <- quantile(gss$CONRINC, 0.75, na.rm = TRUE) iqr_income <- IQR(gss$CONRINC, na.rm = TRUE) lower_bound <- Q1 - 1.5 * iqr_income upper_bound <- Q3 + 1.5 * iqr_income outliers <- gss$CONRINC[gss$CONRINC < lower_bound | gss$CONRINC > upper_bound] outliers Flagging a value as a statistical outlier is not the same thing as deciding it's an error. Sometimes an "outlier" is a data entry mistake worth investigating and correcting. Sometimes it's just Jeff Bezos, again, being exactly as real and exactly as high-earning as the data claims. The 1.5 × IQR rule tells you where to look; it doesn't tell you what to do once you're looking, and that judgment call belongs to you, informed by domain knowledge the formula doesn't have. Chapter Summary Central tendency alone can hide dramatically different distributions behind an identical mean — variability measures fill that gap. The range is simple but fragile, driven entirely by two extreme values; the IQR fixes this by focusing on the middle 50% of the data, making it robust to outliers in the same way the median is. Variance captures average squared deviation from the mean but lives in unhelpful squared units; standard deviation takes the square root and returns to the data's original, interpretable units. Sample variance and SD divide by n − 1 rather than n because calculating the mean first spends one degree of freedom — a concept this chapter explained via Sudoku, and one that will keep reappearing, in different forms, through every remaining inferential chapter in this book. R computes the sample version of variance and SD by default, which is almost always what social science research needs. Box plots visualize the five-number summary and are especially useful for comparing variability — not just central tendency — across sociologically meaningful groups. The 1.5 × IQR rule offers a systematic way to flag potential outliers, though flagging is not the same as deciding they're errors. Practice Examples Example 1: Standard Deviation of EDUC The Research Question: How much do respondents' years of education typically vary from the mean, and what does "one SD above the mean" actually mean for a real person? The Code: mean(gss$EDUC, na.rm = TRUE) sd(gss$EDUC, na.rm = TRUE) The Output: [1] 13.46 # mean [1] 3.15 # sd The Interpretation: The mean is 13.46 years; the standard deviation is roughly 3.15 years. A respondent "one SD above the mean" has about 13.46 + 3.15 ≈ 16.6 years of education — very roughly, a bachelor's degree plus a bit. This is a concrete, human-scale way of using a standard deviation: not as an abstract number, but as a translation between a statistic and an actual respondent you could imagine meeting. Follow-Up Challenge: Find a respondent (or hypothetical respondent) at two SDs below the mean. What does that level of education correspond to in real terms? Does the answer feel like a "typical" amount of variability to you, or more/less than you expected? Example 2: Box Plot of AGE by SEX The Research Question: Does age vary more within one sex than the other in this sample? The Code (base R): boxplot(gss$AGE ~ gss$SEX, main = "Age by Sex", xlab = "Sex", ylab = "Age") The Code (ggplot2): library(ggplot2) ggplot(gss, aes(x = factor(SEX), y = AGE)) + geom_boxplot(fill = "lightblue") + labs(title = "Age by Sex", x = "Sex", y = "Age") The Output: Two box plots side by side, one per sex category, each showing the five-number summary of age within that group. (See the FIGURE note above — actual box positions and spread depend on real 2010 GSS data.) The Interpretation: Comparing box heights (not just median lines) tells you whether one group is more internally variable in age than the other. If the boxes are roughly the same height with similar median lines, age is distributed similarly across sexes in this sample; a taller box for one group would indicate more internal age variability within that group specifically — a genuinely different claim than "the two groups have different average ages," and one a box plot reveals far more directly than two separate means would. Follow-Up Challenge: Add a third grouping variable — say, MARITAL — and produce a box plot of age by both sex and marital status simultaneously. (Hint: in ggplot2, this is a job for facet_wrap(), which you haven't formally met yet — see if you can find it in the documentation and get it working.) Example 3: Outliers in CONRINC The Research Question: Are there respondents whose income is unusually extreme relative to the rest of the sample, by the 1.5 × IQR standard? The Code: Q1 <- quantile(gss$CONRINC, 0.25, na.rm = TRUE) Q3 <- quantile(gss$CONRINC, 0.75, na.rm = TRUE) iqr_income <- IQR(gss$CONRINC, na.rm = TRUE) upper_bound <- Q3 + 1.5 * iqr_income sum(gss$CONRINC > upper_bound, na.rm = TRUE) The Output: [1] 36 The Interpretation: 36 respondents fall above the upper outlier threshold — a meaningful chunk of the sample, which is fairly typical for a right-skewed variable like income (recall Chapter 7). This is a useful reminder that "outlier," for a skewed variable, doesn't necessarily mean "rare data error." It often means "a real, if less common, part of a genuinely lopsided distribution" — exactly the population Chapter 8's mean-versus-median discussion was built around. Follow-Up Challenge: Rerun this same outlier check on EDUC instead of CONRINC. Do you find as many outliers? What does the difference tell you about how symmetric or skewed each variable is? For a third comparison, try WORDSUM — with a true zero and a hard ceiling of 10, would you even expect the 1.5×IQR rule to flag many outliers here? References