Statistics for Sociologists/Chapter 15: ANOVA and Effect Sizes

Chapter 15: ANOVA and Effect Sizes Chapter 13's t-test compares two groups. Real sociological variables rarely stop at two categories — marital status, religious attendance, region, education level all naturally split respondents into three, four, or more groups. This chapter extends the t-test's logic to exactly that situation, and introduces a new way of thinking about variance that will turn out to be the conceptual foundation for regression in Chapter 16 as well. Why Not Just Run Multiple t-Tests? If you have four marital status categories and want to compare education across all of them, why not just run every possible pairwise t-test — married vs. never married, married vs. divorced, and so on? Because doing so inflates your Type I error rate well beyond the α you think you're using. Each individual t-test carries a 5% false-positive risk on its own; run six pairwise comparisons across four groups, and the probability that at least one of them turns up "significant" by pure chance climbs well above 5% — closer to 25% or more, depending on how many comparisons you run. This is the multiple comparisons problem, and it's a real, well-documented contributor to spurious findings in the published literature (recall Chapter 1's replication-crisis discussion, and Chapter 18's fuller treatment of p-hacking to come). ANOVA solves this by asking one question across all groups simultaneously, controlling the overall error rate properly, rather than asking many separate questions and hoping none of them get lucky. One-Way ANOVA Logic Analysis of Variance (ANOVA)Fisher, R. A. (1925). Statistical Methods for Research Workers. Oliver and Boyd. is, despite the name, a test about means — it just gets there by partitioning variance into two components: variance between groups (how much the group means differ from the overall mean) and variance within groups (how much individual respondents vary around their own group's mean). ANOVA is just a fancy way of asking: is the spread between groups bigger than the spread within groups? If the groups vary more from each other than people vary within groups, something is probably going on. If people within each group are all over the place anyway, a difference between group means starts looking a lot less impressive by comparison. The F-statistic formalizes this directly as a ratio: between-group variance divided by within-group variance. Each MS ("mean square") is just a sum of squared deviations (SS, the same building block behind variance since Chapter 9) divided by its own degrees of freedom — dfbetween = k − 1 (one degree of freedom spent per group mean, where k is the number of groups) and dfwithin = N − k (one degree of freedom spent per group, subtracted from the total sample size), the same Sudoku-style logic from Chapter 9 showing up again at the level of whole groups instead of individual values. A large F means the between-group differences are large relative to the ordinary noise within each group; an F close to 1 means the groups aren't really separating from each other any more than random chance within a single group would produce. ANOVA's assumptions echo the t-test's from Chapter 13: approximate normality within each group (again, fairly robust with large samples), homogeneity of variance across groups (each group should show roughly similar spread — a group-level version of the equal-variance assumption Welch's t-test relaxed for two groups), and independence of observations. Running ANOVA in R model <- aov(EDUC ~ factor(MARITAL), data = gss) summary(model) dplyr, for group descriptives before the test: library(dplyr) gss %>% group_by(MARITAL) %>% summarize(mean_educ = mean(EDUC, na.rm = TRUE), sd_educ = sd(EDUC, na.rm = TRUE), n = n()) Note the factor() wrapped around MARITAL inside the formula — aov() needs to know this is a categorical grouping variable, not a number to treat arithmetically, which should feel like familiar territory after Chapter 4's entire discussion of exactly this distinction. Post-Hoc Tests: Which Groups Actually Differ? A significant ANOVA result tells you that the groups differ somewhere — it does not tell you which specific groups differ from which others. That's the job of a post-hoc test, run only after a significant overall ANOVA result, and Tukey's HSD ("Honestly Significant Difference")Tukey, J. W. (1949). Comparing individual means in the analysis of variance. Biometrics, 5(2), 99–114. is the standard choice — a name that sounds less like a statistical method and more like something you'd say to reassure a nervous coauthor. It compares every pair of groups while still properly controlling the overall Type I error rate, solving the exact problem that made running separate t-tests a bad idea in the first place. TukeyHSD(model) The output gives you a confidence interval and adjusted p-value for every pairwise group comparison — married vs. divorced, divorced vs. never married, and so on — letting you say specifically where the differences live rather than just that they exist somewhere in the table. Effect Size: Eta-Squared Eta-squared (η²) is ANOVA's effect size, expressing the proportion of total variance in the outcome that's explained by group membership — conceptually a close relative of R² from the regression chapter just ahead. library(effectsize) eta_squared(model) Rough benchmarks: around 0.01 is a small effect, around 0.06 is medium, and 0.14 or above is large — again, guides rather than laws, and worth interpreting in context rather than mechanically. Visualizing Group Differences library(ggplot2) ggplot(gss, aes(x = factor(MARITAL), y = EDUC)) + geom_boxplot() + labs(title = "Education by Marital Status", x = "Marital Status", y = "Years of Education") Box plots (Chapter 9) remain the right tool here, exactly as they were for the two-group comparisons in earlier chapters — they show both central tendency and spread across every group at once, which is precisely the between-versus-within comparison ANOVA is built around. Reporting Results A one-way ANOVA found a significant effect of marital status on years of education, F(3, 2016) = 8.42, p < .001, η² = 0.012. Tukey's HSD post-hoc comparisons indicated that never-married respondents (M = 14.1) had significantly more education than divorced respondents (M = 13.2, p = .003), though the overall effect size was small, suggesting marital status accounts for only a modest share of the variation in educational attainment. Notice, again, the discipline of reporting a small effect size honestly even when the p-value is comfortably significant — the same habit this book has insisted on since Chapter 1, now showing up for the third inferential test in a row. Chapter Summary ANOVA extends the t-test's logic to three or more groups by comparing between-group variance to within-group variance via the F-statistic, avoiding the inflated false-positive risk that running many separate pairwise t-tests would create. A significant ANOVA result establishes that group means differ somewhere, but identifying specifically which groups differ requires a post-hoc test like Tukey's HSD, which controls the overall error rate properly across all pairwise comparisons. Eta-squared, ANOVA's effect size, expresses the proportion of variance explained by group membership and should accompany every ANOVA result in this book, exactly as Cohen's d and Cramér's V have accompanied t-tests and chi-square tests. Box plots remain the natural visualization for comparing several groups' distributions at once. The core idea here — partitioning variance into explained and unexplained components — isn't just an ANOVA-specific trick; it's the same logic Chapter 16's regression is built on. Practice Examples Example 1: Education by Marital Status The Research Question: Does average education differ across marital status categories, and if so, which specific groups differ? The Code: model <- aov(EDUC ~ factor(MARITAL), data = gss) summary(model) TukeyHSD(model) library(effectsize) eta_squared(model) The Output: Df Sum Sq Mean Sq F value Pr(>F) factor(MARITAL) 3 203 67.7 8.42 1.2e-05 Residuals 2016 16200 8.0 Eta2 = 0.012 [95% CI: 0.005, 1.00] Tukey multiple comparisons of means diff lwr upr p adj Never married-Divorced 0.90 0.24 1.56 0.003 ... The Interpretation: The overall ANOVA is significant (p < .001), and Tukey's HSD narrows that down: never-married respondents have significantly more education than divorced respondents specifically, with a 95% CI for that gap ([0.24, 1.56] years) that excludes zero. The effect size (η² = 0.012) is small by conventional benchmarks — marital status explains about 1.2% of the total variance in education, meaningfully real but far from the dominant factor. Follow-Up Challenge: Look at the full Tukey output (not just the one comparison shown above). How many of the six pairwise comparisons turn out significant? Does the overall pattern make more sense once you see all of them together, rather than just the ANOVA's single p-value? Example 2: Happiness by Religious Attendance The Research Question: Does happiness differ across categories of religious attendance frequency? The Code: model2 <- aov(HAPPY ~ factor(ATTEND), data = gss) summary(model2) TukeyHSD(model2) library(effectsize) eta_squared(model2) library(ggplot2) ggplot(gss, aes(x = factor(ATTEND), y = HAPPY)) + geom_boxplot() + labs(title = "Happiness by Religious Attendance", x = "Attendance Category", y = "Happiness") The Output: Df Sum Sq Mean Sq F value Pr(>F) factor(ATTEND) 8 12.4 1.55 6.90 3.1e-09 Residuals 2011 452.1 0.22 Eta2 = 0.027 The Interpretation: A significant effect of religious attendance on happiness (p < .001), with a small-to-modest effect size (η² = 0.027) — larger than the marital status effect in Example 1, but still leaving the overwhelming majority of variation in happiness explained by factors other than religious attendance. This is a genuinely common pattern in social-science ANOVA results: statistically real, substantively modest, worth reporting honestly as both. Follow-Up Challenge: Does the pattern in the Tukey output look linear (happiness increasing steadily with more attendance) or is it more irregular? What would a linear pattern versus an irregular one each suggest about the underlying relationship? Example 3: Full Write-Up The Research Question: How should the Example 1 result be written up for a sociology paper, including the post-hoc findings? The Code: (No new code — translating Example 1's output into prose.) The Output: A one-way ANOVA found a significant effect of marital status on years of education, F(3, 2016) = 8.42, p < .001, eta-squared = 0.012. Tukey's HSD post-hoc comparisons indicated that never-married respondents had significantly more education than divorced respondents (95% CI [0.24, 1.56], p = .003). Despite reaching statistical significance, the overall effect size was small, indicating that marital status accounts for a modest share of the variance in educational attainment in this sample. The Interpretation: This write-up follows the same three-part discipline established in Chapters 13 and 14: report the overall test statistic, report the effect size, and be explicit about what the effect size implies for practical importance — never letting the p-value stand alone as if it were the whole story. Follow-Up Challenge: Write a comparable full write-up for Example 2's ATTEND/HAPPY result, including at least one specific pairwise comparison from its Tukey output. References