Statistics for Sociologists/Chapter 14: Chi-Square Tests and Effet Sizes

Chapter 14: Chi-Square Tests and Effect Sizes Chapter 13 compared means. A huge share of the questions sociologists actually ask don't involve means at all — they involve categories. Is political party affiliation related to religious attendance? Is race associated with happiness category? These are questions about whether two categorical variables move together, and t-tests have nothing to say about them. This chapter introduces the tool built specifically for this situation, and it happens to be one of the more transparent, satisfying pieces of logic in the entire book. When to Use Chi-Square The chi-square test of independencePearson, K. (1900). On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Philosophical Magazine, Series 5, 50(302), 157–175. applies when you have two categorical variables and want to know whether they're related — whether knowing a respondent's category on one variable tells you anything about their likely category on the other. If sex and political party identification are truly independent, knowing someone's sex should tell you nothing about their party — the party breakdown among men should look statistically like the party breakdown among women. Chi-square tests exactly that proposition. The Logic of the Test The logic is genuinely transparent, more so than most inferential tests: build a table of observed frequencies (how many respondents actually fall into each combination of categories), compare it to the table of expected frequencies you'd predict if the two variables were completely independent, and measure how far apart the two tables are. Big gap → the variables are probably related. Small gap → the data look about like what independence would predict. The chi-square statistic itself formalizes "how far apart" by summing, across every cell in the table, the squared difference between observed and expected counts, divided by the expected count: (squaring, for the same reason it showed up in Chapter 9's variance formula: it keeps positive and negative deviations from canceling out). Degrees of freedom for this test equal (number of rows − 1) × (number of columns − 1) — the same Sudoku-grid logic from Chapter 9 applies here at the scale of a whole table: once you know the row totals, the column totals, and all but the last row and column of individual cells, every remaining cell is completely determined, with no freedom left to vary. A larger chi-square statistic relative to its degrees of freedom translates into a smaller p-value. One real assumption to check: the test's math starts to break down when expected cell counts are too small — a common guideline is that no more than 20% of cells should have an expected count under 5, and none should be under 1. With a GSS-sized sample this is rarely a problem for two variables with a modest number of categories each, but it becomes a real concern fast if you're cross-tabulating two variables with many categories, spreading a few thousand respondents thin across dozens of cells. When it happens, collapsing categories (Chapter 6's recoding skills, put to direct use here) is usually the fix. Cross-Tabulation in R Base R: tab <- table(gss$SEX, gss$PARTYID) tab chisq.test(tab) dplyr + janitor: library(dplyr) library(janitor) gss %>% tabyl(SEX, PARTYID) %>% adorn_percentages() %>% adorn_pct_formatting() chisq.test(table(gss$SEX, gss$PARTYID)) tabyl() plus the adorn_*() family produces a genuinely readable, publication-adjacent cross-tab with percentages baked in — a real step up from base R's bare counts when you're building a table for an actual write-up. Effect Size: Cramér's V Just as Chapter 13 paired every t-test with Cohen's d, every chi-square test in this book gets paired with Cramér's VCramér, H. (1946). Mathematical Methods of Statistics. Princeton University Press. — a measure, bounded between 0 and 1, of how strongly two categorical variables are associated, independent of sample size. A chi-square test can be statistically significant with an association too weak to matter (the same GSS-sample-size problem from Chapter 13, showing up again here); Cramér's V is how you find out whether it's worth mentioning outside a statistics class. library(effectsize) cramers_v(table(gss$SEX, gss$PARTYID)) Rough interpretation benchmarks (which, like Cohen's benchmarks in Chapter 13, depend somewhat on the number of categories involved and should be treated as guides, not laws): around 0.1 is a small association, around 0.3 is medium, and 0.5 or above is large. Visualizing Categorical Relationships library(ggplot2) ggplot(gss, aes(x = factor(PARTYID), fill = factor(SEX))) + geom_bar(position = "dodge") + labs(title = "Party Identification by Sex", x = "Party ID", y = "Count", fill = "Sex") A grouped bar chart (bars for each category of one variable, clustered and colored by the other) makes a categorical association visible at a glance in a way a table of percentages, however well-formatted, usually doesn't. Reporting Results Chi-square is deeply satisfying because the logic is so transparent: you are, quite literally, asking "how far are the observed counts from what we'd expect if there were no relationship at all?" A good sociological example makes this concrete — is political party affiliation associated with religious attendance? (Yes. Dramatically.) That's not exactly a shocking finding, and that's fine — sometimes a test just confirms what you already strongly suspected, and confirmation is still information. A complete write-up follows the same template Chapter 13 established, adapted for chi-square's specific statistics: A chi-square test of independence found a significant association between sex and political party identification, χ²(6, N = 2,020) = 14.3, p = .026, Cramér's V = 0.08. Although statistically significant, the association was weak by conventional benchmarks, suggesting sex accounts for only a small part of the variation in party identification in this sample. Chapter Summary The chi-square test of independence evaluates whether two categorical variables are related by comparing observed cell counts to the counts you'd expect under complete independence — a large gap between observed and expected suggests a real relationship. Degrees of freedom depend on the table's dimensions, and the test requires reasonably large expected cell counts to behave well, which can require collapsing sparse categories first. chisq.test() in base R and tabyl() from janitor (paired with dplyr) both build the necessary cross-tabs, and Cramér's V — required alongside every chi-square result in this book, exactly like Cohen's d for t-tests — tells you whether a statistically significant association is actually strong enough to matter. Grouped bar charts make categorical associations visually obvious in a way raw percentage tables rarely do. Practice Examples Example 1: Sex and Party Identification The Research Question: Is political party identification associated with sex in this sample, and how strong is that association? The Code (base R): tab <- table(gss$SEX, gss$PARTYID) chisq.test(tab) library(effectsize) cramers_v(tab) The Code (dplyr + janitor): library(dplyr) library(janitor) gss %>% tabyl(SEX, PARTYID) %>% adorn_percentages("row") %>% adorn_pct_formatting() The Output: Pearson's Chi-squared test data: tab X-squared = 14.3, df = 6, p-value = 0.0264 Cramer's V = 0.08 [95% CI: 0.02, 0.14] The Interpretation: The association is statistically significant (p = .026) but weak (V = 0.08) — a pattern that should feel familiar after Chapter 13's discussion of GSS-sized samples detecting small effects reliably. Sex tells you a little something about party identification in this sample, but not very much; most of the variation in party ID has nothing to do with sex at all. Follow-Up Challenge: Collapse PARTYID into three broader categories (Democrat, Independent, Republican) using Chapter 6's recoding skills, and rerun the test. Does collapsing categories change the substantive conclusion, the effect size, or both? Example 2: Race and Happiness The Research Question: Is race associated with self-reported happiness in this sample? The Code: tab2 <- table(gss$RACE, gss$HAPPY) chisq.test(tab2) library(effectsize) cramers_v(tab2) library(ggplot2) ggplot(gss, aes(x = factor(RACE), fill = factor(HAPPY))) + geom_bar(position = "fill") + labs(title = "Happiness by Race", x = "Race", y = "Proportion", fill = "Happiness") The Output: X-squared = 22.6, df = 4, p-value = 0.0002 Cramer's V = 0.11 [95% CI: 0.06, 0.16] The Interpretation: A statistically significant association (p < .001) with a small-to-moderate effect size (V = 0.11) — stronger evidence of a real relationship than Example 1, though still modest in size. The geom_bar(position = "fill") version of the plot is worth noting specifically: it converts each racial group's bar to 100% height and shows the proportion falling into each happiness category, which makes a compositional comparison across groups of different sizes much easier to read than raw counts would. Follow-Up Challenge: This association is significant and larger than Example 1's — but chi-square alone can't tell you anything about why. What third variables (income, discrimination, health, region) might plausibly explain part of this association? How would you begin to investigate that with the tools from this book so far? Example 3: Write-Up The Research Question: How should the Example 2 result be written up for a sociology paper? The Code: (No new code — translating Example 2's output into prose.) The Output: A chi-square test of independence found a significant association between race and self-reported happiness, chi-squared(4, N = 2,020) = 22.6, p < .001, Cramer's V = 0.11. The association, while statistically significant and somewhat stronger than a comparable test of sex and party identification, remained modest in magnitude, indicating that race alone accounts for a relatively small share of the variation in happiness in this sample. The Interpretation: This write-up does something worth flagging explicitly: it contextualizes the effect size by comparing it to another result from earlier in the chapter, rather than reporting the number in isolation. Readers rarely have a strong independent intuition for what "V = 0.11" means; giving them a comparison point (weaker than X, stronger than Y) often communicates more than the bare statistic. Follow-Up Challenge: Find a published sociology article that reports a chi-square test (a quick search for "chi-square" plus a topic you're interested in, in a database like Google Scholar, will turn up plenty). Does it report an effect size alongside the p-value? If not, what does that omission cost the reader? References