Statistics for Sociologists/Chapter 13: Hypothesis Testing, t-Tests, and Effect Sizes

Chapter 13: Hypothesis Testing, t-Tests, and Effect Sizes Every inferential technique for the rest of this book follows the same basic logic, and this chapter introduces that logic and its first real application at the same time, on purpose — hypothesis testing stays grounded when it's attached to an actual question about actual people, instead of floating free as an abstract procedure you memorize and then forget. The question here: do two groups actually differ, or does it just look that way in this particular sample? The Logic of Hypothesis Testing Every hypothesis test starts with two competing claims. The null hypothesis (H₀) is the boring one: no difference, no relationship, nothing going on. The alternative hypothesis (H₁) is the interesting one: there is a difference, a relationship, something happening. The entire logic of the test is built around the null hypothesis as a kind of straw man — you assume, for the sake of argument, that nothing is going on, and then ask whether your actual data are surprising under that assumption. If the data would be quite unlikely if the null were really true, that's evidence against the null. If the data look perfectly ordinary under the null, you have no particular reason to abandon it. This setup creates two distinct ways to be wrong. A Type I error (false positive) means rejecting a true null hypothesis — concluding something is going on when it isn't. A Type II error (false negative) means failing to reject a false null hypothesis — missing a real effect that's actually there (this is precisely what low statistical power, from Chapter 11, makes more likely). Every hypothesis test involves a trade-off between these two error types, and no test eliminates either one entirely. The p-value is the probability of observing data this extreme (or more extreme) if the null hypothesis were actually true. Read that definition again, because it is not the definition most people carry around in their heads. A p-value of 0.04 does not mean there is a 4% chance you're wrong, or a 4% chance the null hypothesis is true. It means that if the null hypothesis were true, you'd see data this extreme (or more so) about 4% of the time. These are not the same claim. One of them is useful. Choosing alpha (α) — the threshold below which you'll call a result "statistically significant" — is a decision, not a law of nature. The conventional α = 0.05 is exactly that: a convention, inherited largely from mid-20th-century agricultural statistics, not a principled cutoff between "real" and "not real." Some fields use stricter thresholds; nothing stops you from justifying a different one, provided you say so upfront rather than after seeing your results. Finally: nearly every test in this book is two-tailed — testing for a difference in either direction, rather than committing in advance to only one direction. One-tailed tests exist and are occasionally defensible, but they're easy to misuse (a one-tailed test that happens to point the "right" way after the fact is a classic, quiet form of p-hacking), so this book defaults to two-tailed testing throughout, and you should have a strong, pre-specified reason before doing otherwise. t-Tests A t-testStudent [Gosset, W. S.] (1908). The probable error of a mean. Biometrika, 6(1), 1–25. compares means, and comes in a few varieties depending on your research design: One-sample t-test — is this sample's mean different from a specific known value? Independent samples t-test — do two separate, unrelated groups differ in their means? Paired samples t-test — do the same respondents differ across two related measurements (before/after, spouse/spouse)? This book mentions it here for completeness but doesn't dwell on it, since GSS's cross-sectional design rarely produces natural pairs. For a one-sample t-test, the test statistic has a familiar shape — it's a z-score (Chapter 10) built from a sample mean instead of a single raw value, using the standard error (Chapter 11) instead of the population standard deviation: where μ₀ is the value you're testing against (the scale midpoint, a known population value, whatever your null hypothesis specifies). The independent-samples version follows the same basic logic — a difference between two group means, divided by the standard error of that difference — with a somewhat messier formula this book won't reproduce in full, since R computes it for you and the conceptual picture is what matters. Every t-test result also reports degrees of freedom. A one-sample t-test has df = n − 1, for exactly the reason Chapter 9's Sudoku explanation laid out: calculating the sample mean spends one degree of freedom, leaving n − 1 values genuinely free to vary. An independent-samples t-test spends one degree of freedom per group, for roughly df = n₁ + n₂ − 2 (Welch's version, which this book uses by default, adjusts this further to account for unequal variances, which is why you'll often see a non-integer df like 2017.3 in R's output rather than a clean whole number). t-tests carry assumptions — approximate normality (fairly robust to violation with large samples, per the Central Limit Theorem from Chapter 11), equal variances between groups (for the independent-samples version), and independence of observations. That second assumption, equal variances, is handled automatically in R by default: Welch's t-testWelch, B. L. (1947). The generalization of "Student's" problem when several different population variances are involved. Biometrika, 34(1/2), 28–35., which does not assume equal variances between groups, is R's default (var.equal = FALSE), and this book follows that default throughout rather than the classic Student's t-test, which does assume equal variances and is consequently a bit more fragile. # One-sample t-test: is mean HAPPY different from the scale midpoint (2)? t.test(gss$HAPPY, mu = 2) # Independent-samples t-test: does EDUC differ by SEX? (Welch's by default) t.test(EDUC ~ SEX, data = gss) # Explicit about Welch's vs. Student's, for clarity: t.test(EDUC ~ SEX, data = gss, var.equal = FALSE) # Welch's (default, and this book's choice) t.test(EDUC ~ SEX, data = gss, var.equal = TRUE) # Student's (equal-variance assumption) dplyr, for the group descriptives you'll want alongside any t-test: library(dplyr) gss %>% group_by(SEX) %>% summarize( mean_educ = mean(EDUC, na.rm = TRUE), sd_educ = sd(EDUC, na.rm = TRUE), n = sum(!is.na(EDUC)) ) Effect Size: Cohen's d Here is the point this book has been building toward since Chapter 1. With a GSS-sized sample (n > 1,500, per Chapter 11's power discussion), almost any real difference — even a genuinely tiny one — will turn out "statistically significant." A p-value under 0.05 in a sample this large tells you astonishingly little on its own about whether the difference is worth caring about. You need a second number, answering a different question: "Is the difference real?" is the p-value's job. "Does the difference matter?" is the effect size's job. You need both, and this book has insisted on both since Chapter 1 for exactly this reason. Cohen's d expresses the size of a difference between two means in standard-deviation units — conceptually a close cousin of the z-score from Chapter 10, but describing the gap between groups rather than one value's distance from a mean: Conventional benchmarks, due to Jacob Cohen,Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum Associates. are small (d ≈ 0.2), medium (d ≈ 0.5), and large (d ≈ 0.8) — rough guides meant to aid interpretation, not laws of nature. A d of 0.5 in a study of educational interventions might be a genuinely large, policy-relevant effect; the same d of 0.5 in a study of something with enormous real-world variability might be comparatively unremarkable. Context, not the raw number alone, determines whether an effect size is "big." library(effectsize) cohens_d(EDUC ~ SEX, data = gss) The effectsize packageBen-Shachar, M. S., Lüdecke, D., & Makowski, D. (2020). effectsize: Estimation of effect size indices and standardized parameters. Journal of Open Source Software, 5(56), 2815. handles the computation, along with confidence intervals for the effect size itself, and will reappear for every effect size in the rest of this book. Reporting Results A complete t-test write-up, in the style expected in a sociology paper, includes the test statistic, degrees of freedom, p-value, and effect size — never the p-value alone: An independent-samples t-test (Welch's) found that respondents' years of education did not differ meaningfully by sex, t(2017.3) = 1.42, p = .156, d = 0.06. Although the difference was not statistically significant, the effect size was also negligible by conventional benchmarks, suggesting little practical difference in educational attainment between men and women in this sample. Notice that this write-up reports a null result honestly and completely — no p-value fishing, no quietly reframing the question until something crosses the threshold. A negative result, reported with its effect size, is a genuine contribution; that's precisely the professional habit Chapter 1's replication-crisis discussion was arguing for. Open Science Corner #355C7D #FFFFFF #F8B195 Pre-register your hypothesis before you look at the results. Deciding what you predicted after you've already seen the data isn't hypothesis testing — it's storytelling with extra steps, and it's exactly the kind of "researcher degrees of freedom" that inflates false-positive rates across an entire literature (a full taxonomy of this problem is waiting for you in Chapter 18). Posting a short, timestamped pre-registration to OSF, stating your hypothesis and planned analysis before running it, costs you ten minutes and buys your later results real credibility. Chapter Summary Hypothesis testing works by assuming a boring null hypothesis (nothing going on) and asking whether the observed data would be surprising under that assumption; a small p-value means the data would be unlikely if the null were true, not that the null is unlikely to be true — a distinction worth over-learning. Type I and Type II errors represent the two ways a test can go wrong, and α = 0.05 is a convention, not a natural law. t-tests compare means across one sample against a known value, two independent groups, or (less commonly for GSS data) paired measurements, and this book follows R's default of Welch's t-test, which doesn't assume equal variances between groups. With a sample as large as the GSS's, statistical significance is nearly guaranteed for any real difference, however small — which is why Cohen's d, reported alongside every p-value in this book, answers the question a p-value structurally cannot: does the difference actually matter? A complete result write-up always reports both, honestly, whether or not the result turned out to be significant. Practice Examples Example 1: Education by Sex — Independent Samples t-Test The Research Question: Do men and women differ in years of education, and if so, by how much does it matter? The Code (base R): t.test(EDUC ~ SEX, data = gss) library(effectsize) cohens_d(EDUC ~ SEX, data = gss) The Code (dplyr, for descriptives first): library(dplyr) gss %>% group_by(SEX) %>% summarize(mean_educ = mean(EDUC, na.rm = TRUE), sd_educ = sd(EDUC, na.rm = TRUE), n = n()) The Output: Welch Two Sample t-test data: EDUC by SEX t = 1.42, df = 2017.3, p-value = 0.156 95 percent confidence interval: -0.05 0.31 sample estimates: mean in group 1 mean in group 2 13.75 13.62 Cohen's d = 0.06 [95% CI: -0.02, 0.15] The Interpretation: The p-value (0.156) is above the conventional 0.05 threshold, so this test does not find a statistically significant difference — and the effect size (d = 0.06) confirms there's not much to find in the first place: this is a negligible effect by any conventional benchmark. Both pieces of evidence point the same direction here, which won't always be true (a significant result with a tiny effect size is entirely possible with a large sample, and is exactly the pattern this chapter has been warning about). Follow-Up Challenge: Run the same comparison on WORDSUM instead of EDUC. Is there a meaningful sex difference in vocabulary test performance, both statistically and practically? Example 2: Happiness vs. the Scale Midpoint — One-Sample t-Test The Research Question: Is average happiness in the sample different from the scale's neutral midpoint? The Code: t.test(gss$HAPPY, mu = 2) The Output: One Sample t-test data: gss$HAPPY t = -8.9, df = 1999, p-value < 2.2e-16 mean of x: 1.85 The Interpretation: The sample mean (1.85, on a scale where lower values indicate greater happiness) is significantly different from the scale's midpoint of 2, and the enormous t-statistic combined with the very small p-value reflects both a real pattern and the sample's large size. Because HAPPY's scale runs only 1–3, a raw mean difference of 0.15 is actually a fairly substantial shift relative to the scale's total range — worth confirming with an effect size (Cohen's d comparing the sample mean to the fixed value 2) rather than trusting the dramatic-looking t-statistic alone. Follow-Up Challenge: Compute a one-sample Cohen's d for this comparison (the formula is the mean difference divided by the sample standard deviation). Does the effect size support the same "this matters" conclusion the p-value seems to suggest, or does it complicate the story? Example 3: Full Write-Up The Research Question: How would the Example 1 result be written up for a sociology paper? The Code: (No new code — this example is about translating Example 1's output into prose.) The Output: A publication-style sentence, not R output: An independent-samples t-test (Welch's) found that respondents' years of education did not differ meaningfully by sex, t(2017.3) = 1.42, p = .156, d = 0.06 [95% CI: -0.02, 0.15]. The effect was negligible by conventional benchmarks, suggesting no practically meaningful difference in educational attainment between men and women in this sample. The Interpretation: Notice everything this write-up includes that a bare "p = .156, not significant" would leave out: the test statistic, degrees of freedom, effect size with its own confidence interval, and an explicit statement about practical (not just statistical) significance. This is the template this book will reuse for every inferential test from here forward — chi-square in Chapter 14, ANOVA in Chapter 15, regression in Chapter 16 — because the underlying professional habit doesn't change, only the specific numbers being reported. Follow-Up Challenge: Write your own full report for Example 2's one-sample t-test on HAPPY, following this same template exactly. References