Statistics for Sociologists/Chapter 18: Putting It All Together - Replicating Sociological Research

Chapter 18: Putting It All Together — Replicating Sociological Research Seventeen chapters ago, this book opened with an uncomfortable fact: a large share of published findings across the social and behavioral sciences — sociology included, by name, not just psychology's problem to solve — don't hold up when someone else checks them. Every skill since then has been building toward the ability to do the checking yourself. This capstone chapter asks you to actually do it: find a real, published, GSS-based sociological finding, and see whether it holds up under your own hands. This is not busywork tacked onto the end of a syllabus. It is, quite possibly, the single most professionally useful thing this book has taught you. Why Replication Matters, Revisited Chapter 1 introduced the replication crisis through the 2015 Open Science Collaboration project (psychology)Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. and the 2026 Tyner et al. study (business, economics, education, political science, psychology, and sociology, replicating at 51.1% — statistically indistinguishable from psychology's 49.0%).Tyner, A. H. et al. (2026). Investigating the replicability of the social and behavioural sciences. Nature, 652, 143–150. Both numbers are worth having in your head again right now, at the end of this book, because this chapter is where that abstract statistic becomes a concrete, personal exercise: somewhere between one in three and one in two published quantitative findings in your own field, tested rigorously, fails to replicate. You are about to find out, firsthand, what that actually looks like from the inside. It's worth being precise about two related but distinct ideas this book has used somewhat loosely up to now: Replication — rerunning the same methods on the same data, checking whether you get the same result. This is a check on computational and analytical correctness: did the original researcher's code do what they said it did? Reproduction — rerunning the same methods on new data, checking whether the finding generalizes beyond the original sample. This is a check on whether the underlying claim about the world is actually true, not just whether the original arithmetic was correct. Both matter, and this chapter's capstone project will primarily involve the first — since you'll be working with the same GSS 2010 data the original researchers most likely used, or a closely related wave — while thinking carefully about what the second would require. Replication is a scientific norm, not an insult. Attempting to replicate someone's work is not an accusation of dishonesty; it's the ordinary, expected functioning of a healthy discipline. Treat every replication attempt in this chapter — including any that fail — in that spirit. p-Hacking and Researcher Degrees of Freedom Some researchers, when faced with a non-significant result, try a different analysis. Then another. Then another. This is called "exploring the data" by optimists and "p-hacking" by everyone else. P-hacking refers to any of a family of practices where a researcher tries multiple analytical approaches and reports only the one that produced a significant result — without disclosing the others that didn't. It's rarely outright fraud; more often it's an accumulation of small, individually reasonable-feeling decisions ("researcher degrees of freedom") that, in combination, dramatically inflate the real false-positive rate beyond the nominal α = 0.05 you think you're operating under.Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. A brief, non-exhaustive taxonomy — not to teach you to do these things, but so you recognize them, in your own work and in what you read: Optional stopping — collecting data, checking for significance, and continuing to collect more data only if the result isn't significant yet, stopping as soon as it is. HARKingKerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196–217. ("Hypothesizing After Results are Known") — presenting a hypothesis discovered by exploring the data as though it had been predicted all along. Selective outcome reporting — measuring several outcome variables, and reporting results only for the ones that turned out significant. Flexible outlier handling — trying several different rules for excluding outliers until one produces significance. Undisclosed covariate fishing — trying many different combinations of control variables in a regression until one combination yields a significant result on the variable of interest. Every one of these individually might feel like ordinary, defensible data analysis in the moment. Collectively, and left undisclosed, they're how a field ends up with a replication rate near 50% despite every individual researcher believing they did careful, honest work. The fix isn't paranoia about your own judgment — it's transparency: pre-registration (Chapter 13), reporting effect sizes honestly regardless of significance (every chapter since Chapter 1), and sharing your actual analysis script (every Open Science Corner in this book) so someone else can see exactly which choices you made and why. Finding Papers With Open Data Three practical starting points for locating a published, GSS-based finding to replicate: Google Scholar, searching a substantive topic you're interested in plus "General Social Survey" — this remains the single fastest way to find candidate papers. OSF (osf.io) and Dataverse (dataverse.org), both of which host shared data and code from published studies directly — a paper with materials posted to either is usually the easiest kind to attempt a genuine replication of, since the original analytical choices are visible rather than reconstructed from a methods section. Journal data availability statements — many journals now require authors to state where their data and code can be obtained; this statement, usually near a paper's methods section or in its supplementary materials, is worth checking before you assume you'll need to build an extract from scratch. The Replication Workflow Read the paper. What is the research question? Which GSS variables are used? What analyses were run? Take real notes — you'll need them at every subsequent step. Download the data. The GSS Data Explorer (Chapter 3) or your course-provided extract, matching as closely as possible the wave(s) and variables the original paper used. Clean and recode. Match the paper's sample restrictions and variable definitions as closely as the methods section allows (Chapters 5 and 6) — this step alone often surfaces ambiguities the original paper didn't fully specify. Run the analyses. Reproduce the paper's statistical tests using the tools from Chapters 8 through 17, as applicable. Compare. Did you get the same results? Same direction, same approximate magnitude, same statistical significance? Note discrepancies precisely rather than vaguely. Evaluate. Are the original conclusions actually supported by the analysis? Were effect sizes reported? Was the study adequately powered (Chapter 11)? Would you have reached the same substantive conclusion the authors did? A Worked Example: Replicating a GSS-Based Finding To make this concrete, here's a full walkthrough of a realistic (illustrative, not an actual published citation) replication exercise: a hypothetical paper claims that years of education significantly predicts income, and that this relationship remains substantial after accounting for age. Step 1–2: Read and download. The paper reports a bivariate regression (b ≈ $3,900 per year of education, R² ≈ .25) and states the analytic sample was GSS respondents aged 25 and older, to exclude respondents still likely to be enrolled in school. Step 3: Clean and recode, matching the paper's sample restriction. library(haven) library(dplyr) gss <- read_sav("gss2010.sav") gss_replication <- gss %>% filter(AGE >= 25) %>% filter(!is.na(CONRINC), !is.na(EDUC), !is.na(AGE)) nrow(gss_replication) Step 4: Run the analyses — the original bivariate model, plus the age-adjusted comparison the paper claims. model_simple <- lm(CONRINC ~ EDUC, data = gss_replication) summary(model_simple) model_adjusted <- lm(CONRINC ~ EDUC + AGE, data = gss_replication) summary(model_adjusted) Step 5: Compare. Original paper: b(EDUC) = 3900, R-squared = .25 Your replication: b(EDUC) = 3980, R-squared = .252 (bivariate model) b(EDUC) = 3810, R-squared = .268 (age-adjusted model) Step 6: Evaluate. The replication is close, though not exact — the slope and R² both land within a reasonable range of the original figures, and the direction and approximate magnitude of the relationship hold up, including after adjusting for age as the paper claimed. Minor numerical differences like this are normal and expected, even in a "successful" replication: different sample restrictions (exact age cutoffs, how missing data was handled), slightly different GSS extracts, or rounding in the original paper's reported numbers can all produce small discrepancies without indicating anything is wrong. A good replication report says so explicitly, rather than either overstating a perfect match or treating small differences as a failure. What to Do When You Can't Perfectly Replicate You often won't get an exact match, and that's normal, not a failure on your part. Common, entirely legitimate reasons include: a different GSS wave or extract than the original authors used, ambiguity in the original paper's methods section about exactly how a variable was recoded, missing data handled differently, or — occasionally, and this is fine to say plainly in your write-up — the original finding may simply not hold up as strongly as reported. Per Chapter 1 and this chapter's opening, that outcome is a known, expected feature of the current published literature, not a sign you did something wrong. Document your process, be precise about where and how your results diverge, and resist the temptation to quietly adjust your own analysis until it matches the paper — that would be p-hacking in service of someone else's original p-hacking, and this book has spent an entire chapter (13) plus most of this one explaining why that's not the goal. Open Science Corner #355C7D #FFFFFF #F8B195 Share your replication script on OSF, exactly as you've been asked to do with every analysis since Chapter 5. This is the single most direct, personal way to practice the professional habit this entire book has been building toward: someone else — a classmate, a future employer, your own future self in three years — should be able to open your project, read your script, and understand precisely what you did and why, without having to ask you. Chapter Summary Replication and reproduction are related but distinct checks — same data versus new data — and both matter to a healthy discipline; attempting either is a normal scientific practice, not an accusation. P-hacking, in its many quiet forms (optional stopping, HARKing, selective outcome reporting, and more), is a major contributor to the roughly 50% replication rate Chapter 1 and this chapter both cite, and transparency (pre-registration, honest effect-size reporting, shared scripts) is the field's best available defense against it. A real replication project follows a clear workflow: read, download, clean, analyze, compare, evaluate — and an imperfect match is normal, expected, and worth reporting honestly rather than quietly smoothed over. Every statistical skill in this book — from Chapter 2's first script to Chapter 16's regression diagnostics — exists in service of exactly this kind of careful, honest, checkable work. The Capstone Project Guide This chapter's practice is the capstone project. Use this section as your project guide. Finding a paper. Choose a published, quantitative sociology (or closely related) article that uses GSS data and reports a specific, replicable statistical claim — a mean, a correlation, a t-test, a chi-square result, a regression coefficient. Prefer a paper using a GSS wave reasonably close to 2010, given this book's data, and prefer a paper whose methods section is specific enough to actually follow (vague methods sections are a legitimate, worth-flagging finding in their own right, but make for a harder capstone project). What to submit. At minimum: (1) a full citation and brief summary of the original paper's claim; (2) your annotated R script, following every good-practice habit from this book (comments explaining your decisions, base R or dplyr as appropriate, saved intermediate files where useful); (3) a comparison table or figure, original versus replicated results, in the style of the worked example above; (4) a written evaluation — did it replicate, and what do you make of that. How to write up your results. Structure your write-up the way this book has modeled every inferential result since Chapter 13: state the original claim, state your replicated result with full statistics (test statistic, p-value, effect size — never omit the effect size), and interpret the comparison honestly, including any discrepancies and your best explanation for them. When you can't perfectly replicate. See the section above. Report it precisely, explain your best guess at why, and resist the urge to make it "work." A closing note, and the end of this book. You now have every tool this project needs — R and RStudio (Chapter 2), the GSS itself (Chapter 3), data cleaning and recoding (Chapters 5–6), description (Chapters 7–10), and inference (Chapters 11–17). The field you're entering has spent the last decade and a half doing something genuinely admirable: publicly, uncomfortably, and repeatedly checking its own work, and building better habits as a result. The numbers in Chapter 1 were not flattering. The response to them — pre-registration, open data, shared code, the entire open science movement this book has woven through every chapter — is real, is working, gradually, and is still being built by the people who will read this book next. That's you. Good luck out there, and show your work. References