Statistics for Sociologists/Chapter 16: Bivariate Regression, Correlation, and Effect Sizes

Chapter 16: Bivariate Regression, Correlation, and Effect Sizes If sociology has a single workhorse statistical tool, it's the one in this chapter. Regression shows up in journal articles far more often than t-tests, chi-square tests, or ANOVA combined, for a simple reason: it answers a question sociologists ask constantly — given a predictor, how much of the outcome can we account for, and in which direction? This chapter builds the bivariate (two-variable) version. Go slowly. More students report feeling lost here than in any other chapter in this book, and that's a reasonable reaction to a genuinely dense idea, not a sign you're missing something everyone else finds obvious. What Regression Does Regression predicts a continuous outcome (the dependent variable, Chapter 4) from one or more predictors (independent variables). The bivariate case — one predictor, one outcome — is the foundation everything else in this chapter, and eventually multiple regression, builds on. The Regression Equation Y is the outcome you're predicting. X is the predictor. a (the intercept) is the predicted value of Y when X equals zero. b (the slope) is the amount Y is predicted to change for every one-unit increase in X. ε (epsilon, the residual) is everything the model doesn't explain — the gap between what the model predicts and what actually happened for a given respondent. The slope is the single most important number in this chapter, and it's worth memorizing its interpretation in exactly this form: "for every one-unit increase in X, Y is predicted to change by b units." Get comfortable saying that sentence out loud with your own variables plugged in. The intercept is sometimes meaningful and sometimes almost absurd, and checking which situation you're in is a habit worth building immediately. The intercept is the predicted value of Y when X is zero. This is sometimes meaningful (if someone could plausibly have zero years of education) and sometimes not (if someone could plausibly have zero age, they are not a GSS respondent — the survey only interviews adults, so an intercept describing "predicted income at age zero" is a mathematical artifact of extending a line back to a region with no real respondents in it, not a meaningful prediction about anyone). Always check whether X = 0 falls inside the range of your actual data before interpreting the intercept as though it describes something real. Correlation: Measuring the Strength of a Linear Relationship Before fitting an actual line, it's worth meeting the statistic that measures how tightly two continuous variables move together in the first place: Pearson's correlation coefficient, r.Pearson, K. (1896). Mathematical contributions to the theory of evolution. III. Regression, heredity, and panmixia. Philosophical Transactions of the Royal Society of London. Series A, 187, 253–318. r ranges from −1 to +1. The sign tells you the direction of the relationship (positive: as one variable goes up, so does the other; negative: as one goes up, the other tends to go down), and the magnitude tells you its strength, regardless of direction — an r of −0.6 and an r of +0.6 represent equally strong relationships, just pointed in opposite directions. An r of 0 means no linear relationship at all (though, importantly, it doesn't rule out a strong non-linear one — correlation specifically measures straight-line association). cor(gss$EDUC, gss$CONRINC, use = "complete.obs") cor.test(gss$EDUC, gss$CONRINC) cor() gives you the coefficient itself; cor.test() wraps it in a full hypothesis test (testing the null hypothesis that the true population correlation is zero) with a p-value and confidence interval, following the exact same testing logic as every other inferential procedure in this book since Chapter 13. Rough interpretation benchmarks track closely with Cohen's d's own logic from Chapter 13: around 0.1 is a small relationship, around 0.3 is medium, and 0.5 or above is large — guides, not laws, exactly as every other effect-size benchmark in this book has been. It would be professional malpractice for a statistics textbook to introduce correlation without the obligatory warning, so: correlation is not causation. Ice cream sales and drowning deaths are, across any given year, strongly positively correlated. Ice cream does not cause drowning (nor, to be fair, does drowning cause a craving for ice cream). Both rise and fall together because of a third variable doing the actual causal work: hot weather drives people toward both swimming pools and ice cream trucks simultaneously. A strong correlation tells you two variables move together; it never tells you why, on its own, and mistaking the two is one of the more common — and more avoidable — errors in published social science. The Regression Equation, Revisited Correlation tells you how strongly two variables move together. Regression takes the next step: it fits an actual line, with a specific slope and intercept, that lets you generate a predicted value of Y for any given value of X. The two are close mathematical relatives — in the bivariate case (one predictor, exactly like this chapter), r² and R² (introduced shortly) are numerically identical — but regression gives you something correlation alone doesn't: a concrete, interpretable prediction equation. Ordinary Least Squares How does R decide where to draw the line? Ordinary Least Squares (OLS) finds the specific slope and intercept that minimize the sum of squared residuals — the total squared distance between every actual data point and the line's prediction for it: where Ŷᵢ ("Y-hat") is the line's predicted value for respondent i, and the difference between that prediction and the respondent's actual Y is the residual. Squaring, once again (Chapter 9, Chapter 14), keeps positive and negative errors from canceling out, and it has the added effect of penalizing large errors disproportionately more than small ones — a single wildly mispredicted respondent costs the model more, in this framework, than several respondents each mispredicted by a little. This book won't walk through the calculus that derives the OLS solution; R does that instantly, and the conceptual picture (find the line that leaves the least total squared distance to every point) is what matters for using and interpreting the result. Running Regression in R model <- lm(CONRINC ~ EDUC, data = gss) summary(model) Reading the output: summary(lm()) returns, for each predictor, an estimated coefficient (the slope, for EDUC), a standard error, a t-value, and a p-value — testing the null hypothesis that this particular slope is zero in the population. It also returns the intercept as its own row, with its own standard error and p-value, though the intercept's significance test is rarely the interesting part of the output. Effect Size: R² R² (R-squared) is regression's effect size: the proportion of variance in Y explained by X, ranging from 0 (the predictor explains nothing) to 1 (the predictor explains everything, which never actually happens with real social data). where SS_total is the total squared deviation of every Y value from its own mean (all the variation there is to explain), and SS_residual is what's left over after the model's predictions are subtracted out (the variation the model failed to explain). As promised above, in the bivariate case this is exactly the correlation coefficient squared: R² = r². Adjusted R² applies a small penalty for the number of predictors in the model — irrelevant for a single-predictor bivariate model like this chapter's, but essential once you're comparing models with different numbers of predictors, which is exactly the situation multiple regression (briefly previewed at this chapter's end) puts you in. summary(model)$r.squared summary(model)$adj.r.squared An R² of 0.15, for instance, means education accounts for about 15% of the variation in income in the sample — real and worth reporting, but leaving 85% of the variation explained by everything else: occupation, region, discrimination, luck, and dozens of other factors this bivariate model doesn't include. R² is, not coincidentally, closely related to eta-squared from Chapter 15 — both express "proportion of variance explained," just for different kinds of predictors (continuous here, categorical there). Assumptions and Diagnostics Regression rests on assumptions worth actually checking rather than assuming away: Linearity — does a straight line actually describe the relationship? Check with a scatterplot before you trust the model at all. Homoscedasticity — is the spread of residuals roughly constant across the full range of X, rather than fanning out or narrowing at one end? Check with a residual plot. Normality of residuals — are the residuals themselves roughly normally distributed? Check with a Q-Q plot (Chapter 10). plot(model) # produces four standard diagnostic plots, including residuals-vs-fitted and a Q-Q plot plot() called directly on an lm() object is one of R's more genuinely useful pieces of built-in convenience — four diagnostic plots from one line of code, no manual construction required. Visualizing Regression Base R: plot(gss$EDUC, gss$CONRINC, xlab = "Years of Education", ylab = "Income (CONRINC)") abline(model, col = "red") ggplot2: library(ggplot2) ggplot(gss, aes(x = EDUC, y = CONRINC)) + geom_point(alpha = 0.3) + geom_smooth(method = "lm") + labs(title = "Income by Years of Education", x = "Years of Education", y = "Income (CONRINC)") geom_smooth(method = "lm") conveniently adds both the regression line and a shaded confidence band around it in a single layer — a nice visual complement to Chapter 12's confidence interval material, applied here to a whole predicted relationship rather than a single mean. Reading a Regression Table Published sociology papers rarely show raw summary(lm()) output — they show a formatted regression table, usually with coefficients, standard errors (often in parentheses beneath each coefficient), significance stars, and model fit statistics (R², N) at the bottom. Learning to translate between R's console output and a published table's format is its own small skill, worth practicing now rather than being surprised by it later when reading journal articles for a literature review. A Note on Multiple Regression Everything in this chapter generalizes to multiple regression — more than one predictor in the same model (lm(CONRINC ~ EDUC + AGE + SEX, data = gss)), which is how most published sociological regression analyses actually look, since a single predictor rarely tells the whole story. Multiple regression is beyond this book's scope, but you now have every conceptual piece you need to understand it once you encounter it: the equation just grows more terms, each with its own slope, each interpreted "holding the other predictors constant." That last phrase is doing a great deal of work, and it's exactly the kind of thing a follow-up course will unpack properly. Open Science Corner #355C7D #FFFFFF #F8B195 Share your R script for any published regression analysis. A regression table printed in a journal article is a promise about what was found; the script that produced it is the only thing that turns that promise into something another researcher can actually check. Post it to OSF alongside your data (where licensing and privacy allow) and cite it in the paper itself — this is quickly becoming a baseline expectation in quantitative sociology, not an optional extra. Chapter Summary Pearson's r measures the strength and direction of a linear relationship between two continuous variables, computed with cor() and tested with cor.test() — and, as always, comes with the obligatory reminder that correlation is not causation. Bivariate regression takes the next step, predicting a continuous outcome from a single predictor via the equation Y = a + bX + ε, with the slope (b) — "for every one-unit increase in X, Y changes by b units" — as the most important number to interpret correctly, and the intercept (a) requiring a check of whether X = 0 is actually meaningful before you interpret it. R finds the OLS-best line via lm(), and summary() reports coefficients, standard errors, and significance tests for each. R², regression's effect size, expresses the proportion of variance in Y explained by X, connects directly to eta-squared from Chapter 15, and in the bivariate case is numerically identical to r² — correlation and regression, quantifying the same underlying relationship from two angles. Diagnostics — linearity, homoscedasticity, normality of residuals — deserve an actual check via plot(model) rather than an assumption that they hold, and ggplot2's geom_smooth(method = "lm") visualizes both the fitted line and its uncertainty in one step. Multiple regression, beyond this book's scope, extends everything here to several predictors simultaneously and is where most published sociological regression work actually lives. Practice Examples Example 1: Does Education Predict Income? The Research Question: How much does education predict income, and how should the slope and intercept be interpreted? The Code: model <- lm(CONRINC ~ EDUC, data = gss) summary(model) The Output: Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -8420.5 2150.3 -3.92 9.2e-05 EDUC 3980.2 152.7 26.06 < 2e-16 Multiple R-squared: 0.252, Adjusted R-squared: 0.251 The Interpretation: For every additional year of education, predicted income increases by about $3,980, a highly significant relationship (p < .001). The intercept (−$8,420) is the model's predicted income for zero years of education — a mathematically necessary part of the line, but not a claim about any real respondent, since GSS adults with literally zero education are essentially nonexistent in the sample. Education accounts for about 25% of the variance in income (R² = 0.252) — a genuinely substantial share for a single predictor in social science, though still leaving three-quarters of the variation in income explained by factors outside this simple model. Follow-Up Challenge: Use the slope and intercept to predict the income of a respondent with 12 years of education, and separately, one with 20 years. How large is the gap between your two predictions, and does that gap feel intuitively reasonable to you? Example 2: Scatterplot and Diagnostics The Research Question: Does the education-income relationship actually look linear, and do the model's assumptions hold up? The Code: library(ggplot2) ggplot(gss, aes(x = EDUC, y = CONRINC)) + geom_point(alpha = 0.3) + geom_smooth(method = "lm") + labs(title = "Income by Education", x = "Years of Education", y = "Income") plot(model) # diagnostic plots The Output: A scatterplot with a fitted regression line and confidence band, plus four diagnostic plots (residuals vs. fitted, Q-Q plot of residuals, scale-location, residuals vs. leverage). The Interpretation: The scatterplot typically shows a genuine positive trend, but with substantial scatter around the line at every education level — consistent with the modest-but-real R² from Example 1. The residual plot commonly reveals a classic regression problem with income data specifically: heteroscedasticity — residual spread tends to fan outward at higher education (and higher predicted income) levels, since income variability itself grows among higher earners (recall Chapter 7 and Chapter 9's discussions of income's severe right-skew). This is worth naming honestly rather than ignoring; a more advanced course would address it with a log-transformed outcome or a different modeling approach, both beyond this book's scope but good to know exist. Follow-Up Challenge: Try re-running the model with log(CONRINC) as the outcome instead of raw CONRINC (you'll need to handle any zero or negative values first). Does the residual plot look more well-behaved afterward? Example 3: Full Write-Up The Research Question: How would the Example 1 result appear in a published sociology paper? The Code: (No new code — translating Example 1's output into prose.) The Output: A bivariate OLS regression found that years of education significantly predicted income, b = 3980.2, SE = 152.7, t(2015) = 26.06, p < .001, R-squared = .252. Each additional year of education was associated with an estimated $3,980 increase in income. The model accounted for approximately 25% of the variance in income, indicating that while education is a substantial predictor, the majority of income variation in this sample remains unexplained by education alone. The Interpretation: Notice that this write-up, like every reporting template in this book since Chapter 13, refuses to let the significant p-value stand alone — R² is reported explicitly, and its practical meaning (75% of variance still unexplained) is spelled out in plain language rather than left for the reader to infer. Follow-Up Challenge: Rewrite this same result the way it might appear in a formatted regression table (coefficient, standard error in parentheses, significance stars, R² and N at the bottom) rather than as a sentence. Which format communicates more efficiently, and which communicates more completely? References