Statistics for Sociologists/Chapter 5: Loading, Exploring, and Cleaning GSS Data

Chapter 5: Loading, Exploring, and Cleaning GSS Data Data cleaning is 80% of data analysis. The other 80% is also data cleaning. (The math doesn't add up. That's the joke.) Cleaning data does not feel like a discrete phase you finish and move past — it's more like laundry: you will do it again next week, on a different dataset, and it will still take longer than you think. This chapter teaches that unglamorous, essential work. Real data is messy. GSS data, gathered from tens of thousands of real conversations over five decades, is especially messy. The good news is that "messy" here means "predictably messy" — the same handful of problems show up again and again, and once you know what to look for, cleaning a dataset stops being mysterious and starts being routine. alt=Image showing messy survey data on the left and clean data on the right.|center|thumb|800x800px|Raw data before it is cleaned can be messy. R is an excellent tool for cleaning data. Why Data Cleaning Exists Data as collected is never quite the same thing as data ready for analysis. A respondent skips a question. An interviewer enters "don't know" because that's what was said. A variable that should be numeric gets stored as text because one row has a stray character in it. None of this is anyone's fault, exactly — it's just what happens when real information, gathered from real people, gets forced into a rectangular spreadsheet. The job of this chapter is to close the gap between the data you were handed and the data you can actually trust a mean() or a t.test() to run on. Loading Data (A Recap, With Options) You met the basics of this in Chapter 3, but here's the fuller picture, since Chapter 3 focused specifically on the GSS and this chapter's habits apply to any dataset you'll ever load. R's Own Format If you're lucky, it's possible someone may have already collected and stored the data in R's dataframe format, RDATA. If so, opening the file is very straightforward: load("GSS.RData") If that is not the case, not to worry. R can open pretty much any data format. Base R: gss <- read.csv("gss2010.csv") gss <- read.table("gss2010.txt", header = TRUE, sep = ",") dplyr / tidyverse: library(readr) gss <- read_csv("gss2010.csv") haven (this course's default, from the .sav file): library(haven) gss <- read_sav("gss2010.sav") Three functions, three formats, one goal: a data frame in R's memory that you can start examining. Which one you reach for depends entirely on what format the data arrives in — which, as any working researcher eventually discovers, is not always the format you'd choose. A Field Guide to Other File Formats The GSS today is easy to obtain as a clean .csv or .sav file through the Data Explorer. This has not always been true, and it will not always be true of every dataset you encounter in your research career. Sociologists routinely inherit data from SAS, from Stata, or from decades-old fixed-width .dat files that predate any of the friendlier tools this book has shown you so far — the GSS itself was, for most of its history, distributed exactly this way: a raw ASCII .dat file paired with a separate "syntax" file (for SPSS, SAS, or Stata) that told the software which character positions belonged to which variable. If you ever inherit a dataset from a professor who's been doing this since 1998, this is almost certainly what you'll get. Fixed-width .dat files store every variable in a fixed set of character columns, with no delimiters between them — the file looks like a wall of digits until you know exactly which positions to read. Both base R and readr can handle this if you know (or can find, usually in an accompanying syntax file) the column widths: # Base R: reading fixed-width data gss_fwf <- read.fwf("gss_extract.dat", widths = c(2, 1, 2, 2), # character-width of each variable col.names = c("AGE", "SEX", "EDUC", "HAPPY")) # readr equivalent library(readr) gss_fwf <- read_fwf("gss_extract.dat", fwf_widths(c(2, 1, 2, 2), col_names = c("AGE", "SEX", "EDUC", "HAPPY"))) SAS files (.sas7bdat) and Stata files (.dta) are both handled by the same haven package you're already using for .sav files — and both carry similar embedded metadata (value labels, sometimes missing-value definitions) that haven will bring along automatically, the same way it does for SPSS files: library(haven) sas_data <- read_sas("some_dataset.sas7bdat") stata_data <- read_dta("some_dataset.dta") None of this requires memorizing new concepts — it's the same library() then read_something() pattern you already know, applied to a different file extension. The only genuinely new skill is fixed-width files, since you have to supply column positions the file itself won't tell you (that's what the syntax file, or a coauthor with institutional memory, is for). The lesson worth internalizing: the format a dataset arrives in says nothing about how interesting or usable the data inside it is. Adapt to the format; don't let it decide whether the project is worth doing. Importantly, think of R and RStudio as the Swiss Army knife of datasets: it can handle pretty much anything you throw at it. It is remarkably robust software for data management and analysis. First Look at a Dataset Whatever format it arrived in, once it's a data frame in R, the routine is the same: head(gss) # first six rows tail(gss) # last six rows str(gss) # structure: variable names and types summary(gss) # quick numeric summary of every column dim(gss) # rows and columns names(gss) # just the variable names View(gss) will open a spreadsheet-style viewer in RStudio, which is genuinely useful for a quick visual scan — but use it sparingly on a dataset with thousands of rows. Scrolling through 2,000 respondents by eye is not a data cleaning strategy; it's a way to convince yourself you've checked something you haven't. A useful note here is in order. Some people may have initially been taught to use SPSS for their statistical analyses. SPSS uses a spreadsheet like layout that also allows users to edit individual data points. That is not a best practice for open science-style research. All changes to a dataset should be made with code so they can be replicated. Any researcher should be able to hand their raw data and script files to another researcher and the new researcher can quickly get a clean dataset from the raw data by following the script file. That is good, open science practice (see below). Try not to get into the habit of looking at your data as a spreadsheet or feeling like you need to visually inspect data. You can always retrieve specific information from the dataset without visually inspecting the entire dataset. This is especially true with very large datasets. Hypothetically, let's say you've imported some data, messed around with it, and now there is a problem. Not to worry. You can always remove rm() objects, including dataframes/datasets from the R environment: rm(gss) Identifying Problems Three categories of problems show up in almost every real dataset, GSS included: Missing values — represented as NA in R natively, but hiding as ordinary-looking numbers (0, 8, 9, 98, 99) in a freshly loaded GSS CSV file. Values out of expected range — an age of 150, a year of education of -3. Rare in a well-curated dataset like the GSS, but worth a sanity check with summary() or range() before you trust anything. Variables stored as the wrong type — a categorical variable stored as a number (Chapter 4's whole subject), or occasionally a numeric variable stored as text because of one stray non-numeric entry. Catching these three things, in this order, before you run a single substantive analysis, is most of what separates a careful data analyst from someone who's going to have an uncomfortable conversation with their professor about a mean that turned out to include forty-four "refused to answer" codes. Handling Missing Data R represents "no value here" with a special symbol, NA, which behaves differently from any ordinary number — critically, NA is contagious: almost any calculation that touches an NA returns NA right back, on the reasonable logic that R can't know the answer to a question that includes a missing piece. is.na(gss$CONRINC) # TRUE/FALSE for each value: is it missing? sum(is.na(gss$CONRINC)) # how many missing values, total? mean(gss$CONRINC) # if any NAs are present, this returns NA mean(gss$CONRINC, na.rm = TRUE) # this tells R to ignore NAs and compute anyway na.rm = TRUE is not optional decoration — most summary functions in R (mean(), sd(), sum(), and others) will return a bare NA for the whole calculation if even one value is missing, unless you explicitly tell them to skip missing values first. This will happen to you at least once. It is not a bug; it's R declining to guess. Removing rows with missing values, base R: gss_complete <- na.omit(gss) # drops any row with ANY missing value gss_no_missing_income <- gss[!is.na(gss$CONRINC), ] # drops rows missing CONRINC specifically Removing rows with missing values, dplyr: library(dplyr) gss_no_missing_income <- gss %>% filter(!is.na(CONRINC)) Notice that na.omit() is a blunt instrument — it drops a whole respondent for being missing on any variable in the dataset, which can quietly shrink your sample far more than you intended if your data frame has fifty columns and each respondent is missing at least one of them. Prefer the targeted, variable-specific approach unless you have a good reason not to. Replacing GSS Missing Codes With NA If you loaded your data from a plain CSV (rather than a .sav file via haven, which typically handles a meaningful share of this for you automatically, as you saw in Chapter 3), GSS's numeric missing-value codes — 0, 8, 9, 98, 99, depending on the variable — are sitting in your data disguised as ordinary numbers, and you have to tell R which ones to treat as missing. This problem is worst for banded, categorical GSS variables, where a small integer like 98 or 99 is easy to confuse with a real category code. RINCOME (respondent's personal income, reported in brackets) is a textbook case: Base R: gss$RINCOME[gss$RINCOME == 99] <- NA dplyr, with na_if() for a single code: library(dplyr) gss <- gss %>% mutate(RINCOME = na_if(RINCOME, 99)) dplyr, with case_when() for multiple codes at once: gss <- gss %>% mutate(RINCOME = case_when( RINCOME %in% c(0, 98, 99) ~ NA_real_, TRUE ~ RINCOME )) car recode command gss$RINCOME <- car::recode(gss$RINCOME, '99=NA') This step is exactly the manual work haven's read_sav() often does for you automatically, based on the .sav file's own embedded metadata — but "often" is doing real work in that sentence. Not every missing code is guaranteed to be flagged in a file's metadata, some datasets you'll work with beyond this course won't be .sav files at all, and understanding what's actually happening under the hood is what lets you catch it when the automatic version doesn't. Learn this the manual way regardless of which import method you used. This is also, incidentally, exactly why this book prefers CONRINC over RINCOME or banded INCOME for continuous-income examples. CONRINC is a derived variable — NORC has already done the work of converting bracketed responses into a single constant-dollar figure — and it generally arrives without this same small-integer sentinel-code problem, since a real income value in the thousands of dollars is never going to be confused with a placeholder like 98 or 99. That doesn't mean CONRINC is automatically free of missing data (respondents who didn't answer the underlying income questions will still show up as missing), only that you're less likely to need the manual sentinel-code detective work above to find it. Always confirm this against the actual codebook rather than assuming — but it's a real, and reasonable, part of why CONRINC is the simpler variable to hand to students. Open Science Corner #355C7D #FFFFFF #F8B195 Script your cleaning steps. Never clean data by hand in a spreadsheet program and save over the original file — a hand-edited spreadsheet leaves no trace of what you changed or why, and "I'm pretty sure I fixed that one weird row" is not a reproducible research method. Your cleaning script IS part of your research documentation, not a throwaway step before the "real" analysis begins. Subsetting Data Once your data is clean, you'll frequently want to work with only some of it — certain rows, certain columns, or both. Base R bracket notation: data[rows, columns], where leaving either side blank means "keep all of them." under_40 <- gss[gss$AGE < 40, ] # rows: respondents under 40; all columns demo_vars <- gss[, c("AGE", "SEX", "RACE", "EDUC")] # all rows; only these columns Base R subset: subset under_40 <- subset(gss, AGE < 40) dplyr: filter() for rows, select() for columns. library(dplyr) under_40 <- gss %>% filter(AGE < 40) demo_vars <- gss %>% select(AGE, SEX, RACE, EDUC) The dplyr versions read almost like English sentences, which is exactly the readability advantage this book keeps pointing to — "filter to rows where age is under 40" versus a bracket with a comma in a specific position that means something only because you've memorized the convention. Saving a Cleaned Dataset Once you've cleaned a dataset, save it — as its own file, distinct from the raw original, so you (and anyone checking your work) can always see exactly what the data looked like before and after your changes. write.csv(gss, "gss2010_cleaned.csv", row.names = FALSE) save(gss, file = "gss2010_cleaned.RData") write.csv() produces a plain-text file readable by essentially any software on Earth; save() produces an R-specific .RData file that preserves R object types exactly (including any factors or haven-labelled columns) but requires R to open. Save both if storage space isn't a concern — the CSV for portability, the RData for fidelity. Chapter Summary Data cleaning closes the gap between data as collected and data as analyzable, and it follows a predictable routine: load the data (from whatever format it happens to arrive in — CSV, SPSS, SAS, Stata, or an old-school fixed-width file), take a careful first look with head(), str(), summary(), and names(), identify missing values and out-of-range or mistyped variables, and handle missing data explicitly — either by letting R's native NA do its job with na.rm = TRUE, or by manually recoding GSS's numeric missing codes (0, 8, 9, 98, 99) to real NA values when they haven't already been handled by your import method. From there, select()/filter() (or their base R bracket equivalents) let you subset to exactly the rows and columns your analysis needs, and saving a distinct cleaned file preserves an honest record of what changed. None of this is the exciting part of the semester. All of it is the part that determines whether the exciting part can be trusted. Practice Examples Example 1: Find and Replace Missing Codes in RINCOME (and Compare to CONRINC) The Research Question: What do the raw values of RINCOME look like, which ones need to be treated as missing before any analysis, and how does this compare to the continuous CONRINC variable? The Code (base R): table(gss$RINCOME) # if you imported the data via haven, try this instead: table(gss$RINCOME, useNA = "ifany") # that will show you missing cases that have been labeled "NA" by the haven package gss$RINCOME[gss$RINCOME %in% c(0, 98, 99)] <- NA sum(is.na(gss$CONRINC)) The Code (dplyr): library(dplyr) gss %>% count(RINCOME) gss <- gss %>% mutate(RINCOME = case_when( RINCOME %in% c(0, 98, 99) ~ NA_real_, TRUE ~ RINCOME )) The Output: 1 2 3 ... 0 98 99 120 340 410 ... 8 22 15 [1] 30 # sum(is.na(gss$CONRINC)) The Interpretation: The frequency table for RINCOME reveals a handful of respondents coded 0, 98, or 99 — "not applicable," "don't know," and "refused," respectively, none of which describe an actual income bracket. Recoding them to NA removes them from any future calculation without deleting the respondents' other answers; that respondent's AGE or HAPPY response is still perfectly usable, only their income is now honestly marked as unknown rather than silently treated as the lowest bracket (which would be a very different, and very wrong, claim). CONRINC, by contrast, simply shows 30 missing values with no sentinel-code detective work required — a direct illustration of why this book uses it as the default continuous income variable from here forward. Follow-Up Challenge: Compute mean(gss$RINCOME, na.rm = TRUE) before and after this recode. How much does the recoded mean change? Then compute mean(gss$CONRINC, na.rm = TRUE) and compare — does the "simpler" variable tell a similar story? Example 2: Subset to Respondents Under 40 The Research Question: How does the demographic profile of a data subset — say, respondents under 40 — compare to the full sample? Start by getting the total number of rows in the GSS dataset: Get Number of Rows: nrow(gss) You should get a result like: [1] 2044 Now, let's subset the dataset: The Code (base R): under_40 <- gss[gss$AGE < 40, ] nrow(under_40) The Code (dplyr): library(dplyr) under_40 <- gss %>% filter(AGE < 40) nrow(under_40) The Output: [1] 1058 The Interpretation: Both approaches produce an identical subset — 1058 of the original respondents are under 40 — using two different but equally valid syntaxes. Subsetting like this is the first step toward almost every comparative analysis in this book: you'll do exactly this kind of filtering before running a t-test comparing two groups in Chapter 13, or an ANOVA comparing several in Chapter 15. Follow-Up Challenge: Create a second subset of respondents 40 and older, and compare the two groups' average EDUC using nothing more than two calls to mean(..., na.rm = TRUE). Resist the urge to conclude anything definitive about whether the difference is "real" — that question is exactly what Chapter 13 will teach you to answer properly. Example 3: Select Variables and Save a Cleaned Dataset The Research Question: How do you produce a smaller, cleaned working file containing only the variables relevant to a specific analysis? The Code (base R): gss_working <- gss[, c("AGE", "SEX", "EDUC", "CONRINC", "HAPPY")] write.csv(gss_working, "gss2010_working.csv", row.names = FALSE) The Code (dplyr): library(dplyr) gss_working <- gss %>% select(AGE, SEX, EDUC, CONRINC, HAPPY) write.csv(gss_working, "gss2010_working.csv", row.names = FALSE) The Output: A new file, gss2010_working.csv, appears in your project folder, containing only the five selected columns for every respondent. The Interpretation: There's no statistical output here — the "result" is a file on disk, and that's the point. Keeping a lean, purpose-built working dataset alongside (never instead of) your full raw file is good practice: it's faster to work with, it's clearer about what a given analysis actually used, and it can be shared or archived (see this chapter's Open Science Corner) without dragging along every variable you didn't end up needing. Follow-Up Challenge: Open gss2010_working.csv in a plain text editor or spreadsheet program outside of R. Does it look the way you expected? What would happen if you accidentally overwrote your original gss2010.csv with this smaller file by giving it the same name? Example 4: Reading a Fixed-Width or Foreign-Format File The Task: Practice reading a dataset from a format other than CSV or .sav — the kind of file you'll likely encounter with older or externally supplied data. The Code: # A hypothetical fixed-width extract with AGE (2 chars), SEX (1 char), EDUC (2 chars) practice_data <- read.fwf("practice_extract.dat", widths = c(2, 1, 2), col.names = c("AGE", "SEX", "EDUC")) str(practice_data) # A hypothetical Stata file from a collaborator library(haven) stata_practice <- read_dta("collaborator_data.dta") str(stata_practice) The Output: A data frame with the specified columns, correctly split from the fixed-width source; for the Stata example, a tibble with haven-labelled columns similar to what you saw from read_sav() in Chapter 3. The Interpretation: The mechanics barely change across formats — you're still ending up with a rectangular data frame you already know how to explore and clean. The only real new skill is fixed-width files, where you have to supply the column boundaries yourself rather than relying on a delimiter or embedded structure. This is a real, not hypothetical, skill: sociology students who go on to work with historical datasets, restricted-use files, or data inherited from a retiring colleague's SAS or Stata scripts will hit this eventually, often with far less friendly documentation than this textbook provides. Follow-Up Challenge: Find a publicly available dataset (a government statistical agency is a good bet) that is distributed as a .dat file with a separate layout/syntax file. Without necessarily downloading the full thing, look at the syntax file and identify what information you'd need to extract from it to build the widths and col.names arguments for read.fwf(). References