Lecture 10b Activity

Unit D · Chapter 10

An in-class activity. Nothing to turn in and no answer key – this one needs other people, which is why it happens in class rather than at home.

The idea. Once you have seen which way deleting a case pushes your \(p\)-value, you can no longer make an unbiased decision about deleting it. The room writing rules before the reveal is the only way to demonstrate that honestly.

Format. About 15 minutes · threes · one laptop per group.

1. Look, then stop (4 minutes)

library(tidyverse)
library(easystats)

yp <- read_csv("yearspubs.csv")
fit <- lm(n_cites ~ n_pubs + yrs_since, data = yp)

check_outliers(fit)         # the default verdict
plot(check_outliers(fit))   # every case's influence, seen at once

The default threshold reports no outliers. But look at the plot: one case sits far beyond every other – about five times the influence of the runner-up.

Do not run anything else yet.

2. Write your rule (5 minutes)

On a sheet the group signs, write a rule you would be willing to preregister:

  • What counts as an outlier here? Name the statistic and the threshold, and say whether you accept a rule that flags nothing in this dataset.
  • Under what circumstances would you remove a flagged case? Under what circumstances keep it?
  • What will you report either way?

Hand the sheet to the front. No changes after this point.

3. The reveal (6 minutes)

Ask the same function for the other rule from the lecture. Cook’s distance asks how much a case moves the model; Mahalanobis distance asks how unusual its predictor values are. Same data, same function, both at their own defaults.

outliers <- check_outliers(fit, method = "mahalanobis")
outliers      # this rule does flag something
yp[24, ]      # who is it?

yp_drop <- yp[!as.numeric(outliers), ]   # remove it, exactly as in the slides
fit_drop <- lm(n_cites ~ n_pubs + yrs_since, data = yp_drop)
compare_parameters(fit, fit_drop)
  • Apply your own rule. What does your group report?
  • Read out two or three rules from the front. Does the room’s answer depend on whose rule you use? That is the finding.
  • Now the honest question, and answer it to each other: having seen the reveal, would you have written the same rule five minutes ago?
  • The package default said there was nothing to see. What does that tell you about using a default as your decision rule?

If you have more time: run check_model() on four models breaking four different assumptions and name each from its plot before running any test.

If you were not in class

The influential case is a researcher with 69 publications and 13 years since their PhD – a real, plausible person and the most influential observation in the dataset. Removing that single row moves n_pubs from \(+0.20\) to \(-0.04\) and yrs_since from \(p = .10\) to \(p = .04\). One row decides which predictor the paper is about, and reasonable rules written by different groups will disagree about whether to keep it.

That disagreement is the point, and it is not a failure of the room. It is what “researcher degrees of freedom” means, made visible: several defensible analysts with the same data and different, honestly-held rules produce different papers. The protection is not finding the one true threshold; it is fixing your rule before the reveal and reporting the analysis both ways.

Note also what the two rules did. Cook’s distance at its default threshold of 0.8 reported nothing; Mahalanobis distance at its default flagged this case immediately. Neither is wrong – they measure different things, influence versus unusualness in the predictors – but a room that had run only the first one would have concluded there was nothing here to decide. A default is a convention someone else chose, not a finding.