Statistical Methods with R

NHST & p-values

Unit B · Chapter 05 · Lecture 05a

Developed by Jeffrey M. Girard

Statistical Hypotheses

What is a hypothesis?

• A hypothesis is a testable prediction
  • It derives from a broader scientific theory
  • It needs to be operationalized and falsifiable
• A statistical hypothesis needs to be mathematically precise
  • It is about characteristics of the population of interest
  • It is a comparison involving population parameters
    • The mean of the whole population is 100 \((\mu=100)\)
    • The mean is greater in group 1 than group 2 \((\mu_1>\mu_2)\)

Example research question

• Imagine that we study nutrition and obesity

• Participants really like a brand of “low sugar” cereal

• The label claims that it only contains 3 g of sugar per serving

• However, we think this may be false advertising!

• We buy a sample of 20 cereal boxes from various stores

• We measure the amount of sugar per serving in each box

• Is 3 g a reasonable estimate of the population mean?

Null and alternative hypotheses

• Just like in a criminal trial, we need to represent both sides

• The null hypothesis represents the company’s “defense”
  • Boxes do contain 3 g on average, or \(H_0: \mu=3\)

• The alternative hypothesis represents the “prosecution”
  • Boxes contain more than 3 g on average, or \(H_1: \mu>3\)
  • Boxes do not contain 3 g on average, or \(H_2: \mu\neq3\)

NHST Logic

What is NHST?

• In null hypothesis significance testing (NHST), the null hyp. is the defense and the alternative hyp. is the prosecution

• The statistical test is the judge and jury (makes the decision)

• We begin with a “presumption of innocence”
  • The null hypothesis is assumed true unless…
  • …there is good evidence that it is very unlikely
  • The burden of proof thus lies with the prosecution

Potential test outcomes

1. Retain the null hypothesis
   • There is not enough evidence to reject \(\mu=3\)
   • We cannot be certain that \(\mu\) does not equal \(3\)
   • It does not mean that we are certain that \(\mu=3\)
2. Reject the null hypothesis
   • There is enough evidence to conclude \(\mu\neq3\)
   • The cereal’s nutrion label should be changed…
   • It does not prove that this was deliberate fraud

Potential errors

• In reality, there is only one correct decision
  • In the two-sided case, either \(\mu=3\) or \(\mu\neq3\)
• Due to sampling error, our sample may lead us astray
  • We might make one two possible wrong decisions…

	Retain \(H_0\)	Reject \(H_0\)
\(H_0\) is True	Correct Retain	Type I Error
\(H_0\) is False	Type II Error	Correct Reject

Example outcomes

• In a world where the cereal does contain 3g on average
  • The null hypothesis is true and should be retained
  • If our test retains it, that is the correct decision
  • If our test rejects it, that is a Type I error
• In a world where the cereal doesn’t contain 3g on average
  • The null hypothesis is false and should be rejected
  • If our test retains it, that is a Type II error
  • If our test rejects it, that is the correct decision

Controlling error rates

• NHST is particularly interested in preventing Type I errors
  • Researchers want to reject the null, so NHST protects it

• We control the Type I error rate \((\alpha)\) by modifying the test
  • e.g., we set \(\alpha=.05\) to allow only 5% type I error

• We control the Type II error rate \((\beta)\) via large samples
  • We refer to a test’s statistical power as \(1-\beta\)
  • e.g., we decide \(n\) to get 80% power (only 20% type II error)

Outcome probabilities

	Retain \(H_0\)	Reject \(H_0\)
\(H_0\) is True	\(1-\alpha\)	\(\text{[I] }\alpha\)
\(H_0\) is False	\(\text{[II] }\beta\)	\(1-\beta\)

• If we set \(\alpha\) to .05, we are accepting…
  • Up to a 5% (1 in 20) chance of rejecting a true null hyp.

• If we use a sample with 80% power, we are accepting…
  • Up to a 20% (1 in 5) chance of retaining a false null hyp.

NHST Nuts and Bolts

Test statistics

• How do we decide to retain or reject the null hypothesis?

• We compare our sample data to our “null expectations”
  • What would we expect to see if the null were true?
  • If they match, we we retain the null hypothesis
  • If they do not match, we reject the null hypothesis

• We need a way to quantify the extent of this matching
  • The test statistic quantifies the mismatch

Test statistics

• The population mean expected under the null \((\mu_0)\) is 3
  • Our null expectation is a sample mean \((\bar{x})\) close to this
• The extent of matching here is captured by the difference
  • We can use the difference \((\Delta)\) as a simple test statistic \[\Delta = \bar{x} - \mu_0 \qquad \Delta = \bar{x} - 3\]
  • If the null is true, then \(\Delta\) should be very close to zero
  • We would reject the null if \(\Delta\) is too far from zero

Critical values

• How far from zero is too far? When do we reject the null?

• We look at the sampling distribution of \(\Delta\) under the null

  • If we assume that the null hypothesis is true…
  • …what values would we see with repeated samples?

• We can use this to make our statistical decision

  • We reject \(H_0\) if the test statistic is unlikely under the null
  • If greater than the critical value, we reject the null
  • The critical value controls the Type I error rate \((\alpha)\)

Single sample \(z\)-test

• To set \(\alpha=.05\), we reject the null if the test statistic is outside the middle 95% of its null sampling distribution

• The math becomes easier if we standardize the test statistic by dividing it by its standard error

\[ z_{\bar{x}} = \frac{\bar{x} - \mu_0}{\sigma / \sqrt{n}} \\ z_{\bar{x}} \sim \mathcal{N}(0, 1) \]

Single sample \(t\)-test

• This \(z\)-test is rarely used because \(\sigma\) is rarely known

• Instead, we estimate \(\sigma\) using \(s\) and use the \(t\)-test

• With a small sample, the critical values are now higher

\[ t_{\bar{x}} = \frac{\bar{x} - \mu_0}{s / \sqrt{n}} \\ t_{\bar{x}} \sim t(n-1, 0, 1) \]

Example where \(H_0\) is true

Imagine \(H_0\) is true \((\mu=3)\) and we collect the following sample of 20 cereal boxes

sugar_null <- c(
  3.0, 3.1, 3.1, 3.1, 3.2,
  2.5, 3.1, 3.2, 3.0, 3.4, 
  2.7, 2.9, 3.0, 3.4, 3.4, 
  2.5, 3.4, 3.0, 3.4, 3.4
)

Example where \(H_0\) is true

mu_0 <- 3
s <- sd(sugar_null)
se <- s / sqrt(n)
xbar <- mean(sugar_null)
xbar
## [1] 3.09

t_xbar <- (xbar - mu_0) / se
t_xbar
## [1] 1.423962

t_crit <- qt(0.975, df = n - 1)
t_crit
## [1] 2.093024

\(|t_{\bar{x}}|<t_{crit}: \text{Retain }H_0\)

Example where \(H_0\) is false

Imagine \(H_0\) is false \((\mu=4)\) and we collect the following sample of 20 cereal boxes

sugar_alt <- c(
  3.9, 3.0, 2.1, 3.8, 3.4, 
  5.1, 3.1, 5.0, 3.6, 3.5, 
  4.3, 3.6, 4.6, 4.7, 3.4, 
  4.7, 4.6, 4.5, 4.9, 4.6
)

Example where \(H_0\) is false

mu_0 <- 3
s <- sd(sugar_alt)
se <- s / sqrt(n)
xbar <- mean(sugar_alt)
xbar
## [1] 4.02

t_xbar <- (xbar - mu_0) / se
t_xbar
## [1] 5.699162

t_crit <- qt(0.975, df = n - 1)
t_crit
## [1] 2.093024

\(|t_{\bar{x}}|>t_{crit}: \text{Reject }H_0\)

Making decisions

• Critical values let us decide and control \(\alpha\)
  • \(t_{obs} < t_{crit}\) = Retain the null
  • \(t_{obs} \ge t_{crit}\) = Reject the null
• When we reject the null, the test was “significant”
  • Remember that not all rejections are correct
  • Even a correct rejection may not be important
  • In a large sample, a small effect may be significant

\(p\)-values and
Pragmatics

The \(p\)-value of a test

• But what if we had selected a different \(\alpha\) level?
  • Some \(\alpha\) values would yield a different decision
• The \(p\)-value is a popular approach for dealing with this

Correct p-value Definitions

The \(p\)-value is the smallest \(\alpha\) value that would reject the null hypothesis.
The \(p\)-value is the probability of observing a test statistic at least as extreme as ours.

Wrong p-value Definition

The \(p\)-value is the probability that the null hypothesis is true.

Using the \(p\)-value

• The \(p\)-value is a summary of all possible \(\alpha\) values
  • How would you feel about setting \(\alpha\) to \(p\)?
• It is best to set \(\alpha\) for the study before seeing the data
  • Then, you reject the null if and only if \(p<\alpha\)
• Reporting \(p\) directly softens your decision
  • e.g., \(p=.049\) (Reject) vs. \(p=.051\) (Retain)
  • e.g., \(p=.002\) (Reject) vs. \(p=.047\) (Reject)

Reporting a significance test

Notation	Stars	Translation	Decision
p ≥ .05		Test was NOT significant at α=.05	Retain Null
p < .05	*	Test was significant at α=.05	REJECT Null
p < .01	**	Test was significant at α=.01	REJECT Null
p < .001	***	Test was significant at α=.001	REJECT Null

• Report your exact \(p\)-value (unless less than .001)
  • Saying \(p<.05\) or \(p\ge.05\) discards useful information
• It can be helpful to report a CI and your test statistic too

Putting it all together

• We can do a one-sample \(t\)-test in R with t.test()
• We can get a results table with model_parameters()

fit1 <- t.test(sugar_null, mu = 3)

library(easystats)
model_parameters(fit1)
## One Sample t-test
## 
## Parameter  | Mean | mu | Difference |       95% CI | t(19) |     p
## ------------------------------------------------------------------
## sugar_null | 3.09 |  3 |       0.09 | [2.96, 3.22] |  1.42 | 0.171
## 
## Alternative hypothesis: true mean is not equal to 3

Interpreting the results

One Sample t-test

Parameter  | Mean | mu | Difference |       95% CI | t(19) |     p
------------------------------------------------------------------
sugar_null | 3.09 |  3 |       0.09 | [2.96, 3.22] |  1.42 | 0.171

Alternative hypothesis: true mean is not equal to 3

“The mean sugar content per serving in our sample was 3.09 g, 95% CI: [2.96, 3.22], which was not significantly different from the 3 g stated on the label, \(t(19)=1.42\), \(p=.171\). Thus, we retain the null hypothesis that there are 3 g of sugar per serving.”

Putting it all together… again

• Let’s repeat with the other sample

fit2 <- t.test(sugar_alt, mu = 3)

model_parameters(fit2)
## One Sample t-test
## 
## Parameter | Mean | mu | Difference |       95% CI | t(19) |      p
## ------------------------------------------------------------------
## sugar_alt | 4.02 |  3 |       1.02 | [3.65, 4.39] |  5.70 | < .001
## 
## Alternative hypothesis: true mean is not equal to 3

Interpreting the results

One Sample t-test

Parameter | Mean | mu | Difference |       95% CI | t(19) |      p
------------------------------------------------------------------
sugar_alt | 4.02 |  3 |       1.02 | [3.65, 4.39] |  5.70 | < .001

Alternative hypothesis: true mean is not equal to 3

“The mean sugar content per serving in our sample was 4.02 g, 95% CI: [3.65, 4.39], which was significantly greater than the 3 g stated on the label, \(t(19)=5.70\), \(p<.001\). Thus, we reject the null hypothesis that there are 3 g of sugar per serving.”