R: R Hypothesis Testing

Q: Unilateral or bilateral?

Bilateral by default ( alternative = "two.sided" ). Use unilateral only when there is a clear direction (e.g., for "the new drug is superior to the old drug," use greater ).

Q: Does a p-value of less than 0.05 indicate a large difference?

No! A p-value of less than 0.05 only indicates that the result is “unlikely to be due to chance,” but the magnitude of the difference depends on the effect size. Even with a sample size of 1,000,000, a p-value of less than 0.001 might still indicate a very small difference.

Q: Why use ANOVA instead of multiple t-tests when there are three or more groups?

Multiple t-tests can lead to an accumulation of Type I errors (where the p-value gets larger and larger). ANOVA performs a single test, keeping the error rate at 0.05.

Q: What are the requirements for a chi-square test?

The expected frequency in each cell must be ≥ 5

Last updated: 2026-08-26

In the last lesson, we learned about probability distributions—the theory. In this lesson, we’ll dive into hypothesis testing—using samples to draw conclusions about the population. This is the core of statistics: Is an ad claiming “a 20% increase in sales” true or just hype? Is the difference between two sets of data due to chance or is it inevitable? R can answer this with just 4 lines of code.

After completing this lesson, you will be able to use R to perform: t-tests (comparison of means), proportion tests (comparison of pass rates), chi-square tests (independence), and analysis of variance (comparison of multiple groups).

1. What You'll Learn

5-Step Hypothesis Testing Process (Null Hypothesis/Alternative Hypothesis/Test Statistic/p-Value/Decision)
t.test: One-sample, Two-sample, Paired
prop.test Proportionality Test
chisq.test Chi-square test (independence/goodness of fit)
ANOVA (Analysis of Variance)
Nonparametric test: wilcox.test
Interpreting p-Values and Common Pitfalls

2. A Story About an A/B Test

(1) Pain Point: Do ads work?

Bob conducted an A/B test:

Group A (old ad): 1,000 people, 80 conversions (8%)
Group B (New Ads): 1,000 people, 100 conversions (10%)

The manager asked, "Is the 2% difference real or just a fluke?"

(2) Solution using R

# Two-Proportion Test
prop.test(c(80, 100), c(1000, 1000))
# 2-sample test for equality of proportions
# X-squared = 2.46, df = 1, p-value = 0.117
# Conclusion: p > 0.05, Differences **Not significant** (It might be a coincidence.)

1 line of code → p-value → decision.

3. The 5-Step Process for Hypothesis Testing

(1) 5-Step Process

graph TB
    A[1. Propose a hypothesis] --> B[2. Calculate the statistic]
    B --> C[3. Calculation p-value]
    C --> D[4. Decision-making]
    D --> E[5. Report]
    
    A --> A1["H0 Null Hypothesis<br/>H1 Alternative Hypothesis"]
    B --> B1["t/z/Chi-square/F"]
    C --> C1["P Data Visualization H0 An equally extreme probability"]
    D --> D1["p < 0.05 → Reject H0"]
    E --> E1["Report p-value + Effect size + Confidence Interval"]
    
    style A fill:#fff3cd
    style B fill:#cce5ff
    style C fill:#d4edda
    style D fill:#f8d7da
    style E fill:#e1d4ff

(2) Key Concepts

Concept	Meaning
H0 (null hypothesis)	Default assumption: The two groups are indistinguishable (difference = chance)
H1 (Alternative Hypothesis)	What we really want to prove: that the two groups differ
Statistic	A number that measures "how large the difference is" (t, z, χ², F)
p-value	The probability of observing the current or a more extreme result given that H0 is true
α (significance level)	The threshold for the p-value; commonly 0.05
Reject H0	p < 0.05, the difference is statistically significant
Do not reject H0	p ≥ 0.05; it cannot be concluded that there is a difference (≠ no difference)

⚠️ Key point: "Not rejecting H0 ≠ accepting H0." A p-value greater than 0.05 simply indicates "insufficient evidence"; it does not mean that "the two groups are truly equal."

4. t.test(): Comparing Means

(1) Three Types of t-Tests

Type	Scenario	R Syntax
Single Sample	1 group vs. known value	`t.test(x, mu = 0)`
Two Independent Samples	2 independent samples	`t.test(x, y)`
Paired Samples	Pre- and post-tests for the same subject	`t.test(x, y, paired = TRUE)`

(2) One-sample t-test

# Example: Product Weight Standard 100g, test 10 products
weights <- c(98, 102, 99, 101, 100, 97, 103, 99, 101, 100)

t.test(weights, mu = 100)
# One Sample t-test
# t = 0, df = 9, p-value = 1
# 95% CI: [98.7, 101.3]
# mean of x = 100
# Conclusion: p = 1, **Don't reject H0**, the mean is equal to 100g

(3) Two-Sample Independent t-Test

# Example: A/B Testing the conversion times for two groups
group_a <- c(12, 15, 14, 13, 16, 14, 15, 13, 14, 15)
group_b <- c(10, 11, 12, 13, 11, 10, 12, 11, 13, 12)

t.test(group_a, group_b)
# Welch Two Sample t-test
# t = 5.32, df = 14.6, p-value = 0.0001
# 95% CI: [1.36, 3.24]
# Conclusion: p < 0.05, **Reject H0**, Group A was significantly slower than Group B

(4) Paired t-test

# Example: Before medication vs after medication in the same group of patients
before <- c(180, 175, 190, 170, 185)
after <- c(165, 160, 175, 160, 170)

t.test(before, after, paired = TRUE)
# Paired t-test
# t = 8.5, df = 4, p-value = 0.001
# Conclusion: p < 0.05, **Significant decline** (The medication is effective)

(5) Key Parameters

Parameter	Meaning	Default
`mu`	Known value for a single sample	0
`paired`	Is it paired?	FALSE
`var.equal`	Homoscedasticity Assumption	FALSE (Welch)
`alternative`	Alternative direction	"two.sided"
`conf.level`	Confidence Level	0.95

# One-sided test (right-tailed)
t.test(x, mu = 100, alternative = "greater")

# One-sided test (left-tailed)
t.test(x, mu = 100, alternative = "less")

5. prop.test(): Proportionality Test

(1) One-proportion test

# Example: Historical Product Pass Rate 95%, now 92 out of 100 are Qualified
prop.test(92, 100, p = 0.95)
# 1-sample proportions test
# X-squared = 1.27, df = 1, p-value = 0.26
# Conclusion: p > 0.05, Pass Rate **No significant decrease**

(2) Two-Proportion Test (Core of A/B Testing)

# Example: Group A 1000 people, 80 conversions; Group B 1000 people, 100 conversions
prop.test(c(80, 100), c(1000, 1000))
# 2-sample test for equality of proportions
# X-squared = 2.46, df = 1, p-value = 0.117
# 95% CI: [-0.046, 0.006]
# Conclusion: p > 0.05, **The difference is not significant** (It might be a coincidence.)

(3) prop.test vs chisq.test

prop.test(c(s1, s2), c(n1, n2)) = chisq.test(matrix(c(s1, n1-s1, s2, n2-s2), 2))
prop.test Continuity correction added (recommended)

6. chisq.test(): Chi-square Test

(1) 2 Main Uses

Purpose	Scenario	Formula
Goodness of Fit	Observed vs. Expected	`chisq.test(observed)`
Independence	Are the two categorical variables correlated?	`chisq.test(table(x, y))`

(2) Goodness-of-fit Test

# Example: Dice Fairness Test (toss 60 times)
observed <- c(8, 12, 10, 11, 9, 10)  # Number of times each face appears
expected <- rep(10, 6)  # Expect uniformity

chisq.test(observed, p = expected / sum(expected))
# Chi-squared test
# X-squared = 1.2, df = 5, p-value = 0.945
# Conclusion: p > 0.05, Dice **is fair** (Do not reject the uniformity assumption)

(3) Test for Independence

# Example: Gender vs Purchasing Behavior
gender <- c("M", "M", "F", "F", "M", "F", "M", "F")
purchase <- c("Buy", "Not buying", "Buy", "Buy", "Not buying", "Buy", "Buy", "Not buying")

tab <- table(gender, purchase)
print(tab)
#      Not buying Buy
#   M    1  3
#   F    1  3

chisq.test(tab)
# X-squared = 0, df = 1, p-value = 1
# Conclusion: p > 0.05, Gender and Purchasing **Independent**

# Note: Any 2x2 Table with Expected Frequencies < 5, use Fisher's Exact Test
fisher.test(tab)

(4) Expected Frequency Requirements

# Chi-Square Test Requirements: Expected Frequency for Each Cell >= 5
# Use Fisher's Exact Test when not satisfied
result <- chisq.test(tab)
result$expected  # View Expected Frequency

7. aov() Analysis of Variance (ANOVA)

(1) Applicable Scenarios

Comparison of means for 3 or more groups (the t-test is limited to 2 groups).

(2) Practical Application

# Example: Effect of 3 Fertilizer Types on Crop Yields
fertilizer <- c(rep("A", 10), rep("B", 10), rep("C", 10))
yield <- c(20, 22, 21, 19, 23, 20, 21, 22, 19, 20,   # A
           25, 26, 24, 27, 25, 26, 27, 25, 24, 26,  # B
           18, 19, 17, 18, 19, 20, 19, 18, 19, 20)  # C

df <- data.frame(fertilizer, yield)

# Single-factor ANOVA
model <- aov(yield ~ fertilizer, data = df)
summary(model)
#              Df Sum Sq Mean Sq F value Pr(>F)    
# fertilizer     2  220.0   110.0   91.7  <2e-16 ***
# Residuals     27   32.4     1.2
# Conclusion: p < 0.05, 3 Fertilizer Types **Significant difference**

# Post-hoc comparison: Tukey HSD
TukeyHSD(model)
#   diff        lwr        upr     p adj
# B-A   5.0  3.91       6.09    0.000
# C-A  -1.5 -2.59      -0.41    0.005
# C-B  -6.5 -7.59      -5.41    0.000

8. wilcox.test() Nonparametric Test

(1) When should it be used?

The data is non-normal (so a t-test cannot be used)
Small sample size (n < 30)
There are extreme outliers
Ordinal data (such as "Excellent/Good/Average/Poor")

(2) Practical Application

# Non-parametric paired test (Wilcoxon signed-rank)
before <- c(180, 175, 190, 170, 185)
after <- c(165, 160, 175, 160, 170)

wilcox.test(before, after, paired = TRUE)
# V = 15, p-value = 0.0625
# Conclusion: p > 0.05 (Not significant), however, the sample size is small

# Independent non-parametric (Mann-Whitney U)
group_a <- c(12, 15, 14, 13, 16)
group_b <- c(10, 11, 12, 13, 11)
wilcox.test(group_a, group_b)
# W = 25, p-value = 0.028
# Conclusion: p < 0.05, the difference is significant

9. Interpreting p-Values and Common Pitfalls

(1) Understanding p-Values Correctly

Interpretation	Explanation
✅ Correct	The probability of observing the current or more extreme outcome when H0 is true
❌ Error	"Probability that H0 is true"
❌ Error	"The difference is due to chance"
❌ Error	"Intensity of the difference"

(2) 5 Common Pitfalls

graph TB
    A[p Value Trap] --> B[p-hacking<br/>After trying it several times p<0.05]
    A --> C[Large sample size p Bi Xiao<br/>Depends on the effect size]
    A --> D[p<0.05 Does not mean it is useful<br/>Check effect size]
    A --> E[Don't refuse ≠ Accept H0]
    A --> F[Significance ≠ Practical Significance]
    
    style A fill:#fff3cd
    style B fill:#f8d7da
    style C fill:#cce5ff
    style D fill:#d4edda
    style E fill:#e1d4ff
    style F fill:#ffe1d4

(3) Effect Size

It’s not enough to look at the p-value alone; you need to look at the effect size—how large the difference is:

Test	Effect Size
t-test	Cohen's d = (m1 - m2) / s_pooled
ANOVA	η² (eta squared)
Chi-square	Cramer's V
Correlation	r (correlation coefficient)

# Cohen's d Calculation
cohens_d <- function(x, y) {
  n1 <- length(x)
  n2 <- length(y)
  s_pooled <- sqrt(((n1-1)*var(x) + (n2-1)*var(y)) / (n1 + n2 - 2))
  (mean(x) - mean(y)) / s_pooled
}

Cohen's d	Meaning
0.2	Small effect
0.5	Moderate effect
0.8	Large effect

10. Complete Example: A/B Testing + Multi-Group Comparison

Below is an example of a complete workflow that ties together all the hypothesis testing concepts covered in this lesson.

▶ Example: Comprehensive Evaluation of Marketing Campaign Performance

R 📖 Display only

# ============================================
# Comprehensive Evaluation of Marketing Campaign Performance
# Features: t-test / Proportionality Test / ANOVA combined
# ============================================

set.seed(42)

# 1. A/B Test Conversion Time
group_a <- rnorm(50, mean = 15, sd = 3)  # Original Page
group_b <- rnorm(50, mean = 13, sd = 3)  # New Page

cat("=== A/B Test: Conversion Time ===\n")
cat("A Group Mean:", round(mean(group_a), 2), "s\n")
cat("B Group Mean:", round(mean(group_b), 2), "s\n")
cat("Differences:", round(mean(group_a) - mean(group_b), 2), "s\n\n")

# 2. t-test
t_result <- t.test(group_a, group_b)
print(t_result)
cat("\n")

# 3. Cohen's d
n1 <- length(group_a)
n2 <- length(group_b)
s_pooled <- sqrt(((n1-1)*var(group_a) + (n2-1)*var(group_b)) / (n1+n2-2))
cohens_d <- (mean(group_a) - mean(group_b)) / s_pooled
cat("Cohen's d (Effect size):", round(cohens_d, 3), "\n")
cat("Effect Size:", ifelse(abs(cohens_d) > 0.8, "L",
                  ifelse(abs(cohens_d) > 0.5, "Mid", "S")), "\n\n")

# 4. A/B Testing Conversion Rates
convert_a <- 80
convert_b <- 100
visitors_a <- 1000
visitors_b <- 1000

cat("=== A/B Test: Conversion Rate ===\n")
cat("A Group Conversion Rate:", round(convert_a / visitors_a * 100, 2), "%\n")
cat("B Group Conversion Rate:", round(convert_b / visitors_b * 100, 2), "%\n\n")

# 5. Proportionality Test
prop_result <- prop.test(c(convert_a, convert_b),
                          c(visitors_a, visitors_b))
print(prop_result)
cat("\n")

# 6. Comparison of 3 Marketing Strategies (ANOVA)
strategy <- c(rep("Email", 20), rep("Text Message", 20), rep("Push", 20))
sales <- c(rnorm(20, 100, 15),
           rnorm(20, 110, 15),
           rnorm(20, 105, 15))
df_strategy <- data.frame(strategy, sales)

cat("=== Comparison of 3 Sales Strategies (ANOVA) ===\n")
model <- aov(sales ~ strategy, data = df_strategy)
summary(model)
cat("\nPost-hoc comparison:\n")
print(TukeyHSD(model))

# 7. Gender and Purchasing Behavior (Chi-square)
gender <- sample(c("M", "F"), 200, replace = TRUE)
purchase <- sample(c("Buy", "Not buying"), 200, replace = TRUE,
                   prob = c(0.3, 0.7))
tab <- table(gender, purchase)
cat("\n=== Gender vs Purchase Chi-Square Test ===\n")
print(tab)
cat("\n")
chisq_result <- chisq.test(tab)
print(chisq_result)

# 8. Comprehensive Report
cat("\n=== Summary of Decisions ===\n")
cat("1. Conversion Time: Group A is", round(cohens_d, 2), "standard deviations slower than Group B",
    ifelse(t_result$p.value < 0.05, " (Significant)", " (Not significant)"), "\n", sep = "")
cat("2. Conversion Rate: A vs B Difference",
    ifelse(prop_result$p.value < 0.05, "Significant", "Not significant"), "\n")
cat("3. 3 Strategies:",
    ifelse(summary(model)[[1]]$`Pr(>F)`[1] < 0.05, "Significant difference", "No difference"), "\n")

54 logic lines (exceeds 40-line limit, display only)

Expected Output (Excerpt):

TEXT 📖 Display only

=== A/B Test: Conversion Time ===
A Group Mean: 14.87 s
B Group Mean: 12.94 s
Differences: 1.93 s

Welch Two Sample t-test
t = 3.18, df = 96.3, p-value = 0.002
Cohen's d (Effect size): 0.643
Effect Size: Mid

=== Summary of Decisions ===
1. Conversion Time: Group A is 0.64 standard deviations slower than Group B (Significant)
2. Conversion Rate: A vs B The difference is not significant
3. 3 Strategies: Significant difference

❓ FAQ

Q Unilateral or bilateral?

A Bilateral by default (alternative = "two.sided"). Use unilateral only when there is a clear direction (e.g., for "the new drug is superior to the old drug," use greater).

Q Does a p-value of less than 0.05 indicate a large difference?

A No! A p-value of less than 0.05 only indicates that the result is “unlikely to be due to chance,” but the magnitude of the difference depends on the effect size. Even with a sample size of 1,000,000, a p-value of less than 0.001 might still indicate a very small difference.

Q Why use ANOVA instead of multiple t-tests when there are three or more groups?

A Multiple t-tests can lead to an accumulation of Type I errors (where the p-value gets larger and larger). ANOVA performs a single test, keeping the error rate at 0.05.

Q What are the requirements for a chi-square test?

A The expected frequency in each cell must be ≥ 5

Q What should I do if p > 0.05?

A There are three possibilities: ① There really is no difference; ② The sample size is small; ③ The effect size is small. You cannot directly accept H0; you can only say that "there is insufficient evidence."

Q When should you use the Wilcoxon test instead of the t-test?

A When the data are not normally distributed, contain extreme outliers, have a small sample size (< 30), or are ordinal. However, the Wilcoxon test has 5–10% lower power than the t-test.

📖 Summary

Hypothesis Testing in 5 Steps: H0/H1 → Test Statistic → p-Value → Decision → Reporting
t-test: t.test(x, y) Two-sample / paired = TRUE Paired / mu = One-sample
Proportional testing: prop.test(c(s1, s2), c(n1, n2)) Used for A/B testing
Chi-square test: chisq.test(table) Independence / chisq.test(observed) Goodness of fit
ANOVA: aov(y ~ group) Comparison of means across multiple groups; followed by TukeyHSD()
p-value ≠ magnitude of effect—must be used in conjunction with effect size (Cohen’s d / η²)
5 Major Pitfalls: p-hacking, large sample size leads to a small p-value, p < 0.05 ≠ clinically meaningful, failure to reject ≠ acceptance, and statistical significance ≠ practical significance
p < 0.05 is the threshold for statistical significance; practical significance depends on the effect size and the business context

📝 Exercises

Basic Problem: Simulate the scores of 30 students (mean 80, standard deviation 10) and perform a one-sample t-test to determine whether there is a significant deviation from 75 points; then simulate the scores of another 30 students (mean 78) and perform a two-sample t-test. Record the p-values.
Basic Problem: Simulate an A/B test (Group A: 1,000 people, 50 conversions; Group B: 1,000 people, 80 conversions). Use prop.test to test whether the difference is statistically significant. Calculate the Cohen’s d effect size.
Basic Exercise: Construct a 3x3 contingency table (3 advertising channels × whether a click occurred), and use chisq.test to test whether the channels and clicks are independent. Check result$expected—are all expected frequencies ≥ 5?
Advanced Exercise: Simulate the test scores for three teaching methods (20 students per group), perform an ANOVA using aov, and then use TukeyHSD to determine which two groups show the greatest difference.
Challenge: Conduct a full A/B test—simulate the conversion time and conversion rate for two pages (1,000 users each). Use t-test, prop.test, and Cohen’s d for a comprehensive evaluation, and write a decision report (including p-value, effect size, and business recommendations). Take a screenshot and save it.

Web-Tutorial Tech Team

A team of developers maintaining programming tutorials. Each tutorial is written and reviewed by developers with expertise in that field. We work to keep our content accurate and reliable — if you spot an issue, please let us know.

About Us

R: R Hypothesis Testing

1. What You'll Learn

2. A Story About an A/B Test

(1) Pain Point: Do ads work?

(2) Solution using R

3. The 5-Step Process for Hypothesis Testing

(1) 5-Step Process

(2) Key Concepts

4. t.test(): Comparing Means

(1) Three Types of t-Tests

(2) One-sample t-test

(3) Two-Sample Independent t-Test

(4) Paired t-test

(5) Key Parameters

5. prop.test(): Proportionality Test

(1) One-proportion test

(2) Two-Proportion Test (Core of A/B Testing)

(3) prop.test vs chisq.test

6. chisq.test(): Chi-square Test

(1) 2 Main Uses

(2) Goodness-of-fit Test

(3) Test for Independence

(4) Expected Frequency Requirements

7. aov() Analysis of Variance (ANOVA)

(1) Applicable Scenarios

(2) Practical Application

8. wilcox.test() Nonparametric Test

(1) When should it be used?

(2) Practical Application

9. Interpreting p-Values and Common Pitfalls

(1) Understanding p-Values Correctly

(2) 5 Common Pitfalls

(3) Effect Size

10. Complete Example: A/B Testing + Multi-Group Comparison

▶ Example: Comprehensive Evaluation of Marketing Campaign Performance

❓ FAQ

📖 Summary

📝 Exercises

🙏 帮我们做得更好