Hypothesis Testing

In the classical setup we have a null hypothesis $H_0$, some default position, and an alternative hypothesis $H_1$ we want to compare it with. We use a statistic to decide whether the data let us reject $H_0$.

This builds on the normal approximation to the binomial in Probability.

Is this coin fair?

Let $p$ be the probability of heads. $H_0$ is $p = 0.5$; $H_1$ is $p \neq 0.5$. We flip the coin $n$ times and count heads $X$. Each flip is a Bernoulli trial, so $X$ is $\text{Binomial}(n, p)$, which we approximate with a normal:

def normal_approximation_to_binomial(n: int, p: float) -> tuple[float, float]:
    mu = p * n
    sigma = math.sqrt(p * (1 - p) * n)
    return mu, sigma
 
mu_0, sigma_0 = normal_approximation_to_binomial(1000, 0.5)  # (500.0, 15.81)

With helpers built from normal_cdf and its inverse (normal_probability_between, normal_upper_bound and so on) we can find the symmetric interval around the mean that contains 95% of the probability:

lower_bound, upper_bound = normal_two_sided_bounds(0.95, mu_0, sigma_0)  # (469.0, 531.0)

Significance

The significance of a test is how willing we are to make a type 1 error (a false positive): rejecting $H_0$ when it is true. It is conventionally set at 5% or 1%. With 5%, we reject $H_0$ if $X$ falls outside 469 to 531. If the coin is fair there is only a 5% chance of that, so about 19 times out of 20 the test gives the right answer.

Power

The power of a test is the probability of not making a type 2 error (a false negative): failing to reject $H_0$ when it is false. To compute it we have to say what "false" means. Suppose the coin is actually slightly biased, $p = 0.55$:

mu_1, sigma_1 = normal_approximation_to_binomial(1000, 0.55)  # (550.0, 15.73)
 
# A type 2 error happens when X still lands inside the original interval
type_2_probability = normal_probability_between(lower_bound, upper_bound, mu_1, sigma_1)
power = 1 - type_2_probability  # 0.887

One-sided tests

If instead $H_0$ is "the coin is not biased towards heads" ($p \leq 0.5$), we only want to reject when $X$ is much larger than 500. A 5% one-sided test puts all 5% in the upper tail:

hi = normal_upper_bound(0.95, mu_0, sigma_0)  # 526.0, below 531 because the whole 5% is in one tail
power = 1 - normal_probability_below(hi, mu_1, sigma_1)  # 0.936

This test is more powerful against $p = 0.55$. It no longer rejects when $X$ is below 469, which is very unlikely if $H_1$ is true, and instead rejects when $X$ is between 526 and 531, which is quite likely if $H_1$ is true.

p-values

Instead of fixing bounds in advance, we can compute the probability, assuming $H_0$ is true, of a result at least as extreme as the one we saw:

def two_sided_p_value(x: float, mu: float = 0, sigma: float = 1) -> float:
    if x >= mu:
        return 2 * normal_probability_above(x, mu, sigma)
    return 2 * normal_probability_below(x, mu, sigma)
 
two_sided_p_value(529.5, mu_0, sigma_0)  # 0.062: 530 heads, don't reject at 5%
two_sided_p_value(531.5, mu_0, sigma_0)  # 0.046: 532 heads, reject

Make sure the data are roughly normal before computing p-values this way. Claims that an event has a one-in-a-million chance often mean "one in a million if the data are normally distributed", which says little when they aren't. There are formal tests for normality, but plotting the data is a good start.

Continuity correction

We used 529.5 rather than 530 because $X$ is discrete. normal_probability_between(529.5, 530.5) is a better estimate of the probability of exactly 530 heads than normal_probability_between(530, 531), so normal_probability_above(529.5) is the better estimate of "at least 530 heads".

ANOVA

Analysis of variance asks whether several groups have different means. It splits the total variation into the variation between groups and the variation within them, each as a sum of squares (SS). Each SS is divided by its degrees of freedom to get a mean square (MS), and the test statistic is their ratio:

for $k$ groups and $N$ observations in total. If the group means are equal, $F$ follows an F-distribution with $(k - 1, N - k)$ degrees of freedom, and a large $F$ is evidence against $H_0$.

Adapted from Joel Grus, Data Science from Scratch (2nd edition), chapter 7.

Original notebook

Tags: Statistics