One sample Z-tests

Formulas for power

We want to make sure that we actually reject our null hypothesis in the case that it is false. That is, we would like to make sure that our test has high power. Mathematically, this means that \(P(\text{reject } H_0 | H_0 \text{ is false})\). The problem here is that \(H_0\) can be wrong in many different ways: it could be that the true \(\mu\) is \(2\), it could be \(7\), it could be \(4.26\). So to calculate power as formulated above is not really possible. However, we can calculate power for specific versions of “\(H_0\) is false”.

Let’s consider the case that \(H_0\) is false, and in particular the true value of \(\mu\) is \(\mu_A\). In this case, the power of our test is \(P(\text{reject } H_0 | \mu = \mu_A)\). Recall that we reject \(H_0\) when \(\bar{x} > \mu_0 + z_{1 - \alpha / 2} \frac{\sigma}{\sqrt n}\) or \(\bar{x} < \mu_0 + z_{\alpha / 2} \frac{\sigma}{\sqrt n}\), so the power of our test when \(\mu = \mu_A\) is

\[ P(\text{reject } H_0 \left | \mu = \mu_A \right .) = P\left( \left . \bar x > \mu_0 + z_{1 - \alpha / 2} \frac{\sigma}{\sqrt n} \right | \mu = \mu_A \right) + P\left( \left . \bar x < \mu_0 + z_{\alpha / 2} \frac{\sigma}{\sqrt n} \right | \mu = \mu_A \right). \]

Remember that \(\bar X \sim \mathrm{Normal} \left(\mu, \frac{\sigma^2}{n} \right)\). This means that, given \(\mu = \mu_A\), \(\bar{x} \sim \mathrm{Normal} \left(\mu, \frac{\sigma^2}{n} \right)\), which let’s us calculate the probabilities we need to find the power:

\[ \begin{align} P\left( \left . \bar{x} > \mu_0 + z_{1 - \alpha / 2} \frac{\sigma}{\sqrt n} \right | \mu = \mu_A \right) &= P\left( \frac{\bar{x} - \mu_A}{\sigma/\sqrt{n}} > \frac{\mu_0 + z_{1-\alpha/2}\frac{\sigma}{\sqrt{n}} - \mu_A}{\sigma/\sqrt{n}} \right) \\ &= P\left( Z > \frac{\mu_0 - \mu_A}{\sigma/\sqrt{n}} + z_{1-\alpha/2} \right), \end{align} \]

and similarly

\[ P\left( \left . \bar{x} < \mu_0 + z_{\alpha / 2} \frac{\sigma}{\sqrt n} \right | \mu = \mu_A\right) = P\left( Z < \frac{\mu_0 - \mu_A}{\sigma/\sqrt{n}} + z_{\alpha/2} \right). \]

So, the power of our test, if the true population mean is \(\mu_A\), is

\[ \begin{align} \text{Power} &= P\left( Z > \frac{\mu_0 - \mu_A}{\sigma/\sqrt{n}} + z_{1-\alpha/2} \right) + P\left( Z < \frac{\mu_0 - \mu_A}{\sigma/\sqrt{n}} + z_{\alpha/2} \right) \\ &= \left[1 - P\left( Z < \frac{\mu_0 - \mu_A}{\sigma/\sqrt{n}} + z_{1-\alpha/2} \right)\right] + P\left( Z < \frac{\mu_0 - \mu_A}{\sigma/\sqrt{n}} + z_{\alpha/2} \right) \end{align} \]

Let’s calculate this if \(\mu_A = 5\).

power_lower <- (3 - 5) / (2 / sqrt(10)) + quantile(Z, 0.025)
power_upper <- (3 - 5) / (2 / sqrt(10)) + quantile(Z, 1 - 0.025)

cdf(Z, power_lower) + (1 - cdf(Z, power_upper))
#> [1] 0.8853791

This means that the probability that we reject the null hypothesis (\(H_0: \mu = 3\)) if the true mean is \(5\) is about 0.89.

Formulas for sample size calculations

Often times researchers like to go in the other direction: aim for a specific level of power, and calculate how many observations are needed to reach that level. To achieve a power of \(1 - \beta\) for a one sample Z-test with \(H_0 : \mu = \mu_0\), you need

\[ n \approx \left( { \sigma \cdot (z_{\alpha / 2} + z_\beta) \over \mu_0 - \mu_A } \right)^2 \]

samples. If \(n\) is not an integer, round up. Often, the denominator is thought of as the detectable difference. So, the question becomes how many samples are required to have sufficient power to detect a difference of some particular size.

This equation is simply a rewrite of the equation presented above for power. Recall, the power for a two sided test is

\[ \text{Power} = P\left( Z > \frac{\mu_0 - \mu_A}{\sigma/\sqrt{n}} + z_{1-\alpha/2} \right) + P\left( Z < \frac{\mu_0 - \mu_A}{\sigma/\sqrt{n}} + z_{\alpha/2} \right). \]

Usually, only one of these terms is contributing while the other is very close to zero. Let’s say the first term is the one clearly different from zero. To determine the sample size, we want to determine \(n\) such that \(P\left( Z > \frac{\mu_0 - \mu_A}{\sigma/\sqrt{n}} + z_{1-\alpha/2} \right) = 1-\beta\). Or, similarly, \(P\left( Z < \frac{\mu_0 - \mu_A}{\sigma/\sqrt{n}} + z_{1-\alpha/2} \right) = \beta\). I.e. we need \(\frac{\mu_0 - \mu_A}{\sigma/\sqrt{n}} + z_{1-\alpha/2} = z_{\beta}\).

\[ \begin{align} z_{\beta} = \frac{\mu_0 - \mu_A}{\sigma/\sqrt{n}} + z_{1-\alpha/2} & \Longleftrightarrow z_{\beta} - z_{1-\alpha/2} = \frac{\mu_0 - \mu_A}{\sigma/\sqrt{n}} \\ & \Longleftrightarrow \sigma (z_{\beta} - z_{1-\alpha/2}) = (\mu_0-\mu_A)\sqrt{n} \\ & \Longleftrightarrow \frac{\sigma (z_{\beta} - z_{1-\alpha/2})}{\mu_0-\mu_A} = \sqrt{n} \\ & \Longleftrightarrow \left(\frac{\sigma (z_{\beta} - z_{1-\alpha/2})}{\mu_0-\mu_A}\right)^2 = n \end{align} \]

Since \(z_{1-\alpha/2} = -z_{\alpha/2}\), we have the equation above:

\[ \left(\frac{\sigma (z_{\beta} + z_{\alpha/2})}{\mu_0-\mu_A}\right)^2 = n. \]

As an example, say the student prior to the experiment had determined that they wanted to test if the number of memes their professors know and love is 2. They want to make sure their sample size is large enough so that they are likely to reject the null hypothesis if the true number is 3. They determine that they want a probability of 0.9 of rejecting the null if the true number is 3. So, a sample size calculation looks like this:

\[ n \approx \left( { 2 \cdot (1.96 + 1.28) \over 2-3 } \right)^2 = 41.99 \]

So to make sure that they reject the null hypothesis with a probability of 0.9 if the true value is 3, they would have to ask 53 professors.

Below is this same calculation done in R. Remember, \(\beta = 1-\text{Power}\). Note the small discrepancy. This is due to rounding error.

(2 * (quantile(Z, 0.05 / 2) + quantile(Z, 1 - 0.9)) / (3 - 2))^2
#> [1] 42.02969