Access this content offline
Omniguides by NOX
Skip to the reading

4.3 Introduction to Significance Testing

Section Overview

  • Gallery 4.2.1 - 1 & 2 sided tests for 1 Prop.
  • Example 4.2.1
  • Gallery 4.2.2 - 1 Samp. z-test on Calc
  • Interactive 4.2.1 - Example 4.2.1
  • Pop Quiz 4.2.1
Source gap (S0371): Source numbering note: this section’s overview and prose references use 4.2.x, while gallery, interactive, and quiz widget titles use 4.3.x. Example 4.2.1 appears under that number throughout. The original labels are retained.

Confidence Intervals vs Significance Tests

The second type of statistical inference we’ll be covering in this chapter is significance testing. Confidence interval and significance tests are similar in a few ways, such as needing similar conditions in order to conduct the inference technique as shown in Chapter 4.1. Whereas the purpose of the confidence interval is to estimate the population parameter based on a random sample, the purpose of the significance test is to assess the probability that a claim made about the population parameter is true based on a random sample. As we will learn in the coming sections, these two techniques complement one another very nicely.

Stating Hypotheses

The first thing we do when performing a significance test is to state a claim that we try to find evidence against. This claim is also known as the null hypothesis. It is usually a statement of “no difference” and is abbreviated as H0H_0.

When we try to find evidence against the null hypothesis, we represent this with the alternative hypothesis. It claims that the null hypothesis is incorrect – that the population parameter in question is different than what is claimed in the null hypothesis and is abbreviated as HaH_a.

When we say in the alternative hypothesis that the population parameter is different than what is stated by the null hypothesis, we can express this is three different ways. We can either say that the true population parameter is:

  • Less than the population parameter stated by the null hypothesis,
  • Greater than the population parameter stated by the null hypothesis, or
  • Less than or greater than the population parameter stated by the null hypothesis

The first two are used in one-sided tests and the last one is used in two-sided tests.

Test Statistics

It is difficult to understand one and two-sided tests without first understanding what a test statistic is. A test statistic uses the data acquired from taking a random sample and, much like the z-score, comes up with a value that has a corresponding probability called the p-value. One-sided tests are only concerned about the corresponding probabilities in only one of the tails of the distribution, while two-sided tests are concerned about both tails. In this section, we will be covering the test statistic that comes from conducting the one-sample z test for the sample proportion. This test statistic is called the z-statistic and is found as follows:

z=p^−p0p0(1−p0)n,wherez=\frac{\hat p-p_0}{\sqrt{\frac{p_0(1-p_0)}{n}}},\quad\text{where}
p^=sample proportion\hat p=\text{sample proportion}
n=sample sizen=\text{sample size}
p0=population proportion stated by null hypothesisp_0=\text{population proportion stated by null hypothesis}

Gallery 4.2.1 combines what we’ve learned so far about one and two-sided tests and test statistics. The one and two-sided tests being represented are for the sample proportion.

Captured alternative hypotheses in frame order:

Ha:p>p0;Ha:p<p0;Ha:p≠p0H_a:p>p_0;\qquad H_a:p<p_0;\qquad H_a:p\ne p_0

The two-sided plot marks −∣z∣-|z| and ∣z∣|z|.

Statistical Significance

We can compare the p-value we calculated with the the value of α\alpha, also called the significance level. This value serves as the threshold to which the p-value is compared to. If the p-value is as small or smaller than α\alpha, we say that the data is statistically significant at level α\alpha. For instance, if the p-value for a significance test is 0.021, we can reject the null hypothesis if α=0.05\alpha=0.05, but fail to reject it if α=0.01\alpha=0.01. Similar to what we learned in Chapter 4.1, three of the most common significance levels are α=0.10\alpha=0.10, α=0.05\alpha=0.05, and α=0.01\alpha=0.01.

The significance level we choose depends on the situation. For instance, if a pharmaceutical company wanted to assert that a treatment that it spent billions of dollars in research and development on was more effective than a placebo, the company might want to use a smaller value of α\alpha to more effectively prove its effectiveness.

We’ve talked a bit about the mechanics of the p-value, but what is it exactly? It’s the probability that we would come up with a value at least as extreme as the one we obtained in our sample if the null hypothesis were true. The smaller the p-value, the less believable it is that the null hypothesis is actually true.

Example 4.2.1

Example 4.2.1

From Example 4.1.1, the company Will it Mend? discovered through random sampling that they have an 87% satisfaction rate among their customer base. The company wants to round this number up to 90%. Is there sufficient evidence at the α=0.05\alpha=0.05 level to suggest that the population proportion is actually different than p=0.90p=0.90?

Recorded answer to Example 4.2.1

Before starting the problem, we want to make sure that the three conditions for a e are met: Random, Normal, and Independent.

However, since we proved that the one-sample z interval in Example 4.1.1 met all three conditions, we can omit this step.

Since we want to know if the proportion of satisfied customers is different than 0.90 (not exclusively greater than or less than), we use a two-sided test:

H0:p=p0,Ha:p≠p0H_0:p=p_0,\quad H_a:p\ne p_0

Using the formula for the one-sample z-test, we get:

z=p^−p0p0(1−p0)nz=\frac{\hat p-p_0}{\sqrt{\frac{p_0(1-p_0)}{n}}}
z=0.87−0.900.90(0.10)1537=−3.92z=\frac{0.87-0.90}{\sqrt{\frac{0.90(0.10)}{1537}}}=-3.92

Refer to Gallery 4.2.2 for how to perform these steps on your calculator. Note that the value from the calculator (z=−3.937z=-3.937) will be slightly different than the value we derived manually (z=−3.92z=-3.92). Due to the relative ease of use of the calculator, we will use z=−3.92z=-3.92, which corresponds to a p-value of 4.13⋅1054.13\cdot10^5

Since the p-value is much smaller than the significance level (4.13⋅105≪0.054.13\cdot10^5\ll0.05), we can reject the null hypothesis and conclude that the odds of getting a sample where the customer satisfaction rate is 87% when the true satisfaction rate is actually 90% are astronomically low. It would be highly unethical if Will it Mend? decided to round their satisfaction rate up to 90%, given the evidence against this claim.

Our decision to reject the null hypothesis is also supported by the confidence intervals that we calculated in Example 4.1.1. None of them contain the population parameter p=0.90p=0.90. We will further explore this relationship in the sections to come.

Source gap (S0376): Source wording note: the recorded answer begins “the three conditions for a e are met”; no missing words have been supplied.
Source gap (S0380): Source calculation note: the prose gives the calculator value z = −3.937, then says it will use z = −3.92 because of the calculator’s ease of use. Both printed p-values have exponent positive 5, while the captured calculator screen below displays 4.1337207E−5. The prose, exponents, and comparison are retained as captured.
Source gap (S0378): Source input/result note: the captured input screen highlights p < p₀, while the following result screen in S0379 labels the test “prop ≠ .9.” The highlighted input and the two-sided result label are both retained as captured.
Captured calculator inputs and results for Example 4.2.1
ScreenFieldCaptured value
Inputp0p_00.9
Inputxx1337
Inputnn1537
InputSelected alternativep<p0p<p_0
Resultzz−3.936613727
Resultp-value4.1337207⋅10−54.1337207\cdot10^{-5}
Resultp^\hat p0.8698763826
Resultnn1537

Interactive 4.3.1 Example 4.2.1

Captured annotation overview. The vertical axis is Probability, and the visible left tail of a curve rises toward the right edge where the plot is clipped. A label z = −3.934 lies near the left end of the horizontal axis. Callouts identify z-statistic of Interest and p-value of interest, with leaders to the left-tail location. The p-value callout is cut off at its right edge. Open full-size image S0381

Captured labels: Probability; z-statistic of Interest; p-value of interest; z=−3.934z=-3.934.

Source gap (S0381): Only the annotation overview is captured. The right side of the plot and callout is clipped in the source, and expanded callout explanations are absent. Its z = −3.934 label differs from the values printed in the worked answer.

Pop Quiz 4.3.1

Question 1 of 5

The p-value is:

Question 2 of 5

An archer claims to hit his target on 80% of his shots. In a random sample of 50 shots he shoots the target 32 times. What can be concluded about the archer’s claim?

Question 3 of 5

In a survey among graduate students in the XYZ University, out of 200 students, 120 said that they had never drunk alcohol. What is the 95% confidence interval for this proportion?

Question 4 of 5

Considering the scenario from question 3, the conclusion is:

Question 5 of 5

Considering the scenario from question 3, a state survey was also conducted. The state result shows that 50% of graduate students have never drunk alcohol. The hypotheses to be tested are:

Source gap (S0382): All five quiz prompts and choices are captured in S0382–S0386. None displays a checked answer or answer feedback. The words “shoots” and “change” in Question 2, and the hypothesis values in Question 5, are retained from the original.
Glossary and deeper reading · 10 refreshers

These editorial refreshers supplement the reading. Definitions and examples are added guidance; original source wording and recorded answers remain in their original context.

  • Statistical inference

    Statistical inference uses data and a study design to draw conclusions beyond the observed values. Random sampling supports population generalization, while well-conducted random assignment supports treatment-effect conclusions. For example, a random sample can estimate a school-wide preference without surveying every student.

    Read more: Statistical inference in the captured Statistics reading →

  • Confidence interval

    A confidence interval is a data-based range produced by a method with a specified long-run coverage rate for a population parameter. For a fixed population, the calculated interval either contains the parameter or it does not; the confidence level describes the method over repeated samples. For example, a 95% method captures the true parameter in about 95% of repeated intervals when its assumptions hold.

    Read more: Confidence interval in the captured Statistics reading →

  • Null hypothesis

    A null hypothesis specifies the reference claim or model used to calculate a test’s probabilities. It commonly states no population difference or a particular parameter value. For example, a null hypothesis can state that the population mean travel time equals 15 minutes.

  • Alternative hypothesis

    An alternative hypothesis specifies the population difference or direction for which a test seeks evidence. It should be chosen from the research question before inspecting results. For example, an alternative can state that the population mean travel time is greater than 15 minutes.

  • Population parameter

    A population parameter is a numerical feature of the population or probability model being studied. It is fixed for that defined population, although its value may be unknown. For example, the true proportion of all enrolled students who walk to school is a parameter.

    Read more: Population parameter in the captured Statistics reading →

  • P-value

    A p-value is the probability, under the null model and its assumptions, of results at least as extreme as those observed in the direction or directions specified by the test. It does not give the probability that the null hypothesis is true or measure the practical size of an effect. For example, a p-value of 0.03 describes a 3% tail probability under the null model, rather than a 3% probability that the null is correct.

  • Significance level

    A significance level is a threshold selected before testing to limit Type I errors under the null model. A p-value at or below that threshold leads to rejection under the stated rule, while a larger p-value leads to failure to reject. For example, an alpha level of 0.01 demands stronger incompatibility with the null model than a level of 0.05.

  • Treatment

    A treatment is a specific condition imposed on an experimental unit. In a multifactor experiment, a treatment is a combination of factor levels. For example, low fertilizer with eight hours of light is one treatment combination.

    Read more: Treatment in the captured Statistics reading →

  • Placebo

    A placebo resembles an active treatment but lacks the active intervention under study. It helps provide a comparison for effects associated with receiving treatment and expectations. For example, an inactive pill can be made to look like the experimental pill.

    Read more: Placebo in the captured Statistics reading →

  • Probability

    Probability quantifies how likely an event is under a specified chance model. It lies between 0 and 1, with larger values representing greater likelihood. For example, drawing a red card from a well-shuffled standard deck has probability one half.

    Read more: Probability in the captured Statistics reading →

Search lessons, terms and problems