Access this content offline
Omniguides by NOX
Skip to the reading

2.2 Observational Studies and Experiments

Section Overview

Introduction to Experiments

You might hear the words “observe” and “experiment” in the same sentence. However, there is a difference between observational studies and experiments. Observational studies aim to collect data on individuals as they are. Experiments, which will be the focus of this section, aim to administer treatments to some aspect of the individuals (quantified as the explanatory variable). These individuals are called experimental units, or subjects if human. Then their reaction, quantified as the response variable, is measured.

When there are multiple explanatory variables in an experiment, each one is called a factor. A specific value of a factor is called a level. When you combine levels, you get different treatments. A representation of this can be found in Interactive 2.2.1, where an experiment was conducted to see how types of breakfast food and hours of sleep a person gets the night before affect test performance.

Type of Breakfast / Hours of Sleep5812
ProteinTreatment 1Treatment 2Treatment 3
CarbTreatment 4Treatment 5Treatment 6

Good Experimental Design

We’ve talked about experiments, but do we actually know what constitutes good experimental design? If an experiment is well-designed, then it has the following three characteristics:

  1. Control - The simplest type of control is comparison, or ensuring that the only difference between two or more groups is the treatment being imposed. This balances out the effects of the independent variable being confounded with the effects of the lurking variable. Sometimes this means adding a control group, a baseline by which to compare the treatments. The control group usually gets the inactive treatment. However, the control group can also be receiving a treatment that has been proven to work in the past, such as a placebo.
  2. Random Assignment - Randomly assigning subjects to treatments can also eliminate lurking variables that we haven’t thought of or can’t easily control. It ensures that the only difference between groups is chance variation. Control and randomization work together to prevent any bias in our experiments.
  3. Replication – When we use enough experimental units in our experiment, we greatly reduce chance variation within our groups so we are better able to detect differences between our treatments.

There are other ways to prevent bias and confounding. These include:

Blinding - Single-blind experiments are when one party knows which treatment each group is receiving (usually the doctor or researcher). Double-blind experiments are when neither party knows. Experiments should have at least single blinding, but it’s important to keep in mind that doctors and researchers might react differently to patients when they know which group is getting which treatment (like paying less attention to individuals in the control group because they already know if the treatment works or not).

Placebos – Giving a placebo, or fake treatment, to a control group can help prevent confounding due to the placebo effect, a psychological effect where individuals show a positive physical response to the treatment because of a believe that the treatment works.

Source gap (S0160): Source inconsistency: the Control paragraph calls a placebo an example of “a treatment that has been proven to work in the past,” while the source’s Placebos paragraph defines it as a fake treatment. The original wording is retained.

Types of Experimental Designs

One more concept we must cover is the types of designs one can choose from when setting up an experiment, and when to use each one:

  1. Completely Randomized Design – When a completely randomized design has been implemented, it means that the experimental units have been randomly assigned to groups at the start of the experiment. Randomization will help balance the effects of the lurking variables.
  2. Blocked Design – Researchers implement a blocked design when they notice a lurking variable that could make it difficult to detect a difference between treatments, even when experimental units are randomly assigned to groups.
  3. Matched Pairs Design – A subset of the blocked design. It is implemented when differences between treatments can be hard to detect with either a completely randomized or a blocked design proper. Both treatments are assigned in a randomly determined order to a single experimental unit, or each treatment is assigned a pair of two similar experimental units.

Imagine the following scenarios. Which experimental design should be implemented for each one?

  • An online review site compares two selected brands of stock-picking software to see if either one is more effective than using “intuition” alone.
  • A university lets students pick between a traditionally-taught College Algebra course and an online College Algebra course to figure out which method is more effective.
  • A drug company administers a pill designed to enhance mental acuity to a group of randomly selected people to see if it actually works.
  • A tutoring company wants to know which special, sense-based learning technique they have developed is most effective. They consider that males and females might respond to each technique differently).
  • A biotech lab has developed a patch designed to end migraines faster than aspirin. They consider that the causes and characteristics of migraines are highly varied between each individual.

Interactive 2.2.2 is based on the first scenario and illustrates the importance of taking lurking variables into account when choosing an experimental design and setting up your experiment Although it is quite comprehensive, it will help you better understand similar graphs in Gallery 2.2.1 also based on the aforementioned scenarios. Gallery 2.2.2 (yes, also based on the scenarios) will also help you understand the general steps for the different types of experimental design.

Source gap (S0174): Source labeling inconsistency: Gallery 2.2.1 frame 4 still labels its placebo control “No Pill Group,” although its caption says that group takes a placebo. The original label is preserved.
Source gap (S0176): The captured sixth frame of Gallery 2.2.1 clips the right edge of the female olfactory bar and parts of its labels. The visible figure and full recorded caption are preserved.

The source’s design-flow labels and counts are transcribed here:

FrameStarting groupRandom assignmentTreatments / comparison
1800 studentsUpper: Group 1, 400 students. Lower: Group 1, 400 students.Treatment 1 (Traditional course); Treatment 2 (Online course); Compare performance
21200 Paid subscribers to Tutoring Company: Men (n=615n=615), Women (n=585n=585)Men: Group 1, Group 2, Group 3, each n=305n=305. Women: Group 1, Group 2, Group 3, each n=195n=195.Visual Task; Auditory Task; Olfactory Task; Compare performances within each block
3Migrane Sufferers (n=1500n=1500): Subject 1, Subject 2, Subject 3, …, Subject 1500Random Assignment for each subjectSubjects 1 and 3: Aspirin → Patch. Subjects 2 and 1500: Patch → Aspirin. Compare Treatments in each row.
Source gap (S0171): Source inconsistency: both 400-student branches in Gallery 2.2.2 frame 1 are labeled “Group 1.” The two labels are retained.
Source gap (S0177): Source inconsistency: the male block is labeled 615, but its three treatment groups are each labeled 305. The source’s counts are retained. This capture also clips parts of the leftmost starting label and rightmost comparison labels; the visible wording is transcribed above.

Example 2.2.1

Example 2.2.1

A group of scientists working in the research and development division of a fitness company have developed a revolutionary new drug, Automatic Athlete, which promises to help people gain more muscle mass during a workout than just working out alone. A rival fitness company has also put out a similar product, Beef and Brawn. Early one morning, the scientists set up shop outside one of the city’s largest gyms, where the gym members can sign up to participate in a study. There are 30 bottles of each brand on display and the members are given the choice between the two brands, although the scientists make sure their Automatic Athlete bottles are displayed in front. When the members decide, they check off their choice on the form. The scientists end up getting 28 signed up for Automatic Athlete and 19 for Beef and Brawn. Lastly, it is suspected that men and women might react differently to each brand of pill.

a) How could the scientists have prevented bias in the study?

b) What can make the results less reliable, even when bias is taken into account?

Example 2.2.1 — Recorded answer

a) Random Assignment: The members that selected Automatic Athlete because it was displayed more prominently might lead the researchers to believe their brand is more effective than it really is. Also, the members that came first might be more health-conscious than their later-day counterparts, which could contribute more to their muscle gain than just the pill alone. Random assignment would make sure that the effects of this lurking variable, overall health, would be spread out relatively evenly between the two groups.

Control: The scientists not only wanted to know that their brand was more effective than the competition, but that it was more effective than just working out alone. They can’t do this without a control group, a baseline to compare the other two treatment groups to. They would want to give the control group a placebo rather than nothing at all because they don’t know how much (1) the effect of the brand and (2) the psychological effect of simply taking pills each individually contributes to the gain in muscle mass. In other words, the effects are confounded.

Blinding: The gym members knew which treatment they were getting, and so did the scientists. Now that we have introduced a placebo group, we don’t want any treatment group knowing whether or not they got an active treatment. If there isn’t going to be a lot of scientist-patient interaction, then single-blind might be appropriate. Otherwise, they should go with double-blind.

Blocking: It was stated earlier that men and women are suspected to have different reactions to each of the brands, but the scientists chose not to react upon it. This could make it more difficult to distinguish a difference between the effects of the two treatments. For instance, perhaps females gain more on Automatic Athlete and males gain more on Beefy Brawn. This could be solved by a blocked design, where males and females are purposely separated into two groups, with the individuals in each groups being randomly assigned to the placebo, Automatic Athlete, or Beefy Brawn treatments.

b) Replication: A grand total of 47 subjects seems like a rather small number, especially for a product of research that likely had an investment of millions of dollars put into it. If the scientists could gather hundreds, or preferably even thousands of subjects for each group, then the effects of the pills could be distinguished from chance variation between the groups.

Source gap (S0180): Source naming inconsistency: the prompt names the rival product “Beef and Brawn”; the recorded Blocking answer calls it “Beefy Brawn.” Both original names are retained.

Pop Quiz 2.2.1

Question 1 of 5

Which of the following defines an observational study?

Question 2 of 5

In a completely randomized experimental study, the randomization tends to:

Question 3 of 5

Four types of online ads are created for a campaign: (1) A text ad on the side of a webpage, (2) A text ad on the top of a webpage (3) An image-based ad on the side of a webpage (4) An image-based ad on the top of a webpage. This experiment:

Question 4 of 5

The two most important features of a well designed experiment include:

Question 5 of 5

If 300 subjects are numbered and 165 are randomly assigned to a treatment after being divided into two groups, this is:

Source gap (S0181): The captures of Pop Quiz 2.2.1 show all five questions and their choices, but no checked answers or answer feedback.
Glossary and deeper reading · 13 refreshers

These editorial refreshers supplement the reading. Definitions and examples are added guidance; original source wording and recorded answers remain in their original context.

  • Observational study

    An observational study measures individuals without assigning the treatments or conditions being compared. It can reveal associations, but confounding can prevent a causal conclusion. For example, recording students’ existing sleep habits and scores is observational when researchers do not assign sleep schedules.

  • Treatment

    A treatment is a specific condition imposed on an experimental unit. In a multifactor experiment, a treatment is a combination of factor levels. For example, low fertilizer with eight hours of light is one treatment combination.

  • Explanatory variable

    An explanatory variable is used to predict or explain variation in a response variable. In an experiment, researchers deliberately assign values or treatments related to an explanatory variable. For example, assigned study time could be an explanatory variable when test score is the response.

    Read more: Explanatory variable in the captured Statistics reading →

  • Experimental factor

    A factor is an explanatory variable deliberately varied in an experiment. Each assigned setting of that factor is called a level. For example, fertilizer type is a factor with levels corresponding to the fertilizer choices.

  • Confounding

    Confounding occurs when the effects of different variables cannot be distinguished in the observed comparison. A potential explanation is that another variable is related to both the treatment or explanatory variable and the response. For example, comparing fertilizer A outdoors with fertilizer B indoors mixes fertilizer and growing-location effects.

  • Lurking variable

    A lurking variable is an unmeasured or unaccounted-for variable that helps explain an observed relationship. It may create confounding when it is related to both variables being studied. For example, prior preparation might influence both time spent studying and test score.

  • Statistical bias

    Statistical bias is a systematic tendency to favor some outcomes or miss a target value. For an estimator, bias is its long-run average estimate minus the parameter it estimates. For example, a survey that consistently misses evening workers may systematically misrepresent commuting patterns.

    Read more: Statistical bias in the captured Statistics reading →

  • Replication

    Replication uses multiple independent experimental units for each treatment. It helps estimate chance variation and increases the information available for comparing treatment responses. For example, assigning many independently treated plants to each fertilizer is replication, while repeatedly measuring one plant adds a different kind of information.

  • Experimental unit

    An experimental unit is the smallest unit independently assigned a treatment. A human experimental unit is often called a subject or participant. For example, if fertilizer is assigned to an entire planter, the planter rather than each leaf is the experimental unit.

  • Blinding

    Blinding keeps specified participants or study personnel unaware of assigned treatments. The study should state who is blinded because labels such as single-blind and double-blind can be used differently. For example, both participants and outcome assessors can be kept unaware of which pill was assigned.

  • Control group

    A control group supplies a comparison for evaluating an experimental treatment. It may receive a placebo, no active treatment, or an established treatment, depending on the study’s design. For example, a fertilizer experiment may compare a new fertilizer with the usual fertilizer rather than with no fertilizer.

  • Placebo

    A placebo resembles an active treatment but lacks the active intervention under study. It helps provide a comparison for effects associated with receiving treatment and expectations. For example, an inactive pill can be made to look like the experimental pill.

  • Placebo effect

    A placebo effect is a response associated with expectations or the experience of receiving a treatment. Observed change in a placebo group can also include natural recovery or other influences, so it is not all automatically a placebo effect. For example, reported discomfort may change after a participant receives an inactive pill.

Search lessons, terms and problems