Experimental Design: Make the Comparison Fair
An experiment tries to learn what changes when a treatment changes. The challenge is that many other things change too. A useful design makes a comparison, uses chance to assign treatments, gathers enough repeated information, and keeps avoidable differences from overwhelming the treatment comparison.
Four Elements and Careful Comparisons
A well-designed experiment includes comparison, random assignment, replication, and direct control. Compare at least two treatment groups; one can be a control group. Assign treatments to experimental units using chance. Use more than one experimental unit for each treatment—this is replication. Keep the settings of relevant extraneous sources of response variation the same from unit to unit when possible—this is direct control.
A control group supplies a comparison. It may receive a different treatment, including an inactive placebo, rather than necessarily receiving nothing. For a new pain treatment, a matching inactive pill can separate effects of the active ingredient from the experience of taking a pill. The placebo effect is the difference between the average response to a placebo and the average response to no treatment. That is not the same comparison as active treatment minus placebo.
An extraneous variable is a known or suspected influence on the response that is not the explanatory variable being studied. In a seedling experiment, watering amount might affect height while the investigated treatment is fertilizer type. Direct control might give every seedling the same measured water amount. Random assignment aims to make treatment groups as similar as possible in their distributions of remaining extraneous variables. If successful, those distributions are approximately alike, reducing systematic alternative explanations.
An experimental confounder is related to the explanatory variable in a way that makes it difficult to tell whether the response change comes from the treatment or that other variable. If every fertilized seedling gets more water and every unfertilized seedling gets less, fertilizer and water are tangled. A well-designed experiment reduces that potential. Random assignment helps; direct control can remove the specific watering difference.
Blinding, also called masking, addresses expectations. In a single-blind or single-masked experiment, participants do not know their treatment while interacting research-team members do, or vice versa. In a double-blind or double-masked experiment, neither participants nor the research-team members who interact with them know the assigned treatment. A hidden label can help maintain blinding, but the definition concerns who knows, not merely whether the package looks identical.
| Element | Seedling example | Why it matters |
|---|---|---|
| Comparison | Fertilizer A versus B | There is an alternative treatment outcome to compare |
| Random assignment | Randomly allocate seedlings to A or B | Avoids systematic pre-existing group differences |
| Replication | 24 seedlings receive each fertilizer | One unusual seedling does not determine the treatment result |
| Direct control | Same water amount, pots, and measurement time | Reduces avoidable response variation |
Worked Example: Three Pain-Response Groups
Suppose average improvement is 7 points with active treatment, 3 points with placebo, and 1 point with no treatment. The placebo-versus-no-treatment difference is 2 points. The active-versus-placebo difference is 4 points. These answer different questions. Whether the differences supply convincing treatment evidence depends on the design and variation; calculating a difference alone does not supply random assignment.
Randomized Designs
In a completely randomized design, treatments are assigned to experimental units entirely by chance. With 48 seedlings, select 24 randomly for A and give B to the other 24. Equal group sizes are common and often convenient, but completely randomized treatment groups do not have to be equal in size.
A blocking variable is an extraneous source of response variation used to group similar units. In a randomized block design, form blocks using similar blocking-variable values first, then randomly assign treatments within each block. All treatments occur in every block. If initial seedling size affects later height, form small, medium, and large size blocks and randomize A/B separately inside each. Blocking separates variation associated with initial size from other extraneous variation, making within-block treatment comparisons more precise. Within a size block, a treatment contrast is less affected by large differences in initial size.
A matched pairs design is a randomized block design with exactly two treatments. Match units in pairs using relevant extraneous characteristics, then randomly assign one treatment to one member and the other treatment to the second. Each pair receives both treatments. Alternatively, the same unit can receive both treatments, with their order randomized. In the latter design, consider whether an earlier treatment could carry over into the later response; the mere presence of two measurements does not guarantee a fair comparison.
| Design | Before assignment | Random assignment step |
|---|---|---|
| Completely randomized | All available units together | Randomly allocate treatments across all units |
| Randomized block | Similar units grouped by blocking variable | Randomize treatments within every block |
| Matched pairs | Similar pairs, or one unit measured under both conditions | Randomize within pair, or randomize treatment order |
Worked Example: Two Shoe Soles
To compare sole materials A and B, match runners into similar pairs by usual weekly distance and running surface. Randomly assign A to one runner and B to the other in each pair. Alternatively, each runner can test both materials over comparable periods, with A-first versus B-first randomly assigned. The pairing controls relevant runner differences; the random assignment determines which condition goes where. A plan giving A to all high-mileage runners and B to all low-mileage runners does neither job well.
Choose the Design
No design is best merely because it sounds sophisticated. Choose in light of the investigative goal, population characteristics, available sample, and variables. Blocking on a characteristic strongly associated with the response can improve precision; blocking on a meaningless label adds complexity without addressing response variation. A same-unit matched pair can be efficient when a unit can receive both treatments fairly, but may be inappropriate when the first treatment has a lasting effect. A completely randomized design is sensible when important blocking information is unavailable or units are already fairly similar.
Causation and Generalization
Random assignment supports cause-and-effect conclusions between explanatory and response variables because it reduces potential confounding. It does not mean treatment groups will be perfectly identical by chance, and it does not select a representative population sample.
Randomly selecting people for an experiment may be unethical or impractical. Studies may instead recruit volunteers. Their results then concern the participating experimental units and may be relevant to similar units; they do not automatically represent every person in a broad population. A treatment comparison can have strong random-assignment support while having limited generalization. State both the strength and the limit in the conclusion.
Independent Practice
Use the information in each question to make your own calculation, representation, or argument. Explain what your result means in its setting.
Original generated practice for the Fall 2026 course.
Glossary and deeper reading · 17 refreshers
These editorial refreshers supplement the reading. Definitions and examples are added guidance; original source wording and recorded answers remain in their original context.
-
Experiment
An experiment deliberately imposes treatments and measures responses. Random assignment, comparison, and replication help distinguish treatment effects from other variation. For example, assigning plants to fertilizer treatments is an experiment.
-
Treatment
A treatment is a specific condition imposed on an experimental unit. In a multifactor experiment, a treatment is a combination of factor levels. For example, low fertilizer with eight hours of light is one treatment combination.
-
Random assignment
Random assignment uses chance to allocate experimental units to treatment conditions. It tends to balance other influences across groups and supports causal inference in a well-conducted experiment, without guaranteeing identical groups. For example, shuffled labels can assign plants to two fertilizer treatments.
Read more: Random assignment in the captured Statistics reading →
-
Replication
Replication uses multiple independent experimental units for each treatment. It helps estimate chance variation and increases the information available for comparing treatment responses. For example, assigning many independently treated plants to each fertilizer is replication, while repeatedly measuring one plant adds a different kind of information.
-
Control group
A control group supplies a comparison for evaluating an experimental treatment. It may receive a placebo, no active treatment, or an established treatment, depending on the study’s design. For example, a fertilizer experiment may compare a new fertilizer with the usual fertilizer rather than with no fertilizer.
Read more: Control group in the captured Statistics reading →
-
Placebo
A placebo resembles an active treatment but lacks the active intervention under study. It helps provide a comparison for effects associated with receiving treatment and expectations. For example, an inactive pill can be made to look like the experimental pill.
-
Explanatory variable
An explanatory variable is used to predict or explain variation in a response variable. In an experiment, researchers deliberately assign values or treatments related to an explanatory variable. For example, assigned study time could be an explanatory variable when test score is the response.
Read more: Explanatory variable in the captured Statistics reading →
-
Blinding
Blinding keeps specified participants or study personnel unaware of assigned treatments. The study should state who is blinded because labels such as single-blind and double-blind can be used differently. For example, both participants and outcome assessors can be kept unaware of which pill was assigned.
-
Expected value
Expected value is the probability-weighted mean of a random variable when that mean exists. It describes a long-run average rather than a value that must occur on one trial. For example, a fair die has expected value 3.5 even though no face is labeled 3.5.
Read more: Expected value in the captured Statistics reading →
-
Completely randomized design
A completely randomized design assigns all experimental units to treatments using chance without first separating them into blocks. The design uses comparison and replication to assess treatment differences. For example, all eligible plants can be randomly allocated among three fertilizers.
Read more: Completely randomized design in the captured Statistics reading →
-
Experimental unit
An experimental unit is the smallest unit independently assigned a treatment. A human experimental unit is often called a subject or participant. For example, if fertilizer is assigned to an entire planter, the planter rather than each leaf is the experimental unit.
Read more: Experimental unit in the captured Statistics reading →
-
Randomized block design
A randomized block design first groups similar experimental units using a relevant characteristic. Treatments are then randomly assigned within each block, and comparisons account for the blocking. For example, fertilizer treatments can be randomized separately within sunny and shaded locations.
Read more: Randomized block design in the captured Statistics reading →
-
Matched pairs design
A matched pairs design compares two treatments within matched units or within the same unit. Assignment within pairs or treatment order is randomized when the design permits, and analysis uses the pairwise differences. For example, two shoe designs can be compared on the same person with their left-right assignment randomized.
Read more: Matched pairs design in the captured Statistics reading →
-
Population
A population is the full set of individuals or measurements a study aims to describe. It must be defined clearly before deciding whether a sample supports generalization. For example, all students enrolled in a particular school this year form a population.
-
Sample
A sample is the subset of a population actually studied. How it is selected matters for the conclusions that can be drawn about the population. For example, 60 selected students may be a sample from a school’s full enrollment.
-
Response variable
A response variable records the outcome being studied. Its relationship to an explanatory variable can be investigated without assuming that the explanatory variable causes it. For example, plant height is a response when researchers compare fertilizer treatments.
Read more: Response variable in the captured Statistics reading →
-
Confounding
Confounding occurs when the effects of different variables cannot be distinguished in the observed comparison. A potential explanation is that another variable is related to both the treatment or explanatory variable and the response. For example, comparing fertilizer A outdoors with fertilizer B indoors mixes fertilizer and growing-location effects.