Bias: Identify What Can Go Wrong

Two surveys of the same size can tell very different stories. The difference may come from whose responses were collected or how the question was asked, rather than from the population itself. Finding the source of a problem is more useful than labeling every imperfect study “biased.”

Selection and Response Bias

Bias in a sampling method is a systematic error that tends to make a statistic consistently larger or consistently smaller than the parameter it estimates. It is not simply the ordinary sample-to-sample difference produced by chance. A larger sample can reduce random variation while leaving a systematic problem in place.

In voluntary response, the sample consists entirely of people who choose to take part. People with strong opinions may respond more often than those who feel indifferent. A posted online poll about a new bus route may overrepresent unusually satisfied or unusually dissatisfied riders. Convenience sampling selects people who are easy to reach, such as riders at one nearby stop. These nonrandom methods do not use chance to select individuals and can introduce bias, even when the final number of responses is large.

Undercoverage occurs when the sampling method omits part of the population or makes that part less likely to be selected. A telephone list can omit households without listed numbers; an app-only survey can make nonusers less likely or unable to take part. This concerns the selection opportunity, not a refusal after selection.

Nonresponse occurs when some people chosen for a sample cannot be reached or do not respond. If selected night-shift workers are often asleep during daytime calls, their responses may be missing, and they may differ from respondents in ways important to the question. A carefully randomized original selection can still encounter nonresponse.

Response bias concerns what people report or what the measurement records. Responses can systematically differ from the true value in one direction. A leading question—“Don’t you agree that our wonderful new bus route is a success?”—can encourage agreement. A confusing double question can make answers difficult to interpret. A self-report of exercise may overstate activity if respondents want to look healthy. A measuring process can also produce systematic errors; the problem is not confined to questionnaires.

Source	Where the problem enters	Bus survey example
Voluntary response	People choose participation	An open “vote now” link attracts strong opinions
Undercoverage	Population units lack a fair selection opportunity	App-only invitations omit riders without the app
Nonresponse	Selected people do not provide responses	Chosen riders cannot be contacted at calling times
Response bias	Recorded answer differs systematically from truth	Leading wording encourages favorable ratings

Worked Example: Trace the Missing Riders

The city selects 200 names randomly from a complete rider roster, but only 80 answer the calls. The selection itself was random, so this is not an open voluntary-response sample. The 120 missing selected responses create a nonresponse concern. If a later reminder reaches only app users, that collection strategy may worsen the difference between respondents and other selected riders. To assess a likely direction of bias, we need a reason the missing riders differ in satisfaction; missing responses alone do not tell us whether the estimate is too high or too low.

Worked Example: Repair the Question

“How much money did you waste on taxis last week?” builds a negative judgment into a spending question. A more neutral version is “How many dollars did you spend on taxi rides during the seven days ending Sunday?” It names the quantity and time period without describing spending as waste. A clearer question reduces one response-bias concern, but it does not fix an incomplete sampling list.

When diagnosing a study, name the mechanism, connect it to the response of interest, and distinguish what you know from what you suspect. “The method could overestimate satisfaction because enthusiastic volunteers may be especially likely to respond” is an argument. “The sample is small, so it is biased” is not a diagnosis of a systematic selection error.

Independent Practice

Use the information in each question to make your own calculation, representation, or argument. Explain what your result means in its setting.

Original generated practice for the Fall 2026 course.

Variation or bias?

A bathroom scale adds 2 kg to every true weight. A random sample mean also differs from its population mean simply because the selected people vary.

Part a

Identify which difference is systematic bias and which can be ordinary sampling variation. Connect bias to the estimated parameter.

Your reasoning

Worked answer and rubric

Who decides to answer?

A neighborhood website invites anyone to click and rate the proposed parking rules. It reports 10,000 responses. A second interviewer asks only people standing near the office entrance.

Part a

Identify the selection concern in each method and explain why the large response count does not fix it.

Your reasoning

Worked answer and rubric

Not listed or not responding?

An app-only list omits residents without smartphones. Among listed residents randomly selected, some never reply.

Part a

Identify the two distinct bias concerns and explain when each enters the study.

Your reasoning

Worked answer and rubric

What might the answer conceal?

A questionnaire asks “Do you support our fair and necessary fee?” and then asks participants to report their exercise time while the fitness coach watches.

Part a

Identify two response-bias mechanisms and give one improvement for each.

Your reasoning

Worked answer and rubric

Audit the bus-feedback study

A city wants the proportion of all bus riders satisfied with a new timetable. It posts an open poll in its phone app, asks “Do you agree our excellent timetable makes travel easier and cheaper?”, and receives 6,000 replies. Riders without smartphones cannot see the poll. A separate follow-up randomly selects 200 riders from a complete roster, but 130 do not respond; late-night riders are especially hard to contact during daytime calls. The city knows late-night riders face more cancelled trips.

Part a

Identify the voluntary-response mechanism and explain a plausible satisfaction distortion.

Your reasoning

Part b

Identify undercoverage, including both outright omission and a way a group could be less likely to be selected.

Your reasoning

Part c

Identify the response-bias problems in the poll question. Rewrite it to collect one clearly defined, neutral response.

Your reasoning

Part d

Identify the follow-up’s nonresponse concern. Use the stated facts to explain a plausible direction of bias in its satisfaction estimate.

Your reasoning

Part e

Explain why increasing the open-poll replies to 60,000 would not automatically repair the design. Name another nonrandom method the city should avoid.

Your reasoning

Part f

Suppose riders report fewer missed rides because they are embarrassed about arriving late. Identify this additional bias source and explain what differs from what.

Your reasoning

Part g

Propose one repair for each of coverage, nonresponse, and wording. Explain why none guarantees that all bias is gone.

Your reasoning

Worked answer and rubric