Access this content offline
Omniguides by NOX
Skip to the reading

1.6 Expanding on Linear Regression

Section Overview

  • Interactive 1.6.1 - Example 1.5.1
  • Example 1.6.1 - R. Plot & SD of Res. on Calc
  • Gallery 1.6.1
  • Interactive 1.6.2 - Example 1.6.1
  • Pop Quiz 1.6.1

Predicting the Response

In the previous section, we talked about how y^\hat y in the least-squares regression line equation was the predicted value of the response variable yy for a given value of an explanatory variable xx. However, that predicted value was probably different than the actual observed value. This difference is called the residual, and is one of the previously mentioned “vertical lines” minimized through linear regression. It is defined as:

residual=y−y^\text{residual}=y-\hat y

where

  • yy = observed value of response
  • y^\hat y = predicted value of response

See Interactive 1.6.1 to see the residuals of the scatterplot from Example 1.5.1, the backpack problem.

Interactive 1.6.1 Example 1.5.1 (Observed and Predicted Values)

Annotated scatterplot from Example 1.5.1, with Backpack Weight (kg) on the horizontal axis, labels 1 through 7, black observations, and a blue increasing regression line. Dashed vertical segments connect observed points with the line. Near weight 5.3 kg, callouts label the black point Observed Value, the vertical difference Residual, and the point on the line Predicted Value. No vertical-axis labels are visible. Open full-size image S0111
Source gap (S0111): The captured interactive records “Observed Value,” “Residual,” and “Predicted Value” callouts but no activated callout text. Its vertical-axis labels are not displayed.

We can also use the regression line to predict a y-value from an x-value not originally in our data set. However, we have to be careful of extrapolation, using the regression line to predict values outside our range of data.

Fitting the Regression Line to the Data

One way to determine if a regression line fits your data is to construct a residual plot, a graph where the regression line becomes y=0y=0 and the residuals are perpendicular to it.

When examining a residual plot, we want to look for two things:

  • The residual plot should not have a clear form. In other words, the data points should be scattered about randomly on the plot. Otherwise, it is indicative that a form other than a line would be a better model for the data.
  • The residuals should be relatively small. A good measure of this is the standard deviation of the residuals ss, the average amount of variation in the residuals. It is found by using the formula
s=∑residuals2n−2s=\sqrt{\frac{\sum\text{residuals}^2}{n-2}}

where

  • ∑residuals2\sum\text{residuals}^2 = sum of squared residuals and
  • nn = number of observations in data set

Another way to determine if a regression line fits your data is to calculate the coefficient of determination r2r^2. We might have caught a glimpse of this value on our calculator screen when performing the linear regression function on our backpack problem. It is the percentage of variation accounted for in your response variable by relating your response variable yy to your explanatory variable xx. If this sounds confusing, don’t worry; the following example problem will clear things up for you!

Before we attempt to tackle the example problem however, we want to make one last point about fitting a regression line to data. As we mentioned in the last section, the correlation coefficient is not resistant to outliers and by extension neither is the least-squares regression line. If either the regression line or the correlation coefficient change significantly by removing an outlier, the outlier is called an influential observation.

Example 1.6.1

Example 1.6.1

Recall from Example 1.5.1 that a high school Statistics teacher wanted to see if the backpack weight of his 21 students could be correlated with their GPA:

Backback Weight (kg)Grade Point Average (%)
0.5679
0.6780
0.8679
1.0984
1.5482
2.1686
2.4882
2.5372
2.8388
3.1083
3.3385
3.5989
3.8987
3.9185
4.1386
4.5690
5.1181
5.3292
6.2787
8.3081

Using what we’ve learned in this section:

a) Extrapolate the data set to calculate the weight of a student’s backpack if he/she had a 100% GPA and a 60% GPA.

b) Interpret the residual plot and calculate the standard deviation of the residuals.

c) Find and interpret the coefficient of determination.

d) Determine if either of outliers in the data is an influential observation.

Source gap (S0112): The repeated table on pages 50–51 contains 20 displayed rows, although its prompt says 21 students. It goes from 1.54 kg / 82% on page 50 directly to 2.16 kg / 86% on page 51. The 1.89 kg / 83% row appears in the original Example 1.5.1 table and in the calculator gallery, but is absent from this repeated source table.
Example 1.6.1 a) — recorded answer

a) When we extrapolate, we are predicting values outside of our given range of data. If we wanted to find out what backpack weight would be needed for a student to have a 100% GPA, using the regression formula we found in Example 1.5.1, we get:

100=81.097+0.8508x100=81.097+0.8508x

where x=22.2 kgx=22.2\text{ kg}!

If the student had a 60% GPA, we get:

60=81.097+0.8508x60=81.097+0.8508x

where x=−24.8 kgx=-24.8\text{ kg}! Stop the madness!

The list-entry instructions shown in frames 2 and 3 are:

L3=81.097+0.8508L1L_3=81.097+0.8508L_1
L4=L2−L3L_4=L_2-L_3

The displayed list rows in those two frames are transcribed as captured:

L1L2L3L4
.567981.145-.2573
.678081.154-1.667
.867981.17-2.829
1.098481.191.9756
1.548281.228-.4072
1.898381.258.29499
…………
6.278781.63.56848
8.38181.803-7.159

The final calculator screen displays these residual summary values:

StatisticDisplayed value
Mean3.0971429E-4
Sum.006504
Sum of squares350.572248
Sample standard deviation4.186718563
Population standard deviation4.085818951
n21
Example 1.6.1 b) — recorded answer

b) See Gallery 1.6.1 for instructions on creating a residual plot. Examining the residual plot, we can see that replacing the observed response with the residual for the y-value gives us the y=0y=0 line with the data points above or below this line. We can see that residual plot does not have a clear form with the data points scattered about randomly on the plot. This means that a line is the best model for the data. However, the residuals are fairly large, especially the ones for the outliers.

To determine just how large on average the residual is, or the standard deviation of the residuals, we look at Gallery 1.6.1 (last image) for calculating the sum of the squared residuals. We get ∑residuals2=350.572\sum\text{residuals}^2=350.572. Plugging this into our formula, we get:

s=∑residuals2n−2=350.57219=4.295s=\sqrt{\frac{\sum\text{residuals}^2}{n-2}}=\sqrt{\frac{350.572}{19}}=\mathbf{4.295}

This means our predictions for GPA on average will be off by about 4.3%.

Example 1.6.1 c) — recorded answer

c) The coefficient of determination was found when we went through the steps in performing the LinReg(a+bx) function in Gallery 1.5.3. We found the value to be r2=0.138r^2=0.138. This means that only about 14% of the variation in GPA is accounted for by the least-squares regression of GPA on backpack weight. 86% of the variation unaccounted for could be due to other factors, such as study habits, motivation, and intelligence.

Example 1.6.1 d) — recorded answer

d) A quick and easy way to tell if either of our outliers is also an influential observation is to remove each outlier one at a time from our data set, perform the LinReg(a+bx) function, and record the correlation. If we want to remove the outlier in the y direction, we have to remove 2.53 kg and 72% from the L1L_1 and L2L_2 lists on our calculator.

Performing the LinReg(a+bx) function, we get r=0.404r=0.404. This doesn’t seem like a big change from our original correlation with the y outlier included, r=0.371r=0.371, but to be sure, we want to perform the same procedure on the x outlier, the student with the 8.3 kg backpack and the 81% GPA. Putting 2.53 and 72 back into L1L_1 and L2L_2, removing the coordinates for the x outlier in the two lists, and performing the LinReg(a+bx) function, we get r=0.571r=0.571. This seems like a pretty significant change relative to removing the y outlier, so we can conclude that the child with the the 8.3 kg backpack and the 81% GPA is an influential observation. This is confirmed with Interactive 1.6.2, where the regression line representing the data set without the x outlier is in a significantly different position than the other two regression lines.

What can we conclude from this problem? From the large standard deviation of the residuals, the small coefficient of determination, and the presence of influential outliers, we can conclude that backpack weight alone cannot effectively predict the GPA of a student. Maybe those students should have bought some Omniguides, instead!

Interactive 1.6.2 Example 1.6.1

Captured comparison scatterplot with Backpack Weight (kg) labels 1 through 7. Blue and orange regression lines are close together; a green line is steeper. Callouts identify Change in Regression Lines and Influential Observation. The centered orange box reads Without Y Outlier, y=81.577+0.7469x, r=0.319. Blue and green boxes and the point indicated by Influential Observation are clipped at the image edges; vertical-axis labels are absent. Open full-size image S0125

The fully visible comparison box reads Without Y Outlier, y=81.577+0.7469xy=81.577+0.7469x, r=0.319r=0.319.

Source gap (S0125): The left blue and right green comparison boxes are horizontally clipped. The blue fragments end in “Points,” “0.8508x,” and “371”; the green fragments begin “Withou,” “y = 79.2,” and “r =”. Their complete headings and formulas are not recorded. The “Influential Observation” callout points beyond the right edge, and no activated callout states are captured. Vertical-axis labels are absent.

Pop Quiz 1.6.1

Question 1 of 5

For an observation with an x-value of 10 and a y-value of 14, that belongs to a data set with the regression line y^=8.1+0.50x\hat y=8.1+0.50x, what is the residual?

Question 2 of 5

For question 1, r2=0.73r^2=0.73. This means that

Option D’s square-root expression is 0.73\sqrt{0.73}.

Question 3 of 5

Larger values of r2r^2 imply that the observations are more closely grouped about the

Question 4 of 5

If a scatterplot had a smaller correlation coefficient when its influential observation to the right of the other data points was removed, and this observation was larger than its predicted value, then keeping the outlier would lead to:

Question 5 of 5

If a scatterplot had a larger correlation coefficient when its influential observation to the right of the other data points was removed, and this observation was smaller than its predicted value, then keeping the outlier would lead to:

Source gap: The five captured quiz states S0126–S0130 display no checked answers, grading, or feedback. No answer key was recorded.
Glossary and deeper reading · 10 refreshers

These editorial refreshers supplement the reading. Definitions and examples are added guidance; original source wording and recorded answers remain in their original context.

  • Least-squares regression

    Least-squares regression fits a line predicting a response from an explanatory variable. The fitted line minimizes the sum of squared vertical residuals between observed and predicted responses. For example, a fitted line can predict a plant’s height from its age, within the range of observed ages.

    Read more: Least-squares regression in the captured Statistics reading →

  • Response variable

    A response variable records the outcome being studied. Its relationship to an explanatory variable can be investigated without assuming that the explanatory variable causes it. For example, plant height is a response when researchers compare fertilizer treatments.

    Read more: Response variable in the captured Statistics reading →

  • Residual

    A residual is an observed response minus its predicted response. A positive residual places the observed point above the fitted line, while a negative residual places it below. For example, an observed height of 18 centimeters and a predicted height of 16 centimeters give a residual of 2 centimeters.

  • Scatterplot

    A scatterplot displays paired values of two quantitative variables as points. The horizontal coordinate represents one variable and the vertical coordinate represents the other. For example, plotting hours studied against test scores helps reveal a pattern without establishing causation.

    Read more: Scatterplot in the captured Statistics reading →

  • Observation

    An observation is a recorded value or set of values for an individual or occasion. The individual may be a person, object, or event rather than a human subject. For example, one plant’s measured height is an observation in a plant-growth study.

    Read more: Observation in the captured Statistics reading →

  • Standard deviation of residuals

    The standard deviation of residuals estimates the typical prediction error around a fitted regression line. For simple linear regression it is the square root of the sum of squared residuals divided by the sample size minus 2. For example, a residual standard deviation of 3 points indicates scatter around predictions on the scale of a few score points.

  • Coefficient of determination

    The coefficient of determination is the fraction of observed response variation accounted for by a fitted regression model. In simple linear regression with an intercept, it equals the square of the correlation coefficient. For example, a value of 0.64 means that the fitted linear model accounts for 64% of the response variation in those data.

  • Explanatory variable

    An explanatory variable is used to predict or explain variation in a response variable. In an experiment, researchers deliberately assign values or treatments related to an explanatory variable. For example, assigned study time could be an explanatory variable when test score is the response.

    Read more: Explanatory variable in the captured Statistics reading →

  • Correlation coefficient

    Pearson’s correlation coefficient measures the direction and strength of a linear relationship between two quantitative variables. It ranges from −1 to 1, and a value near 0 does not rule out a strong curved relationship. For example, points closely following a decreasing straight line have correlation near −1.

    Read more: Correlation coefficient in the captured Statistics reading →

  • Resistant measure

    A resistant measure changes relatively little when a small number of observations become extreme. Resistance means reduced sensitivity, rather than complete immunity to every change in the data. For example, changing a largest value from 30 to 300 may leave the median unchanged while substantially increasing the mean.

    Read more: Resistant measure in the captured Statistics reading →

Search lessons, terms and problems