Scatter Plots, Covariance and Correlation, statistics lesson

A teacher records, for each student, the hours spent studying in the week before a test and the score on that test. Do students who study more tend to score higher, and how closely do the two go together? The answer lives in the pairs, not in the two columns separately. This lesson plots the pairs, measures how they vary together with the sample covariance and the Pearson correlation, shows what a single point or a narrow range can do to those numbers, and adds a rank-based measure for relationships that keep rising without following a straight line.

Key terms

Scatter plot

Definition A scatter plot displays $n$ paired observations $(x_1, y_1), \dots, (x_n, y_n)$ of two numerical variables, measured on the same individuals, as $n$ points in a plane, with one variable on the horizontal axis and the other on the vertical axis. It shows the direction of a relationship (upward or downward), its form (straight, curved or clustered), its strength (how tightly the points follow the form) and any unusual points. When one variable is thought of as explaining the other, it usually goes on the horizontal axis.

In plain words Each point is one individual, so the pairing has to stay intact. A scatter plot shows association in the data. It does not show that one variable causes the other, because other conditions may affect both.

Example Three constructed students studied 1, 2 and 3 hours and scored 2, 1 and 3 points on a short quiz, which gives the points $(1, 2)$ , $(2, 1)$ and $(3, 3)$ . The pattern tilts weakly upward. Sorting the two columns separately would produce $(1, 1)$ , $(2, 2)$ and $(3, 3)$ , a perfect line, but the pairs $(1, 1)$ and $(2, 2)$ belong to no student.

Sample covariance

Definition For $n \ge 2$ pairs with means $\bar{x}$ and $\bar{y}$ , the sample covariance is

s_{xy} = \frac{\sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})}{n - 1}

A product $(x_i - \bar{x})(y_i - \bar{y})$ is positive when both deviations have the same sign and negative when their signs differ. The covariance is positive when the positive products outweigh the negative ones in total, which happens when values above the mean of one variable tend to go with values above the mean of the other. Its unit is the product of the units of the two variables. If every $x_i$ is replaced by $a + bx_i$ and every $y_i$ by $c + dy_i$ , the covariance becomes $bd\, s_{xy}$ . The covariance of a variable with itself is its sample variance, so $s_{xx} = s_x^2$ .

In plain words Center both variables, multiply the paired deviations and add the products, then divide by $n - 1$ . The sign gives the direction of the joint variation. The size depends on the units, so a covariance of 5 can be large or tiny depending on how the variables were measured.

Example For the three students, $\bar{x} = 2$ hours and $\bar{y} = 2$ points. The deviation pairs are $(-1, 0)$ , $(0, -1)$ and $(1, 1)$ , with products 0, 0 and 1, so

s_{xy} = \frac{0 + 0 + 1}{3 - 1} = 0.5,

measured in hours times points. If every score is multiplied by 10, every $y$ deviation is 10 times as large and the covariance becomes 5, although the pattern of the points is unchanged.

Pearson correlation

Definition For $n \ge 2$ pairs whose sample standard deviations satisfy $s_x > 0$ and $s_y > 0$ , the Pearson correlation is

r = \frac{s_{xy}}{s_x s_y}.

It has no unit and always satisfies $-1 \le r \le 1$ . It equals 1 exactly when all points lie on a line with positive slope, and it equals $-1$ exactly when they lie on a line with negative slope. It is undefined when either variable is constant. Replacing every $x_i$ by $a + bx_i$ leaves $r$ unchanged when $b > 0$ and reverses its sign when $b < 0$ , and the same holds for $y$ . The correlation measures linear association, the tendency of the points to follow a straight line, in both direction and strength.

In plain words Correlation is covariance with the units divided out, so it measures how closely the points follow a rising or falling straight line. A value of 0.5 does not mean that half the points agree, or that one variable produces half of the other. A value near 0 means little linear association, not that the variables are unrelated, since a strong curved pattern can give $r = 0$ .

Example For the three students, both sample standard deviations equal 1, so $r = 0.5 / (1 \cdot 1) = 0.5$ . Multiplying every score by 10 multiplies both $s_{xy}$ and $s_y$ by 10, so $r$ stays 0.5. Now take $x = -2, -1, 0, 1, 2$ and $y = x^2$ , which gives $y = 4, 1, 0, 1, 4$ . Here $\bar{x} = 0$ and $\bar{y} = 2$ , and the deviation products are $-4$ , $1$ , $0$ , $-1$ and $4$ . They add to 0, so $r = 0$ even though every $y$ is determined exactly by its $x$ .

Influential observation

Definition An observation is influential for a summary or a fitted relationship when including or removing it changes the result substantially. An outlier is an observation that is unusual relative to the pattern of the rest of the data. A point far from $\bar{x}$ or $\bar{y}$ has a large deviation, so its product can dominate the covariance and the correlation. The two ideas overlap but are not the same, since a point far from the others can be influential while still following the general direction of the rest.

In plain words Look at the scatter plot with and without a questioned point, and report both results. Check whether the point is a recording error or comes from a different situation, but do not delete it only because the answer looks better without it.

Example Add a fourth student, who studied 8 hours and scored 8 points, to the three students. Both means become 3.5, the sum of the deviation products becomes 28, and the new point alone contributes $4.5 \times 4.5 = 20.25$ of it. The correlation rises from 0.5 to about 0.97. Had the fourth student scored 1 point instead, the correlation would be about $-0.39$ . One point far out on the horizontal axis decides both the sign and the size of $r$ here, so the value describes that student more than the other three.

Range restriction

Definition Range restriction occurs when the data cover only part of the range of values that a variable takes in the population of interest, often because individuals were selected on that variable or on a related one. A correlation computed from restricted data describes only the restricted group. It can be smaller or larger than the correlation in the full group, and no general correction recovers the full value without further assumptions.

In plain words A narrow slice of the data can hide a relationship or exaggerate one. Say which group a correlation describes, for example applicants who passed an entrance test rather than all applicants.

Example The six constructed pairs $(1, 1)$ , $(2, 3)$ , $(3, 2)$ , $(4, 4)$ , $(5, 6)$ and $(6, 5)$ have sample covariance 3.1, and both sample variances are 3.5, so $r = 3.1/3.5 \approx 0.89$ . Keeping only the pairs with $x \ge 4$ leaves $(4, 4)$ , $(5, 6)$ and $(6, 5)$ , whose correlation is 0.5. Restriction can also raise a correlation. The curve $y = x^2$ above has $r = 0$ for $x$ from $-2$ to $2$ , but its three points with $x \ge 0$ , namely $(0, 0)$ , $(1, 1)$ and $(2, 4)$ , have $r \approx 0.96$ .

Spearman rank correlation

Definition To compute the Spearman rank correlation $r_s$ , replace each $x_i$ by its rank among the $x$ values, with rank 1 for the smallest, and each $y_i$ by its rank among the $y$ values. Tied values receive the average of the ranks they would occupy. The Spearman correlation is the Pearson correlation of the $n$ pairs of ranks, and it is defined when neither variable is constant. An association is monotonic when $y$ tends to move in one direction as $x$ increases, upward throughout or downward throughout, at a rate that need not be constant. With no ties, $r_s = 1$ exactly when a larger $x$ always comes with a larger $y$ , and $r_s = -1$ exactly when a larger $x$ always comes with a smaller $y$ .

In plain words Pearson correlation asks how close the points are to a straight line. Spearman correlation asks only how well the two orderings agree, so a curve that keeps rising gets a perfect score. Replacing a variable by a strictly increasing function of it, such as the square or the logarithm of positive values, keeps the order and leaves $r_s$ unchanged. A pattern that rises and then falls is not monotonic, and both measures can be near 0 for it. Neither measure establishes causation.

Example For $x = 1, 2, 3, 4$ and $y = 1, 4, 9, 16$ , the relationship $y = x^2$ is curved and the Pearson correlation is about 0.98. Both lists are already in increasing order, so both rank lists are 1, 2, 3, 4 and $r_s = 1$ . For the values $y = 4, 1, 0, 1, 4$ of the earlier curve, the two values 1 share ranks 2 and 3 and each gets 2.5, and the two values 4 share ranks 4 and 5 and each gets 4.5. The ranks of $y$ are 4.5, 2.5, 1, 2.5 and 4.5, and their Pearson correlation with the ranks 1 to 5 of $x$ is 0.

Check yourself

For the pairs $(1, 1)$ , $(2, 3)$ , $(3, 2)$ and $(4, 4)$ , compute the sample covariance and the Pearson correlation.

Show the answer

Both means are 2.5. The deviation products are $(-1.5)(-1.5) = 2.25$ , $(-0.5)(0.5) = -0.25$ , $(0.5)(-0.5) = -0.25$ and $(1.5)(1.5) = 2.25$ , which add to 4, so $s_{xy} = 4/3$ . Each variable has squared deviations 2.25, 0.25, 0.25 and 2.25, which add to 5, so $s_x^2 = s_y^2 = 5/3$ and $s_x s_y = 5/3$ . The correlation is

r = \frac{4/3}{5/3} = 0.8.

Starting from the pairs in question 1, predict the covariance and the Pearson correlation after each change separately. First, every $x$ is converted from hours to minutes. Second, 10 is added to every $y$ . Third, every $y$ is replaced by $5 - y$ . Fourth, every $y$ is replaced by its square.

Show the answer

Converting hours to minutes multiplies every $x$ by 60, so the covariance becomes $60 \cdot 4/3 = 80$ . The standard deviation $s_x$ is also multiplied by 60, so $r$ stays 0.8.

Adding 10 to every $y$ raises the mean by 10 and leaves every deviation unchanged, so the covariance stays $4/3$ and $r$ stays 0.8.

Replacing $y$ by $5 - y$ reverses the sign of every $y$ deviation, so the covariance becomes $-4/3$ and $r$ becomes $-0.8$ .

Squaring gives $y = 1, 9, 4, 16$ . This change is not linear, so nothing guarantees that $r$ is preserved, and in fact the covariance becomes $20/3$ and the correlation about 0.79. The Spearman correlation stays 0.8, because squaring positive numbers keeps their order.

Five constructed students slept 4, 5, 6, 7 and 8 hours before a test and scored 60, 75, 80, 75 and 60 points. Compute the Pearson correlation. A classmate concludes that the score is unrelated to sleep. Is that right?

Show the answer

The means are 6 hours and 70 points. The deviation products are $(-2)(-10) = 20$ , $(-1)(5) = -5$ , $0 \cdot 10 = 0$ , $1 \cdot 5 = 5$ and $2 \cdot (-10) = -20$ , which add to 0, so $s_{xy} = 0$ and $r = 0$ .

The conclusion does not follow. The scores rise and then fall in an exactly symmetric pattern, so these five pairs show a clear relationship between sleep and score, but not a linear one. The Spearman correlation is also 0 here, because the ranks of the scores, 1.5, 3.5, 5, 3.5 and 1.5, rise and fall in the same symmetric way. Both measures can miss a pattern that is not monotonic, while a scatter plot shows the curve at once. And even a strong correlation in data like these would show an association, not that sleep causes the scores.

Scatter Plots, Covariance and Correlation

Key terms

Scatter plot

Sample covariance

Pearson correlation

Influential observation

Range restriction

Spearman rank correlation

Check yourself

How this lesson connects

Builds on

Used later in