Correlation: What r Really Tells You

Full transcript

Word for word what is said in the video — the same text as the subtitles. Times are minutes:seconds into the video.

0:00Which cloud is stronger? ▶

Two classes of students, one dot each: hours of sleep against hours on the phone. In class C the dots fall steeply, in class D they drift down gently but hug their band. Which class has the stronger link, the steep one or the gentle one? One number gives the answer: class D. That number is r, the correlation coefficient. By the end of this video: what that number really measures. And the one picture where it says zero while the pattern is perfect.

0:39What "correlation" means in everyday talk ▶

In everyday talk, correlation means any link at all, two things that go together. So a correlation of zero sounds like no link whatsoever, and even a national statistics office writes exactly that. Here is the crack: a pattern you can predict perfectly can still score zero. Stay with me for three things: what the number measures, how to read it, and where it breaks.

1:06The cross and the four boxes ▶

Twenty-five athletes on a school sports day, two numbers each: their sixty metre sprint time and their long jump. Faster runners tend to jump further, so the cloud falls from left to right. Draw a cross through the two averages, and every dot lands in one of four boxes. Fast and far, or slow and short: twenty-three athletes, and they pull r down. Fast and short, or slow and far: just two, and they pull r up.

1:42How hard each dot pulls ▶

How hard a dot pulls depends on how far it sits from the cross. Seconds and metres do not mix, so measure both in spreads, the typical distance from average: the z-score. This athlete is far out on both sides: left of the cross, which counts as minus, and above it, a plus. Multiply the two: minus times plus is minus, so this dot pulls r down hard. Add all twenty-five pulls and you get minus nineteen point two. Divide by twenty-four, one less than the number of athletes, and you have roughly the average pull. That is r: minus zero point eight.

2:27Sign and size ▶

That minus sign only gives the direction. Count seconds ahead of the slowest runner instead, and the cloud climbs: plus zero point eight, just as strong. The size, ignoring the sign, says how tightly the dots hug a sloping straight line. Minus one and plus one are perfect sloping lines, and zero means no straight-line trend. So a perfect pattern should score close to one, right?

2:56A thrown ball ▶

Now throw a ball straight up and mark its height every tenth of a second, for two seconds. Give me the time, and I can give you the height exactly. Before I show you, what number would you give this cloud?

3:15A perfect pattern, r = 0 ▶

Zero, and not roughly zero but exactly zero. Draw the cross again, through the average time and the average height. Every dot on the way up has a twin on the way down, at exactly the same height. One pulls r up, the other pulls it down just as hard, so they cancel pair by pair. The pattern is perfect, and r does not see it. So how can a perfect pattern score nothing at all?

3:47Rule 1: r only sees straight lines ▶

The answer was hiding in one word, straight: r only measures how tightly dots hug a sloping straight line. A curve, however perfect, is not a straight line, so r can miss it completely. Zero means no straight-line trend, not no link. That is why textbooks keep repeating one rule: look at the cloud before you trust the number.

4:16Rule 2: your eye cannot measure r ▶

The second rule runs the other way. Do not trust your eye on its own, either. Here are the same athletes again, with their long jump against their high jump. The real r is plus zero point five. In experiments, a cloud this strong looks much weaker than it is, closer to zero point three. Below about zero point two, a cloud looks almost the same as no link at all. Your eye cannot measure r, and r cannot show you the cloud, so you need both.

4:52Rule 3: one dot can move it ▶

The third rule: one dot can move r a long way. Add one more athlete: the fastest runner of all, at seven point three seconds. But this runner jumps only two point eight metres. That dot sits far out, in a box that pulls r up, so its pull is huge. One dot out of twenty-six, and r weakens from minus zero point eight to minus zero point five.

5:24One look back ▶

Back to the last video: class A and class B share every single number, only paired differently. Same numbers, so do they get the same r? Not even close: the band scores minus zero point nine six. The one without a shape scores minus zero point zero one, because r lives in the pairing.

5:50In one picture ▶

So, in one picture: r runs from minus one to plus one. The sign is the direction, and the size is how tightly the dots hug a sloping straight line. Zero means no sloping line, not no link, so always look at the cloud first. Your eye misjudges r and one far dot can drag it, so read the picture and the number together,

6:17Next: does one cause the other? ▶

and even then, one question is left that most headlines skip: when two things move together, did one cause the other? That is exactly where the next video starts. And if you have ever seen a correlation used as proof, tell me in the comments. And if you want to keep learning, subscribe, the next video will find you. Until then: happy learning.