Variance: Why We Divide by n − 1

Full transcript

Word for word what is said in the video — the same text as the subtitles. Times are minutes:seconds into the video.

0:00Same average, different player ▶

Two players, eight turns each, and the same average reaction time: 280 milliseconds. On paper they are identical; watch them play for one minute and you would never swap them. By the end of this video, you will measure that difference with one number. And you will know why that number divides by one less than you counted.

0:27The idea that always gives zero ▶

So we need a ruler for spread, and the obvious one is this: how far is each turn from the average? Take those distances, add them up, and divide by eight. Try it on either player and you get exactly zero. That is not bad luck, it is arithmetic: the average sits at the balance point, so everything above it cancels everything below. Stay with me and you get three things: a ruler that works, the number every report quotes, and the reason it divides by seven.

1:02Square every distance ▶

The fix is older than statistics: square each distance. Squaring throws away the sign, so nothing cancels any more. And it punishes a big miss far harder than a small one: twice as far off counts four times as much. Picture every squared distance as a real square, standing on the line where that turn landed. For the steady player they are small change. For the streaky one, two of them are enormous. Add the areas up and the totals are 700 against 6300.

1:41Divide, then take the root ▶

A total is not a ruler, though: measure eighty turns instead of eight and it grows, without anything getting wilder. So divide, and you get the average square. That is the variance — and it is measured in milliseconds squared, which is not something anybody can picture. Take the square root and you are back in milliseconds: 10 for the steady player, 30 for the streaky one. Three times the spread, from two lists with the same average. That is the standard deviation, and it is the answer the average could not give. But I divided by seven, not eight, and you only saw eight turns out of thousands. Before you watch: is the spread you just measured too big, too small, or right?

2:35Slide the point you measure from ▶

Here is the whole reason, in one move. Keep the eight turns where they are, and slide the point you measure from. Every square grows or shrinks as it moves, so the total area rises and falls with it. Watch where that total is smallest. It bottoms out exactly at the average of those eight turns, and nowhere else. That is not a quirk of this player: it is true of every list of numbers there is. And your eight turns are not the whole story — the player has thousands more, with a true average you never get to see. So you measured from the one point that makes your own spread look as small as it possibly can.

3:19What n − 1 repairs ▶

So a sample tends to understate the spread: here it came out too low in two of every three draws. I ran two hundred thousand samples of eight, from a population whose spread I already knew. Dividing by eight gave 87.6 on average, when the truth was 100. Dividing by seven gave 100.1. One less in the divider, and the bias is gone. That is what n minus one buys you: not a bigger number for its own sake, but an honest one.

3:54Which divider, when ▶

So when do you use which? If your numbers are the whole group, divide by n: every student in one class, say, or all the turns a game ever had. If they stand in for something bigger, and they nearly always do, divide by n minus one. That is why your calculator has two spread buttons, and why the sample one is the default. Spreadsheets do the same: one function assumes a sample, the other assumes you already have everybody.

4:28One look back ▶

One quick look back: last video you saw twenty-eight travel times, and two hills a wide bin could hide. Which of those two groups was the more spread out — the cyclists around 11 minutes, or the bus students around 34? The bus group, and now you can say by how much: 4.7 minutes against 2.4. The picture gives you the shape; this number puts a size on it.

4:55In one picture ▶

So here it is in one picture. Distances from the average cancel, so you square them. Add the squares, divide by one less than you counted, and take the root: that is the standard deviation. Two lists can share an average and be nothing alike, and this is the number that says so. Which raises the next question: everything so far has been one column of numbers —

5:22Next: two numbers per person ▶

— so what happens when you have two? That is next time: two numbers for every person on one picture, and what a cloud of dots can and cannot tell you. And if you want to keep learning, subscribe, the next video will find you. Until then: happy learning.