Wednesday, February 13, 2013

Definitive Proof That The NCAA's "New" SOS Calculation Method Is Wrong

We had a revelation this week courtesy of Dave McHugh of D3hoops.com (and Hoopsville). The NCAA Championships Committee has changed the way they're computing the strength of schedule component that is a part of the primary selection criteria for the NCAA Tournament.

Strength of schedule (SOS) is made up of two components: opponents' winning percentage (OWP) and opponents' opponents' winning percentage (OOWP). They're calculated in a similar manner, but the concept of OWP is easier to grasp and "see" so I'll focus on that one component for the purpose of this blog post (but you can apply the same principles to OOWP).

The NCAA realizes that home games and road games differ in difficulty, so they've decided to add in a multiplier (to both OWP and OOWP) of 0.75 for home games, 1.25 for road games, and 1.00 for neutral site games. This isn't a change from last year.

Anyway, the old way to compute OWP and OOWP (I'll also call it the correct way) was computed using the average of the opponents' W/L percentage. For example, if a team played two teams with records of:

at home vs. 3-1 (.750)
on the road vs. 4-2 (.667)

Their OWP was calculated to be [ (.750 x .75) + (.667 x 1.25) ] / 2 = .698.

The new way (I'll also call it the incorrect way) is computed using the sum of the opponents' wins and losses to come up with an overall percentage. So for the same two teams above it would be [(3 x .75) + (4 x 1.25)] / [(4 x .75) + (6 x 1.25)] = .690.

This small example is only to show that there is a difference between the two calculation methods. It's only a two-game portion of a schedule, and a .008 difference isn't huge, but let's consider another example.

Opponent W L PCT Location MULT Old OWP New W New L New OWP
Team A 19 1 0.950 Away 1.25 1.1875 23.8 1.3 -
Team B 8 8 0.500 Home 0.75 0.375 6.0 6.0 -
Team C 8 8 0.500 Home 0.75 0.375 6.0 6.0 -
Team D 8 8 0.500 Home 0.75 0.375 6.0 6.0 -
Team E 8 8 0.500 Home 0.75 0.375 6.0 6.0 -
Team F 8 8 0.500 Home 0.75 0.375 6.0 6.0 -
Team G 8 8 0.500 Home 0.75 0.375 6.0 6.0 -
Team H 8 8 0.500 Home 0.75 0.375 6.0 6.0 -
Team I 8 8 0.500 Home 0.75 0.375 6.0 6.0 -
Team J 8 8 0.500 Home 0.75 0.375 6.0 6.0 -
Team K 8 8 0.500 Home 0.75 0.375 6.0 6.0 -
Team L 2 10 0.167 Away 1.25 0.208 2.5 12.5 -
TOTAL 0.429 86.3 73.8 0.539

So it's a road game against a really good 19-1 team, a bunch of home games against 8-8 teams, and a road game against a bad 2-10 team. The old method gives an OWP of .429 and the new method gives an OWP of .539. Which is right? It's hard to say, but all those home games against .500 teams makes it seem like it should be a below .500 OWP. I like the first method. But that's really not my point. The real point I'm going to make comes when we flip all those home games to road games.

Opponent W L PCT Location MULT Old OWP New W New L New OWP
Team A 19 1 0.950 Away 1.25 1.188 23.8 1.3 -
Team B 8 8 0.500 Away 1.25 0.625 10.0 10.0 -
Team C 8 8 0.500 Away 1.25 0.625 10.0 10.0 -
Team D 8 8 0.500 Away 1.25 0.625 10.0 10.0 -
Team E 8 8 0.500 Away 1.25 0.625 10.0 10.0 -
Team F 8 8 0.500 Away 1.25 0.625 10.0 10.0 -
Team G 8 8 0.500 Away 1.25 0.625 10.0 10.0 -
Team H 8 8 0.500 Away 1.25 0.625 10.0 10.0 -
Team I 8 8 0.500 Away 1.25 0.625 10.0 10.0 -
Team J 8 8 0.500 Away 1.25 0.625 10.0 10.0 -
Team K 8 8 0.500 Away 1.25 0.625 10.0 10.0 -
Team L 2 10 0.167 Away 1.25 0.208 2.5 12.5 -
TOTAL 0.637 126.3 113.8 0.526

It's an all-road schedule against nine teams that are at least .500 and one team below .500. The old method says this is a decently tough schedule and gives it a .637 OWP. The new method says it's only a decently tough schedule and gives it a .526 OWP.

BUT WAIT! The new method gives a lower OWP value for the all-road schedule (.526) than it did when eight of the games were at home (.539)!

We we simply sum the games up like this (instead of taking the average of the percentage), we're not giving each game (opponent) equal weight in our OWP calculation. This is especially true in conjunction with the home/away multiplier (sometimes known as HAM). In the "new" method, the HAM doesn't make it so road games are "tougher" than home games, it just makes it so road games weigh more heavily than home games. This is probably the opposite of what should be true.

Want even more fun? Let's make all those games in the above example home games.

Opponent W L PCT Location MULT Old OWP New W New L New OWP
Team A 19 1 0.950 Home 0.75 0.7125 14.3 0.8 -
Team B 8 8 0.500 Home 0.75 0.375 6.0 6.0 -
Team C 8 8 0.500 Home 0.75 0.375 6.0 6.0 -
Team D 8 8 0.500 Home 0.75 0.375 6.0 6.0 -
Team E 8 8 0.500 Home 0.75 0.375 6.0 6.0 -
Team F 8 8 0.500 Home 0.75 0.375 6.0 6.0 -
Team G 8 8 0.500 Home 0.75 0.375 6.0 6.0 -
Team H 8 8 0.500 Home 0.75 0.375 6.0 6.0 -
Team I 8 8 0.500 Home 0.75 0.375 6.0 6.0 -
Team J 8 8 0.500 Home 0.75 0.375 6.0 6.0 -
Team K 8 8 0.500 Home 0.75 0.375 6.0 6.0 -
Team L 2 10 0.167 Home 0.75 0.125 1.5 7.5 -
TOTAL 0.382 75.8 68.3 0.526

Yeah, so the "new" method gives an all-home schedule the exact same OWP as when they were all road games. That's because the HAM is only scaling the number of games in the new method, it's not actually changing the perceived difficulty of the game (the percentage).

This isn't just my opinion, this isn't just a different way to do things, this is simply wrong.