Seeing Can be Deceiving, So Look at the Data!

Kevin Rudy 16 September, 2011

People like to say that seeing is believing, but the fact is that sometimes simply “seeing” isn’t enough. Whenever you’re making important decisions, like during a Lean Six Sigma project, it’s important to take a close look at the data. Because if you don’t, sometimes seeing is deceiving!

Consider college basketball. Since 1990, the team ranked #1 in the AP preseason poll  has won the national championship twice as often as the team ranked #1 in the final regular season poll. So after "seeing" every college basketball game, the voters opinion of the best team in the country gets worse!

Part of the problem is that when the voters see team A beat team B, they automatically think that means team A is better. But sometimes that just isn't true.

This made me wonder if the same thing applies to other sports, like football. Each year in the NFL, there are 48 games that are rematches. I collected the outcome of each game in 2010, then used Minitab Statistical Software to tally the number of times the winner of the first game also won the second game.
2010 Outcome of the Second Game in NFL Rematches
In 2010, the team that won the first game lost the second 54.17% of the time. That means if team A beat team B, they were more likely to lose the rematch! That's not much evidence for thinking the better team always wins. But we do have a small sample size, so let's go back and look at the data for the last 5 NFL seasons.
Second Game Outcome for NFL Rematches for the Last 5 Years

So over the last 5 NFL seasons, the team that wins the first game also wins the rematch about 56% of the time. That's still pretty low. But maybe it's because the team that won the first game played at home. Then they would be more likely to lose the second game because it's away. I'll use Minitab's Table of Descriptive Statistics to see if this is true.
Percentage of home and away teams that win the rematch
When the away team won the first game, they also won rematch 55.9% of the time. When the home team won the first game, they won the rematch 56.6% of the time. Those numbers are almost exactly the same! It doesn't matter if a team wins the first game at home or on the road. They're just as likely to win the next game either way! Now let's see what happens when the first game is close.
Percentage of teams that win the rematch when they win the first game by a TD or less
When the first game was decided by a touchdown or less, the winner won the rematch 53% of the time. Otherwise they won the rematch almost 60% of the time. I ran a 2 Proportions test on the data, and got a p-value of 0.36. That means the difference between the proportions isn't statistically significant. I ran the numbers for when the score of the first game was decided by 3 points or less and got the same results. So the margin of victory of the first game doesn't significantly change your chance of winning the rematch.

This just shows the randomness in football. If you are really trying figure out if team A is better than team B, you need to look at more than just the head to head match-up. Seeing one team beat another simply isn't enough.

And think of college football for a minute. In 2010, the Auburn Tigers (BCS Champions) won 7 games by one possession or less. If we were able to play that same season over again with the exact same players, odds are low that they would win all 7 of those rematches.

So am I suggesting that the voters should have voted somebody else #1 instead of Auburn? Of course not. Auburn won all of their games, and deserved their championship (although TCU fans might disagree). But it makes you think. With all the randomness, if only 2 teams are chosen to compete for the championship how confident can you be that they are actually the two best teams? Is the BCS Champion always the best team, or sometimes are they just the luckiest?