Part of the fun of sports statistics is that many of the questions are unanswerable in an absolute sense. After a project to improve patient satisfaction scores at a hospital, you can and should measure patient satisfaction scores to determine that the increase met your goals. In sports, we don’t have exact comparisons from player to player, especially when their careers don’t even overlap. Thus, it wasn’t surprising that some people disagreed with my case that Tom Brady is the Best Super Bowl Quarterback Ever. To challenge Brady’s anointment, two suggested follow-up analyses are A) better quarterbacks should win Super Bowls by larger margins, and B) better quarterbacks should outperform their competition. Let’s take a look at these statistics and see what happens.
Margin of Victory
In my initial analysis, I used a very aggressive standard for Super Bowl victories to winnow the field of quarterbacks in consideration. Thus, we’re excluding some quarterbacks with extraordinary performances in the Super Bowl like Phil Simms, Jim Plunkett, and Kurt Warner. The group we’re looking at is Terry Bradshaw, Troy Aikman, Tom Brady, and Joe Montana. Here are the margin of victory statistics in an individual value plot:
By looking at the median margins of victory, Troy Aikman leads this category, but I think it reveals a problem with using margin of victory by itself as a measure of a quarterback’s success in the Super Bowl. Dallas forced 9 turnovers in Super Bowl XXVII. An overmatched Buffalo team that lost Jim Kelly in the second quarter scored enough points to win even if Aikman had thrown 0 touchdown passes. I originally made the argument that margin of victory should count against a quarterback, but let’s see if we can get a more detailed view by including more statistics.
The 3-D scatterplot shows three variables:
- Spread: The expected margin of victory before the Super Bowl. Positive point spreads indicate that the underdog won.
- QBR difference: The difference between the winning quarterback's rating and the average rating of opposing quarterbacks for the losing team during the playoffs.
- Margin of victory: The difference between the scores of the winning and losing teams.
The legend shows that I used different symbols for these events:
- Favorite: The quarterback’s team was expected to win the Super Bowl.
- Push: The winning and losing teams were considered equally likely to win the Super Bowl.
- Underdog: The quarterback’s team was expected to lose the Super Bowl.
In this analysis, I would say that high margins of victory are good, higher differences are good, and higher spreads are good. Interestingly, there are no performances where all 3 variables are high. The victories by Aikman and Montana have the highest margins of victories, but the spreads indicate that those games were not supposed to be competitive. Brady has won the only Super Bowl where any of these quarterbacks was an underdog, but the difference between his passer rating and what the defense allowed is not as high as many other Super Bowls. Terry Bradshaw was the quarterback in the only 2 games where the difference between his rating and the opponent’s allowed rating was greater than 60, but in those games, the margin of victory was only 4.
Outplaying the Losing Quarterback
Let’s also consider the relative performance of the winning quarterback to the losing quarterback and see whether it makes a difference. The raw data look like this. I’ve added in labels to show the losing quarterback who threw the most passes in the game. Asterisks indicate players who do not have 1,500 career passing attempts.
The median difference between Super Bowl-winning quarterbacks is highest for Joe Montana, who also has an impressive list of opponents. Because of the different quality of opponents, it’s probably not fair to look only at the difference between the winning and losing quarterbacks. After all, beating Ken Anderson by 4.8 points might be more impressive than beating Frank Reich by 80.3. As a measure of quality of the opponent, I want to include career quarterback rating in the analysis. League average quarterback ratings increased by about 0.5 every year between 1940 and 2007, so I’m making an adjustment to the career quarterback ratings of the losers.
Adjusted QBR = Losing quarterback’s career QBR – 0.5*(Year of Super Bowl – 1970)
It’s also worth considering how well the losing quarterback played. While Elway and Tarkenton had careers that got them to the hall of fame, their Super Bowl efforts from the chart above both resulted in quarterback ratings below 20. Outplaying a quarterback with a rating below 20 is something Jameis Winston has done every game of his career so far.
This 3-D scatterplot shows these variables:
- Win – Loss QBR: The difference between the winning and losing quarterback ratings for that Super Bowl
- Passer Rating Loser: The passer rating of the quarterback on the losing team
- Adjusted Career QBR Loser: The career quarterback rating of the losing quarterback
Where there are labels, they give the name of the losing quarterback and the number of the Super Bowl.
The ideal performance would be high in every variable and, again, there are no points clearly in that position. Montana and Bradshaw both outplayed quarterbacks by huge margins, but on days when those quarterbacks posted incredibly low ratings. Tom Brady played against quarterbacks who posted the highest passer ratings in their Super Bowls, but both those quarterbacks had higher passer ratings than he did. Terry Bradshaw’s victories over Roger Staubach turn out to be the most impressive victories in terms of the other quarterback’s career, but he didn’t outplay Staubach by as much as Montana outplayed Marino.
The highest median margin of victory belongs to Troy Aikman. The highest median difference between the winning and losing quarterbacks belongs to Joe Montana. I still contend that looking at these statistics by themselves wouldn’t let us make an easy decision about who the best Super Bowl quarterback is. Posting a better passer rating than John Elway sounds great, but it’s not as impressive when Elway’s passer rating is below 20. Winning the Super Bowl by 35 is impressive, but less so when it’s against a Buffalo team that wasn’t expected to be competitive before they lost their starting quarterback. Fortunately, Minitab gives us the power to do a multivariate analysis that considers all of these variables simultaneously. We’ll look at that type of analysis next time.