How Effective Are Flu Shots?
Once again, with the arrival of autumn, it's time for a flu shot.
I get a flu shot every year even though I know they’re not perfect. I figure they’re a relatively easy and inexpensive way to reduce the chance of having a miserable week.
I’ve heard on various news media that their effectiveness is about 60%. But what does 60% effectiveness mean, exactly? How much does this actually reduce the chances that I’ll get the flu in any given year? I'm going to explore this and go beyond the news media simplification and present you with very clear answers to these questions. Quite frankly, some of the results were not what I expected.
We’ll Find Our Answers in Randomized, Controlled Trials (RCTs)
I’m a numbers guy. I use numbers to understand the world. My background is in research, so when I want to understand an issue, I look at the primary research. If I can understand the researchers’ methodology, the data they collect, and how they draw their conclusions, I’ll understand the issue at a deeper, more fundamental level than news reports typically provide.
To understand flu shot effectiveness, I’m only going to assess double-blind, randomized controlled trials, the gold standard. These studies are more expensive to conduct but provide better results than observational studies. (I discuss the differences between these two types of studies in my post about the benefits of vitamins.)
The two influenza vaccination studies I’ll look at satisfy the above criteria and are listed in a section of references for health professionals on the CDC’s website. Presumably these studies make a good case, using trusted data. Along the way, we’ll use Minitab statistical software to analyze their data for ourselves.
Defining the Effectiveness of Flu Shots
Flu shots contain vaccine for three influenza viruses that researchers predict will be the most common in a given flu season. However, plenty of other viruses (flu and otherwise) also are circulating and can make you sick. Many illnesses with flu-like symptoms are incorrectly attributed to the flu.
Consequently, the best studies use a lab to identify the specific virus that infects each of their sick subjects. These studies only count the subjects with confirmed cases of the three types of influenza virus. Effectiveness is defined as the reduction in these three influenza viruses among those who were vaccinated compared to those who were not vaccinated.
The Two Studies of the Flu Vaccine
It’s time to dig into the data! For me, this is where it gets exciting. You can hear about effectiveness on TV, but this is where it all comes from: counts of sick people in the experimental groups.
The Beran Study
The Beran et al. study^{1} assesses the 2006/2007 flu season and tracks its subjects from September to May. Subjects in this study range from 18-64 years old.
Treatment |
Flu count |
Group size |
Shot |
49 |
5103 |
Placebo |
74 |
2549 |
Because we want to compare the proportions between two groups, we’ll use the Two Proportions test in Minitab. To do this yourself, in Minitab, go to Stat > Basic Statistics > 2 Proportions. In the dialog, choose Summarized data and enter the data from the table above. Click OK, and you get the results below:
The p-value of 0.000 tells us that there is a significant difference between the two groups. The estimated difference between the vaccinated group and the placebo group is 1.9 percentage points. Because this is an RCT, it's fairly safe to assume that the vaccination caused the difference between the groups. However, outside of a randomized experiment, it's not wise to assume causality.
The vaccine effectiveness (or efficacy) is a relative reduction in risk between the two groups. You simply take the relative risk ratio of (vaccinated proportion/unvaccinated proportion) and subtract that from 1. We can get the proportion for each group from the Sample p column in Minitab’s output:
1 - (0.009602/0.029031) = 0.669
This study finds a 66.9% vaccine efficacy for the flu shot compared to the placebo.
The Monto Study
The Monto et al study^{2} assesses the 2007-2008 flu season and tracks its subjects from January to April. Subjects in this study range from 18-49 years old.
Treatment |
Flu count |
Group size |
Shot |
28 |
813 |
Placebo |
35 |
325 |
We’ll do the Two Proportions test again for this study. This time, enter the numbers from the above table into the dialog.
Again, the p-value indicates that there is a significant difference between the two groups. The estimated difference between the vaccinated group and the placebo group is 7.3 percentage points. Let's calculate the effectiveness:
1 – (0.034440/0.107692) = 0.680
This study finds a 68.0% vaccine efficacy for the flu shot compared to the placebo.
Conclusions So Far
We’ve looked at the data from two gold-standard studies and have drawn the same conclusions that you commonly hear on the news. Flu shots significantly reduce the number of influenza infections, and they are about 68% effective.
However, looking at the data and analyses myself, I have new insights. Specifically, the low number of influenza cases in the placebo group for each study caught my eye, and that’s what we’re looking at next.
What It Means for You: Relative versus Absolute Risk
If you’re like me, the 68% effective statistic isn’t too helpful. The problem is that it is a relative comparison of risk, not an absolute assessment of risk. To illustrate the difference, consider which type of assessment is more useful:
- Relative assessment: Your car is travelling half as fast as another car, but you don’t know the true speed of either car.
- Absolute assessment: Your car is travelling at 30 MPH and the other car is travelling at 60 MPH.
Clearly, #2 is much more useful. Similarly, it would be more helpful to know the absolute risk of catching the flu if you get the shot versus not getting it!
Vaccine effectiveness is a relative risk
Vaccine effectiveness doesn’t tell you the exact risk of catching the flu for either group. Instead, it involves dividing one proportion by the other for the relative risk. In fact, as you should recall, effectiveness is the inverse of the relative risk, which makes it even harder to interpret. 67% effectiveness indicates that a vaccinated person has one-third the risk of contracting the flu as a non-vaccinated person.
Unfortunately, using these numbers, we don’t know the absolute risk for anyone!
The group proportions are the absolute risks
We can estimate the absolute risk from the studies by looking at the proportion for each group in the Minitab output, and subtracting to calculate the absolute reduction. I’ll summarize this information below as percentages and even add in the results for two more flu seasons from another study that the CDC references (Bridges et al.^{3}):
Flu season |
Placebo |
Flu Shot |
% Point Reduction |
1997/98 |
4.4 |
2.2 |
2.2 |
1998/99 |
10.0 |
1.0 |
9.0 |
2006/07 |
2.9 |
1.0 |
1.9 |
2007/08 |
10.8 |
3.4 |
7.4 |
Average |
7.0 |
1.9 |
5.1 |
Notice how the risk of getting the flu varies by flu season? The differences are not surprising because the studies use different samples and the flu seasons have different influenza viruses.
So let’s look at the average of these four flu seasons. If you aren’t vaccinated, you have a 7.0% chance of getting the flu. However, if you do get the flu shot, your risk is about 1.9%, which is a reduction of 5.1 percentage points.
Hmm. The "5.1% reduction" doesn’t sound nearly as impressive as the "67% effectiveness!" Both statistics are based on the same data, but I think the estimate of absolute risk is a more useful way to present the results.
Closing Thoughts about the Flu Shot Data
I was surprised by the results. While I knew flu shots were not perfect, I always got them because I thought they reduced my risk by more than what the CDC recommended studies actually show. Even if you aren’t vaccinated, your risk of getting the flu isn’t too high.
That probably explains why a number of people have told me that while they never get flu shots, they can’t remember having the flu!
These more subtle results made me wonder about flu vaccinations on a societal scale. Could the flu vaccine possibly reduce flu cases enough to save sufficient money (lost workdays, doctor and drug costs, etc) to pay for the vaccinations?
Bridges et al. conducted a cost-benefit analysis in their study. For the two flu seasons where they tracked flu vaccinations, infections, and expenditures, the vaccinations actually increase net societal costs. It would’ve been cheaper overall not to get vaccinated!
In light of this, I wasn’t surprised when I read an article on CNN.com that said, outside of the U.S. and Canada, other countries do not strongly encourage all of their citizens above 6 months to get the flu shot. According to the article, “global health experts say the data aren’t there yet to support this kind of vaccination policy, nor is there enough money.”
I understand this viewpoint better now.
However, I’m not trying to talk anyone out of getting a flu shot. I’m on the fence myself. While the risk of getting the flu in any given year is fairly small, if you regularly get the flu shot, you’ll probably spare yourself a week of misery at some point! You should always consult a medical professional to determine the best decision for your specific situation.
In another post, I look at the long-term benefits of flu vaccinations.
References
1. Beran J, Vesikari T, Wertzova V, Karvonen A, Honegr K, Lindblad N, Van Belle P, Peeters M, Innis BL, Devaster JM. Efficacy of inactivated split-virus influenza vaccine against culture-confirmed influenza in healthy adults: a prospective, randomized, placebo-controlled trial. J Infect Dis 2009;200(12):1861-9
2. Monto AS, Ohmit SE, Petrie JG, Johnson E, Truscon R, Teich E, Rotthoff J, Boulton M, Victor JC. Comparative efficacy of inactivated and live attenuated influenza vaccines. N Engl J Med. 2009;361(13):1260-7
3. Bridges CB, Thompson WW, Meltzer MI, Reeve GR, Talamonti WJ, Cox NJ, Lilac HA, Hall H, Klimov A, Fukuda K. Effectiveness and cost-benefit of influenza vaccination of healthy working adults: A randomized controlled trial. JAMA. 2000;284(13):1655-63
Name: Paul S • Thursday, January 24, 2013
Jim, there are a couple of aspects not considered. Infection rates are dependent on how much flu is circulating. Another issue that bothers me with these hard statistical test of medicines: I have no way of knowing if my response to a medicine will be above or below the average used in the tests.
Name: Jim Frost • Friday, January 25, 2013
Hi Paul, thanks for reading! Those are good comments.
I've taken the infection rate in the placebo group to represent the amount of virus that is circulating and it does vary by flu season. Relatedly, the absolute risk reduction does vary depending on how much flu virus is circulating. Unfortunately, you won't know that until after the flu season is over.
The issue of response variability is true with all research. There is always variability that muddies the picture. In this case, as with most medicine, it's hard to know how each individual will respond. Overall, the research shows what the typical response is. Research can also identify groups that have a different response. For example, it is known that individuals over the age of 65 tend to have a lessened response to the flu vaccine.
Identifying a personal response beforehand is unusual. I have heard that for warfarin, the anticoagulant, there is a genetic test to determine an individual's specific dosage. Perhaps this is the way of the future? Until then, we'll have to rely on group tendencies.
Jim
Name: Peter Bruce • Saturday, January 26, 2013
Very interesting - I think we shared this at our Facebook page (statistics.com). But is there an error (decimal place) in your first calculation of vaccine efficacy -- should be 0.09xxx instead of 0.009xxx?
Name: Jim Frost • Monday, January 28, 2013
Hi Peter,
The value for the flu shot group is correct. However, I did have a typo for the placebo proportion. That value should be 0.029031. However, I did use the correct value when I calculated the efficacy. So, it's still 66.9%. Thanks for catching this. I've fixed the proportion.
Thanks for reading and sharing this!
Jim
Name: Stephanie • Monday, February 18, 2013
I found this article very interesting, but did find one interpretation error:
"So let’s look at the average of these four flu seasons. If you aren’t vaccinated, you have a 7.0% chance of getting the flu. However, if you do get the flu shot, your risk is about 1.9%, which is a reduction of 5.1%."
A reduction from 7.0% to 1.9% is a reduction of more than 350% (or alternatively 5.1 percentage points).
Name: Jim Frost • Monday, February 18, 2013
Hi Stephanie,
Your calculations are another form of relative risk. And, both measures (relative risk and absolute risk) are accurate. However, relative measures can be difficult to interpret without additional information. You gain valuable contextual information by also looking at it in absolute terms.
To illustrate this, suppose you have two pairs of cars travelling at different speeds:
1) 200 mph vs 100 mph
2) 60 mph vs 30 mph
For both pairs, the relative reduction is 50%. You get the same number for two very different cases, which isn't so helpful. However, if you put that in their absolute terms, suddenly the picture is more clear! And, that's what I wanted to do with the flu shot data.
Thanks for reading!
Jim
Name: Nat Philosopher • Sunday, September 14, 2014
If you are going to compute some kind of expected gain/loss, you are going to have to fold in some estimate of the likelihood the flu shot is doing you long term damage in other ways.
For example, this placebo controlled study:
http://www.ncbi.nlm.nih.gov/pubmed/22423139
followed people who got a flu vaccine or a placebo for 9 months, and saw no significant difference in how much flu they got, but the people who got the vaccine had 4 times as much other respiratory illnesses as the ones who got the placebo.
Then, there are studies indicating the H1N1 vaccine a few years back gave a lot of kids narcolepsy.
And a lot of the flu shots have mercury.