In my previous post, I described how I was asked to weigh in on the ethics of researchers (DeStefano et al. 2004) who reportedly discarded data and potentially set scientific knowledge back a decade. I assessed the study in question and found that no data was discarded and that the researchers used good statistical practices.
In this post, I assess a study by Brian S. Hooker that was recently published with a blast of many social media stories that accompanied it. Hooker reanalyzed the DeStefano data and concluded that certain African American boys have a 340% increased risk of developing autism after receiving the MMR vaccination.The Scenario
After the study by DeStefano et al. was complete, the raw data was made available for other scientists to use. Hooker reanalyzed the data after being contacted by William Thompson, one of the authors in the original study. Thompson is a senior scientist at the CDC. Hooker’s study was published in the peer reviewed journal, Translational Neurodegeneration.
Thompson gave this statement to CNN: "I regret that my co-authors and I omitted statistically significant information in our 2004 article. I have had many discussions with Dr. Brian Hooker over the last 10 months regarding studies the CDC has carried out regarding vaccines and neurodevelopmental outcomes, including autism spectrum disorders. I share his belief that CDC decision-making and analyses should be transparent."
Brian Hooker and the social media stories allege that the original researchers deliberately excluded subjects to hide the increased risk for African American boys.
I personally don’t buy this theory because the DeStefano study analyzed the data from all subjects. The study compared the full results to a subanalysis of just the birth certificate sample, which had more complete data that included potential confounding variables. The two analyses agreed that the timing of the MMR vaccination did not affect the risk of developing autism.
Hooker is a scientific adviser for the Focus Autism Foundation, which believes that vaccines have helped cause an autism epidemic. He also has a 16-year-old son whom he describes as "vaccine-injured."
Hooker (2014)
Hooker used the DeStefano data. Where the two studies performed the same analyses, the results were the same. However, in general, Hooker used the data in a different manner and performed different analyses than DeStefano.
To derive his most startling conclusion, Hooker splits the data into two mutually exclusive groups:
- African-American children
- All children except African-American children
He then uses Chi-square analysis to look for an association between vaccination timing, gender, and the risk of autism within each of these groups. There are no significant results in the non-African American group at all.
Within the African American group, the only significant results are for the boys. The significant results are for African American boys vaccinated between:
- 18 and 23 months, who have a relative risk of 1.73
- 24 and 35 months, who have a relative risk of 3.36
The latter group is the one mentioned in the headlines that cite a 340% increased risk.
My Verdict on the Hooker Study
On the surface, it looks like there might be something to this study. However, there are problems lurking beneath the surface. The two major problems I see with Hooker’s study are:
- The full data set has been sliced and diced into a small, biased sample.
- Low birth weight is an uncontrolled, confounding variable
Sample size
To obtain the results, Hooker has to exclude data for all subjects except for the African American subjects. Even then, only African-American boys vaccinated at specific times were statistically significant.
Let’s look at the number of cases in the subgroup behind the shocking result at the heart of those social media stories and allegations of fraud—African American boys vaccinated between 24 and 35 months.
Consider the following:
- 70% of the sample was vaccinated at less than 18 months because the guideline is to get the MMR vaccination between 12 and 17 months. There were no significant results in this group.
- Only 7% of the sample was vaccinated during the time frame highlighted in the Hooker study.
- Further sample reductions are necessary because we’re only assessing African American kids (37% of the sample), who are boys (80%).
While neither study lists the number of autism cases for this super-specific sub-population, using the percentages and the number of cases, I can estimate that the shocking news of a “340% increase” is based on about 13 cases of autism!
This tiny sample size explains why the confidence interval, which measures the precision of the estimated risk, is so wide [1.5 to 7.51]. In this context, a 1 indicates no increase in risk, which is barely excluded from the CI.
Uncontrolled confounding variable
DeStefano used regression analysis to assess and control the effects of potential confounders. With regression analysis, he could study the effect of the various predictors (e.g., race, gender, birth weight) without having to subdivide the data. Instead, he included the predictors in the model to estimate the effects within the context of the full sample.
Hooker used Chi-squared analysis which cannot control for these confounders. That’s a huge problem for an observational study.
As for confounding variables, DeStefano found that low birth weights are associated with an increased risk of autism. In the original study, this and other potential confounders didn’t influence the uncontrolled results because they were evenly distributed across the groups of data. However, Hooker’s study sliced and diced the data so much that we can’t make this assumption.
Hooker wrote this about his highly pared down dataset: “It was found that there was a higher proportion of low birth weight African-Americans compared to the entire cohort.”
Hooker has a small data set in which a known confounder (low birth weight) is over-represented. Other studies have estimated that low birth weights can increase the risk of autism by 5 times! Because Hooker’s analysis does not control for this factor, we must assume that the estimated risk of autism is positively biased for this group. In other words, the estimated relative risk of 3.36 is likely higher than the true amount.
Thanks to the tiny sample and the uncontrolled confounding variable, Hooker’s results are both imprecise and biased. Consequently, my personal opinion is that Hooker’s results have no scientific value at all.
It turns out that Translational Neurodegeneration, the journal that published Hooker’s study, is having similar thoughts as well. During the course of writing this post, the journal has removed the article from its website and stated:
"This article has been removed from the public domain because of serious concerns about the validity of its conclusions. The journal and publisher believe that its continued availability may not be in the public interest. Definitive editorial action will be pending further investigation."
Closing Thoughts
In a previous post, Why Statistics is Important, I wrote that how you perform statistics matters, and there are many potential pitfalls. Dissecting the problems in Hooker’s study is the perfect illustration of this!
The CDC has stated that not all risk factors are known and further study is required. Because we’re dealing with our children, an abundance of caution is required. Therefore, it is worthwhile to continue to investigate the possible risk factors for autism. Even though I don’t think Hooker’s study has scientific merit, if race is a potential a risk factor, we should study it further.
As for the larger debate of vaccinations and autism, the consensus of the scientific literature overwhelmingly supports the view that vaccinations do not increase the risk of autism. This finding is evident in many studies that assess vaccinations and the eventual diagnoses of autism. Other studies have found that autism starts in utero, long before a child is given the MMR or any other vaccinations.