The other day I received a request from a friend to look into a new study in a peer reviewed journal that found a link between MMR vaccinations and an increased risk of autism in African Americans boys. To draw this conclusion, the new study reanalyzed data that was discarded a decade ago by a previous study.
My friend wanted to know, from a statistical perspective, was it unethical for the original researchers to discard the data for African Americans? Did they set scientific knowledge back a decade?
To answer these questions, I looked at the 2004 study, their decision to discard the data, and the reanalysis of this data in 2014.
I clicked the link that my friend sent and saw this headline: “Fraud at the CDC uncovered, 340% risk of autism hidden from public.” Yikes! Apparently variations on this story are making the rounds via social media. Here is the gist of the story.
In 2004, a major study by DeStefano et al.1 determined that there was no connection between the timing of measles-mumps-rubella (MMR) vaccinations and autism. This study was conducted by researchers at the Center for Disease Control (CDC). The CDC presents this study as a key piece of evidence in the debate about vaccinations and autism.
In 2014, Brian S. Hooker2 reanalyzed the 2004 data and found a possible link between MMR vaccinations and the risk of developing autism for a very specific group. African American boys who had their first MMR vaccination between 24 and 35 months of age were 3.4 times more likely to develop autism.
The fraud allegation is made by anti-vaccination groups who say that the CDC applied political pressure to discard the data in order to high these explosive findings.
I’m very aware that this specific case is just a part of a larger, very controversial issue. For this post, I assess the statistical validity of these two studies, and the decision to discard data.
I’m in favor of reanalyzing previous studies. I think it’s great if a later researcher can draw more or better conclusions from a data set. However, all studies must be statistically valid, or their results can be misleading.
Observational Studies of Vaccination and Autism
The original dataset was collected for an observational study. Because both studies use the same data set, they’re both observational studies. To evaluate the validity of the two studies, we need to understand the differences between observational studies and randomized control trials (RCTs),
In an RCT, all subjects are randomly assigned to the treatment and control groups. This process helps assure that the groups are similar to each other when treatment begins. Therefore, any post-study differences between groups are probably due to the treatment rather than prior differences.
For observational studies, there is no random assignment, which increases the chances that the treatment and control groups are not equivalent at the beginning of the study. Consequently, differences in the outcome at the end could be due to the preexisting differences (confounding variables). If the analysis does not account for the confounding factors, the results will be biased.
These issues are crucial in evaluating these two studies. For more information, read my posts about confounding variables, random assignment, and an in depth look at an observational study about the effects of vitamins.
DeStefano et al. (2004)
In one corner we have the original DeStefano (2004) study that agrees with the scientific consensus and concludes there is no connection between MMR vaccinations and autism risk. Researchers compare 624 children with autism, age 3 to 10, with 1,824 developmentally healthy children. Most of the children were vaccinated between 12 and 17 months of age in accordance with vaccination recommendations.
The first thing that I noticed is that DeStefano et al. did not actually exclude data as the critics claim. The claim is that the study discarded data from the subjects who could not be linked to birth certificates in order to hide the findings about African-American boys.
There are three issues to consider here:
One, the non-birth-certificate group was not based on race, but on whether the subject was matched to a Georgia birth certificate. African-American kids were actually underrepresented in the non-birth-certificate group. The birth certificate group was made up of 37.9% African Americans while the supposedly excluded non-birth-certificate group contained only 32.2%.
Two, while making the distinction between kids based on birth certificates may seem arbitrary, there are good data reasons for doing so. For those kids who could be matched to Georgia birth certificates, the researchers were able to gather more complete information on possible risk factors for autism, such as the child’s birth weight, mother’s age, and education. This information was not available for children who could not be matched to Georgia birth certificates.
Because this is an observational study, it’s crucial to statistically control for these risk factors. Just having more data isn't always a good thing. You need to record and analyze the potential confounders.
Three, the DeStefano study did not exclude the kids who were not linked to birth certificates. Instead, the study performed an analysis on the full sample (subjects linked and not linked to birth certificates) and a separate analysis of those linked to a birth certificate. So, no data was actually excluded. The results of the two analyses were in agreement.
The study found that some of the additional variables in the birth certificate sample are potential confounders because they are associated with autism risk. The risk factor that becomes important for this post is that low birth weight is associated with a higher risk of autism. We come back to that in the Hooker study.
Using regression analysis, the study concluded that the timing of MMR vaccinations is not associated with the risk of developing autism.
My Verdict on the DeStefano Study
The criticism that the study discarded data from African American subjects just doesn’t hold water. No data was discarded. For the subjects who were linked to birth certificates, the researchers performed additional analyses. In this light, I see a careful observational study that assessed the role of potential confounders.
The biggest weakness that I see for this study is that the researchers could not compare subjects who were vaccinated for MMR to those who were not vaccinated at all. The authors wrote that they “lacked an unvaccinated comparison group.” The truth is that the vast majority of kids are vaccinated. Consequently, this study compared the distribution of vaccination ages for case and control children to see if the timing impacted the risk of autism. It didn’t.
DeStefano et al. cite a large Dutch study that did include an unvaccinated group of nearly 100,000 subjects. This study found no increased risk for those who were vaccinated compared to the completely unvaccinated group.
Stay tuned! Tomorrow I’ll take a look at Hooker’s reanalysis of the DeStefano data.
1. DeStefano, Frank, Tanya Karapurkar Bhasin, William W. Thompson, Marshalyn Yeargin-Allsopp, and Coleen Boyle, Age at First Measles-Mumps-Rubella Vaccination in Children with Autism and School-Matched Control Subjects: A Population-Based Study in Metropolitan Atlanta, Pediatrics, 2004;113;259
2. Hooker, Brian S., Measles-mumps-rubella vaccination timing and autism among young African American boys: a reanalysis of CDC data, Translational Neurodegeneration 2014, 3:16