dcsimg
 

Why Anecdotal Evidence is Unreliable: The case against raspberry ketones and Dr. Oz

Photo of Doctor Mehmet OzMy wife occasionally watches The Dr. Oz Show and I’ve gotten drawn into it. I’ve noticed how Mehmet Oz frequently recommends dietary supplements for many conditions. And, he mentions that the show has a research department that checks everything out.

Previously, I’ve written about how scientists have a hard enough time assessing the benefits and risks of vitamins. So, I wondered, how was it possible to draw conclusions about supplements? After all, many of them haven’t had clinical trials with humans.

The final straw was when Dr. Oz called raspberry ketones "the number 1 miracle in a bottle to burn your fat." Dubious, I had to research it myself. It turns out that raspberry keytones have not been clinically tested with humans. In fact, a spokesman for the Dr. Oz Show told the Los Angeles Times:

“An adjective like 'miracle' is used as an editorial device to describe anecdotal results, as exemplified by the guests on our show. Our audience are not scientists, and the show needs to be more lively than a dry scientific discussion.”
  
The translation of this statement might as well be the more succinct version used by horoscopes and psychic hotlines: “For entertainment purposes only.” Basically, it’s a CYA (cover your assets!) disclaimer to protect themselves from anyone who happens to follows their unsubstantiated advice. The truth is that there is no scientific evidence to support these weight loss claims.
  
You’ve probably heard that you can’t trust anecdotal evidence. But do you know why?  I’ll show you!

The Lowdown on Anecdotal Evidence

Anecdotal evidence is essentially a story told by individuals. It often comes in the form of “I know a person who . . .,” but it can take many guises. In advertising, it’s often called a product testimonial. For example, someone takes a dietary supplement and claims to have lost a lot of weight. Or anecdotal evidence can be more personal, such as this classic example: someone doubts smoking is hazardous because they have a relative who smoked for decades and lived to a ripe old age.

Regardless of the form, you can’t trust anecdotal evidence! 

Statistics versus Anecdotal Evidence

In many ways, anecdotal evidence is the opposite of proper statistical and scientific methodology.

Statistical methodology Anecdotal evidence
Large, representative samples Very small, biased samples
Precise measurements in controlled situations Casual observations in uncontrolled circumstances
Measure/control all other relevant factors that affect the outcome Other important factors are unaccounted
Careful about making causal connections Causal connections are made too easily

By itself, the table should indicate the unreliability of anecdotal evidence -- but it’s actually worse! Anecdotal evidence is often cherry-picked to present the best case. Additionally, people are more likely to tell and to remember dramatic, extraordinary stories. And, if B follows A, our brains are wired to assume that A causes B. These dynamics bias the conclusions towards atypical outcomes with unwarranted causal connections. I could go on, but I think you get the idea! I’ll graphically illustrate the problems and show how statistics combats them.

Graphically Illustrating Anecdotal Evidence

Anecdotal evidence of weight loss

The graph above displays the anecdotal stories provided by a few people who took a hypothetical supplement. Looks pretty good, doesn’t it? They lost a lot of weight! However, we’re not getting the full story from this small, non-representative sample. As they say in the fine print, the results are not typical. For the next graph, let’s look at it using a proper, representative sample.

Representative sample of weight loss

This graph shows us the context that we were missing before. The red dots represent those who took the supplement (including those who provided the anecdotal stories) and the black dots represent those who did not take the supplement. The results are not so impressive here. True, some who took the supplement lost a lot of weight, but plenty are near the bottom. In fact, those who took the supplement appear to be randomly scattered throughout the distribution. Collectively, supplement takers lost no more weight than those who did not take the supplement.

For the anecdotal evidence, the individuals may have been cherry-picked for their stories, or perhaps they came forward with the stories on their own initiative. Either way, their results are not typical. Unfortunately, our minds are wired to deceive us. We tend to place a higher weight on dramatic, personal stories and to create causal connections. To get the true picture, you have to go beyond a handful of impressive stories and get the larger context that a representative sample provides.

How Statistics Overcome Anecdotal Evidence

In statistics, there are a two general approaches to determine whether a supplement promotes weight loss: observational studies and randomized controlled trials (RCTs).

In an observational study, researchers measure all relevant factors in a representative sample, and then create a statistical model to understand the role of each factor. For each person, you measure variables such as diet, exercise, basal metabolic rate, health, etc., and the consumption of dietary supplements. Once you factor in the role of everything else, you can determine whether the supplement is associated with weight loss. Anecdotal evidence gives you none of this vital, contextual information.

Another approach is through randomized controlled trials (RCTs). RCTs randomly assign subjects to control and treatment groups. This process helps assure that the groups are similar to each other when treatment begins. Therefore, any post-study differences between groups shouldn’t be due to prior differences. RCTs are more powerful than observational studies because they allow you to make causal conclusions about the effect of the treatment.

I’ve written about these in more detail:

Closing Thoughts on Anecdotal Evidence and Statistics

It aggravates me that Dr. Oz promotes the use of supplements based on anecdotal evidence. As a medical doctor, he must have some science in his background. He should know better. And there is real danger involved in taking supplements that have not been clinically tested in humans.

The raspberry ketones have not been tested in humans. This means that we just don't know if the supplement has any benefits. However -- and this should give you pause--we also don’t know whether raspberry ketones have negative side effects. After all, one pill is equivalent to a whopping 90 pounds of raspberries!

A lab rat in a rigorous scientific experiment may be at risk, but it is at least contributing to science. Anyone taking supplements based on anecdotal evidence is just placing themselves at risk.

If you find yourself being persuaded by anecdotal evidence, keep these thoughts in mind:

  • Results are not typical.
  • The risks are unknown!
  • For entertainment purposes only.

Have you ever been swayed by anecdotal evidence?

Fearless Data Analysis

Minitab 17 gives you the confidence you need to improve quality.

Download the Free Trial


 

Comments

Name: Mercedes • Friday, October 19, 2012

Hi Jim,

Excellent article. You are right, and it is amazing that doctors will recommend anything based on anecdotal data. Sometimes when we have a life threating disease, and there is no treatment for it, we may try something based on anecdotal data, but being aware of it, and because we do not have anything else. For weigt excess...diet and excersise are proven to be effective. However, they need effort and commitment. And we cannot sell it on a pill. If so, maybe more of this doctors will be promoting it vehemently.....


Name: David Moskowitz MD • Saturday, July 26, 2014

RCTs, unfortunately, cost a lot of money--money which is no longer available for clinical research since Big Pharma imploded into just a few companies, and the $14 trillion national debt slashed NIH spending. We're essentially in an era of no research money anymore, like the 1920s and 1930s, before there was an NIH. Medicine did manage to advance during these decades, thanks to the (anecdotal) case report and observational studies. The current refusal to consider anything but an RCT is an anti-innovative stance that hurts public health. Nor is an RCT consistent with a clinician's urge to do the best s/he can for each patient. Observational trials, with patients randomized simply to current vs. experimental treatment, may be the best we can hope for. RCTs may be hurting the research enterprise at this point, since there's no longer any money for them.


Name: Jim Frost • Monday, July 28, 2014

Hi Dr. Moskowitz,

It sounds like a bad situation for science. I didn't realize the severity of the current climate.

I agree that RCTs are more expensive but they provide more reliable results. Observational studies can also be important but if you don't control for an important variable, it can throw your results off completely. And, it's hard to know whether you're controlling for all of the important variables.

You mention observational trials with patients randomized by treatment. That's different than my understanding of observational studies where the assignment of subjects is outside the control of the investigator. An observational study usually assesses subjects who are already in the different treatment groups to see if there are differences.

I definitely agree that simply being able to randomly assign subjects to the treatment groups is a huge benefit by itself. I'd also not lump observational studies in with anecdotal evidence and would agree that observational studies are important, with several caveats.

I talk about these issues in several blog posts:

Random assignment:
http://blog.minitab.com/blog/adventures-in-statistics/use-random-assignment-in-experiments-to-combat-confounding-variables

Assessing the effects of vitamin supplements using an observational study:
http://blog.minitab.com/blog/adventures-in-statistics/statistics-that-affect-you-are-vitamin-supplements-really-harmful

Thanks for writing and adding your real world insight to the blog!
Jim


blog comments powered by Disqus