Why Anecdotal Evidence is Unreliable: The Case Against Raspberry Ketones and Dr. Oz

Minitab Blog Editor 06 September, 2012

Topics: Statistics in the News

Photo of Doctor Mehmet OzMy wife occasionally watches The Dr. Oz Show and I’ve gotten drawn into it. I’ve noticed how Mehmet Oz frequently recommends dietary supplements for many conditions. And, he mentions that the show has a research department that checks everything out.

Previously, I’ve written about how scientists have a hard enough time assessing the benefits and risks of vitamins. So, I wondered, how was it possible to draw conclusions about supplements? After all, many of them haven’t had clinical trials with humans.

The final straw was when Dr. Oz called raspberry ketones "the number 1 miracle in a bottle to burn your fat." Dubious, I had to research it myself. It turns out that raspberry ketones have not been clinically tested with humans. In fact, a spokesman for the Dr. Oz Show told the Los Angeles Times:

“An adjective like 'miracle' is used as an editorial device to describe anecdotal results, as exemplified by the guests on our show. Our audience are not scientists, and the show needs to be more lively than a dry scientific discussion.”
The translation of this statement might as well be the more succinct version used by horoscopes and psychic hotlines: “For entertainment purposes only.” Basically, it’s a CYA (cover your assets!) disclaimer to protect themselves from anyone who happens to follows their unsubstantiated advice. The truth is that there is no scientific evidence to support these weight loss claims.
You’ve probably heard that you can’t trust anecdotal evidence. But do you know why?  I’ll show you!

The Lowdown on Anecdotal Evidence

Anecdotal evidence is essentially a story told by individuals. It often comes in the form of “I know a person who . . .,” but it can take many guises. In advertising, it’s often called a product testimonial. For example, someone takes a dietary supplement and claims to have lost a lot of weight. Or anecdotal evidence can be more personal, such as this classic example: someone doubts smoking is hazardous because they have a relative who smoked for decades and lived to a ripe old age.

Regardless of the form, you can’t trust anecdotal evidence! 

Statistics versus Anecdotal Evidence

In many ways, anecdotal evidence is the opposite of proper statistical and scientific methodology.

Statistical methodology Anecdotal evidence
Large, representative samples Very small, biased samples
Precise measurements in controlled situations Casual observations in uncontrolled circumstances
Measure/control all other relevant factors that affect the outcome Other important factors are unaccounted
Careful about making causal connections Causal connections are made too easily

By itself, the table should indicate the unreliability of anecdotal evidence -- but it’s actually worse! Anecdotal evidence is often cherry-picked to present the best case. Additionally, people are more likely to tell and to remember dramatic, extraordinary stories. And, if B follows A, our brains are wired to assume that A causes B. These dynamics bias the conclusions towards atypical outcomes with unwarranted causal connections. I could go on, but I think you get the idea! I’ll graphically illustrate the problems and show how statistics combats them.

Graphically Illustrating Anecdotal Evidence

Anecdotal evidence of weight loss

The graph above displays the anecdotal stories provided by a few people who took a hypothetical supplement. Looks pretty good, doesn’t it? They lost a lot of weight! However, we’re not getting the full story from this small, non-representative sample. As they say in the fine print, the results are not typical. For the next graph, let’s look at it using a proper, representative sample.

Representative sample of weight loss

This graph shows us the context that we were missing before. The red dots represent those who took the supplement (including those who provided the anecdotal stories) and the black dots represent those who did not take the supplement. The results are not so impressive here. True, some who took the supplement lost a lot of weight, but plenty are near the bottom. In fact, those who took the supplement appear to be randomly scattered throughout the distribution. Collectively, supplement takers lost no more weight than those who did not take the supplement.

For the anecdotal evidence, the individuals may have been cherry-picked for their stories, or perhaps they came forward with the stories on their own initiative. Either way, their results are not typical. Unfortunately, our minds are wired to deceive us. We tend to place a higher weight on dramatic, personal stories and to create causal connections. To get the true picture, you have to go beyond a handful of impressive stories and get the larger context that a representative sample provides.

How Statistics Overcomes Anecdotal Evidence

In statistics, there are a two general approaches to determine whether a supplement promotes weight loss: observational studies and randomized controlled trials (RCTs).

In an observational study, researchers measure all relevant factors in a representative sample, and then create a statistical model to understand the role of each factor. For each person, you measure variables such as diet, exercise, basal metabolic rate, health, etc., and the consumption of dietary supplements. Once you factor in the role of everything else, you can determine whether the supplement is associated with weight loss. Anecdotal evidence gives you none of this vital, contextual information.

Another approach is through randomized controlled trials (RCTs). RCTs randomly assign subjects to control and treatment groups. This process helps assure that the groups are similar to each other when treatment begins. Therefore, any post-study differences between groups shouldn’t be due to prior differences. RCTs are more powerful than observational studies because they allow you to make causal conclusions about the effect of the treatment.

I’ve written about these in more detail:

Closing Thoughts on Anecdotal Evidence and Statistics

It aggravates me that Dr. Oz promotes the use of supplements based on anecdotal evidence. As a medical doctor, he must have some science in his background. He should know better. And there is real danger involved in taking supplements that have not been clinically tested in humans.

The raspberry ketones have not been tested in humans. This means that we just don't know if the supplement has any benefits. However -- and this should give you pause--we also don’t know whether raspberry ketones have negative side effects. After all, one pill is equivalent to a whopping 90 pounds of raspberries!

A lab rat in a rigorous scientific experiment may be at risk, but it is at least contributing to science. Anyone taking supplements based on anecdotal evidence is just placing themselves at risk.

If you find yourself being persuaded by anecdotal evidence, keep these thoughts in mind:

  • Results are not typical.
  • The risks are unknown!
  • For entertainment purposes only.

Have you ever been swayed by anecdotal evidence?