Keep that “Special Someone” Happy When You Perform Multiple Comparisons

Minitab Blog Editor | 26 September, 2011

Topics: Statistics

You know that using statistical quality analysis to ensure that your product meets requirements is a great way to keep your customers happy. But as you analyze your data, there’s another person you want to keep happy—a statistician.

As we saw in a previous blog, performing a hypothesis test repeatedly without controlling the family error rate increases the likelihood that any statistically significant results you find may be errors. And with each unplanned comparison you test, the frown on a statistician’s face deepens.

How can you turn the statistician's frown upside-down? By using a statistical analysis specifically designed to perform multiple comparisons, such as Minitab’s One-Way Analysis of Variance (ANOVA).

Multiple Flutters, Multiple Cures, Multiple Comparisons



Suppose you’re the world’s  leading medical researcher of synchronous diaphragmatic flutter, also known as “hiccups”. You want to compare 4 home remedies for hiccups from among the 78 “cures” you find listed online


Using a randomized study, you measure the amount of time that elapses until hiccups stop after applying each remedy. The results for each remedy are summarized by the Minitab boxplots below:



boxplot of cures
There don't appear to be any outliers that could unduly influence the test results. You wouldn't want to include extreme data due to special causes, such as Charles Osborne from Iowa, who fell down while slaughtering a hog and then hiccupped nonstop for almost 70 years, racking up over 430 million hiccups.

Offering money for hiccups appears to stop them the fastest. Drinking pickle juice and yelling "Awoga" seem to produce the slowest results. But are any of these differences statistically significant?

To find out, run Minitab’s Stat > ANOVA > One Way. Here's the output if you perform the analysis using the default settings:

ANOVA result
Because the p-value is less than alpha (0.05), at least one mean statistically differs from the others. But which means differ significantly?

To compare each pair of means, click the “Comparisons” button when you set up One-Way ANOVA. Then choose a multiple comparison procedure. Tukey’s, the first choice on top, works well for most situations. (For more information on each procedure, click the Help button on the Comparisons dialog box).
Comparisons dialog

Now when you run the analysis, the Minitab output shows you exactly which means are statistically different.
  Tukey results

Here, only the difference between Money (108.93 minutes) and Pickle juice (168.93 minutes) is statistically significant.



Finally, to make a statistician truly happy, think beyond the statistics



The statistical results showed that Money is a significantly faster hiccup remedy than Pickle juice. But the actual difference was only about 60 seconds, on average.

Is one less minute of hiccupping a big deal? And is the faster cure worth all the $20 bills you'd have to dole out to people who couldn't stop hiccupping right away?

“Awoga” is not significantly different than Money. But although yelling doesn't cost anything, it will probably annoy people a lot more than hiccups.

The Pencil Bite remedy also is not significantly different than Money. However, biting a pencil while you swallow water may significantly increase your risk of esophageal graphitus, also known PST (pencil-stuck-in-throat) Syndrome.

Personally, I’d take the pickle juice—or the hiccups.