How to Get Problem Data Even with a “Gold Standard” Measurement System

As I mentioned before, accurate instruments won’t yield good data if you haven’t answered three fundamental questions that should precede every measurement system analysis. Whether you're doing a 6 Sigma project, or data analysis in support of a research project or some other goal, you need to be careful about how, when and where you gather the data.

Here’s how I learned that lesson the hard way.

I was working on a longitudinal study to determine whether impacts from jumping would promote bone density increases. The study called for subjects to jump from a height of 24 inches a specified number of times and, hopefully, produce an impact of 6 body weights. The first step was to do a pilot study to confirm that jumping from that height produces sufficient impact. 

Simple enough, or so I thought.  I’d have subjects jump on a Kistler force plate, which is the gold standard for measuring force in biomechanics research. Then, I’d use a 1-sample t-test to ensure that the average impacts were not different from 6 body weights.  I had the subjects all jump in one afternoon.  Then I plotted the data in time-order, which is generally a good practice.  I was shocked to find what appeared to be a negative trend over time.  Those who jumped later tended to have significantly lower impact forces.  This time series plot shows how the data appears to drop over time.

Time series plot of body weight impacts

It wasn’t a problem with the force plate.  Was it just a fluke?  We knew that there was variability in the landing forces, but we couldn’t assume it was a fluke if something systematic was occurring. Additionally, the lower values reduced the average impacts too much. Further investigation was required!

Now, before taking measurements, I asked some subjects to do test jumps with their shoes on and off, just to determine how they should jump for the pilot study. When they mentioned stinging on landing, I decided they should all jump in shoes for the pilot study.  Simply by chance, those subjects who tried a few jumps without shoes jumped later in the pilot study. Further investigation showed that these same subjects produced the lower scores.
The earlier landings, which stung, affected how they landed later, even when they wore shoes.  These subjects flexed their knees more upon landing to cushion the impact.  This pre-existing event, outside the pilot study, had completely thrown off the data!

I re-analyzed the data, but this time I changed hypothesis tests and used a 2-sample t-test in order to compare Group A that had always jumped with shoes to Group B that had jumped without shoes. This revealed a significant difference between groups, as shown in the plot below.  Additionally, those who always wore shoes had an acceptable average impact. Those who didn’t wear shoes had an unacceptable average.

Plot of body weight impacts by group

Even though I only recorded data while they all wore shoes, I had unintentionally introduced a significant grouping variable that affected the results before the pilot study even started!  Oops!

Whether you're a Six Sigma Black Belt, a researcher, or just a student of statistics, there are several important lessons you can take from this:

  • Be careful about all aspects of data collection
  • Graph your data
  • Investigate anomalies

The issues uncovered during this pilot study had further ramifications, as we shall see in a future post!

7 Deadly Statistical Sins Even the Experts Make

Do you know how to avoid them?

Get the facts >


blog comments powered by Disqus