As we broke for lunch, two participants in the training class began to discuss, debate, and finally fight over a fundamental task in golf —how to drive the ball the farthest off the tee. Both were avid golfers and had spent a great deal of time and money on professional instruction and equipment, so the argument continued through the lunch hour, with neither arguer stopping to eat. Several other class participants chimed in with approval or disagreement over the points being made on both sides.
I thought the problem could be solved with a designed experiment. Back at Minitab, we found avid golfers who were very much interested in the problem and willing to participate and have data collected on their drives.
This was not going to be simple. I talked to other statisticians who had attempted to determine how to drive the longest ball from the tee. Some told me they had failed due to high process variation. Others found the variables they selected did not have a measurable impact.
It was easy for me to relate this back to solving process engineering problems. The difficult ones have many of these characteristics in common, including a complex process, noise variables beyond the researcher’s control, limited resources to run the experiment under the necessary conditions, experts with competing theories, measurement variability making it difficult to measure both responses and inputs, and process variation causing different response values even when repeating the same run conditions.
Teeing Up to Characterize the Process and Define the Problem
The first step is to use process experts, literature or past experiments to characterize the process and define the problem. Based on contributions we solicited from experienced golfers, a Penn State golf coach, other statisticians who had tackled similar problems, and all the Internet had to offer, we were able to assemble a list of potential inputs for flight distance (Carry), rolling distance (Roll), and the corresponding total drive distance (Total), and classify them into four categories according to how they will be managed in our experiment. Let’s walk through each of the four groups.
1. Blocking
There is only one input in this category: the golfer.
Each golfer brings a unique style, swing, and athleticism to the game. Because each data point can be traced back to an individual golfer, the variability between golfers will be handled using a technique called blocking. The data from each golfer will be standardized according to the average driving performance of that golfer compared to the other golfers in the study.
Blocking essentially handicaps each golfer by their average distance so that all the data can be combined into one analysis without concern about golfer-to-golfer variation. When you set up an experiment, blocking allows you to take advantage of all the resources available to you (three manufacturing lines, two measurement technicians, etc.) without concern for the variability from block to block. Our experimental design and analysis will block on the golfer so we can take advantage of using several different golfers in the experiment.
2. Analysis of Covariance (Covariates)
The two inputs in this category are club speed and the contact point on the club face with respect to the center of the club face. They are noise variables we cannot control from drive to drive, but have a strong effect on our responses. The important distinction from other noise variables is that we can measure them on each drive. By establishing the average linear relationship between these covariates and our responses, we can mathematically adjust each Carry and Roll measurement for club speed and club/ball contact location for that drive.
By treating club speed and club/ball contact location as covariates, we can greatly reduce the level of background noise in our experiment data, thus giving us a clearer estimate of the effects of the experimental factors we are studying. Nearly all processes have noise variables that cannot be controlled and contribute to the overall process variation. A key experiment design goal for your study should be to plan to measure these variables during each run so that their effect can be removed from the background variability. This will reduce the overall level of noise you have to deal with in the final analysis.
3. Noise
Noise variables add to the background variability of our data. This variability can obscure our ability to measure the true effect of our research variables, which are the ones of interest to us. Imagine seeing Michael Jordan play basketball on the rare occasions he had a bad game. If you'd never seen him play, you could easily think “He’s not very good.” Of course, you would have reached the wrong conclusion, even though you used data to make your decision, because your data was affected by background variability. In our DOE, we want to minimize the background variability in order to decrease the probability of making an incorrect conclusion.
Planning to control, limit, account for, and finally measure the variability in your data is key to your experiment’s success. In your final analysis, your estimate of the background process variation will be in the calculation of every test statistic in your analysis. You have to work hard to get it right!
These are our Noise inputs:
- Club path and position on contact such as angle, arc and shaft flex
- Golfer’s grip strength on the club
- Ground surface conditions (incline, firmness, grass height, dampness, etc.)
- Air temperature and humidity
- Wearing expensive shorts with a Bubba Watson logo
- Golfer’s arm length
- Club head weight and club length
- Ball spin rate and direction coming off the club
Club path and position on contact and/or ball spin rate and direction coming off the club can result in a very bad drive. We can minimize the impact on our results by discarding any drive that is an obvious slice into the trees. Likewise, as experimenters, we should be attentive to measurements that are strongly influenced by noise variables and should exclude those measurements as outliers. Analysis of residuals is a powerful tool to detect outliers (in a later post), but the best time to identify an outlier is when the sample is made or measured. This way, the extraneous circumstances leading to the outlier can be immediately noted and considered in the decision to remove the data point.
Club head weight and club length could have been factors in this experiment, but the cost of the additional drivers required to study these club properties would have been prohibitive. They are also standardized fairly well, and are not something every golfer can change on demand, so they were kept constant in our experiment. In your own process experiments, you will have to use your engineering knowledge and practical considerations to determine which inputs should be held constant.
A noise variable such as grip strength on the club, will vary from drive to drive within a golfer and this change in grip strength cannot be easily measured. Therefore, grip strength variability will be one of many sources contributing to the overall process variation that we will have to contend with in our experiment. For such variables, the best approach is to remind the experimenters of the importance of consistency in their performance. For your process experiment, standardizing procedures and protocols where possible, along with requiring consistency within an experimenter throughout the study, is a good way to lower your background process variation.
The amount of unexplained process variability (error) will have to be accurately measured so that it can be compared to the size of the experimental factor effects in the final analysis. This comparison allows us to determine which effects are much larger than the level of error in the data — or in other words, which effects are statistically significant.
Also, in the experiment design phase, we will have to estimate the level of process variation we expect to see in the experiment so that the number of measurements (sample size = N) needed to detect the factor effects in midst of the noise can be calculated. This calculation, known as power and sample size, will be illustrated in a future post.
4. Experiment Design for Factors
Now that we have a good handle on our process variation, we can move forward to the research variables (factors) in our experiment: tee height, shaft stiffness, ball quality, and club tilt angle.
One commonly held theory is that greater tilt angle will give higher loft for longer Carry. However, the velocity spent going up is not going forward and a longer “hang time” means that air resistance has more time to have a negative effect. In addition, the sharp angle of descent from high loft will lower the Roll. But does the tee height result in higher or lower loft? How do the shaft stiffness and ball quality affect the velocity of the ball off the tee? Of course a faster ball is going to go further.
In our experiment, as with your process, there will be many competing theories about how to reach the end goal. What levels of the factors should be tested and in what combinations to get to the right solution? What is the smallest number of tests we can run while still achieving an accurate and comprehensive understanding of our process (including a way to optimize the responses)?
In the next post, I’ll discuss the different options for factor level settings/combinations for our four factors and ultimately, determine our basic experiment plan. Many thanks to Toftrees Golf Resort and Tussey Mountain for use of their facilities to conduct our golf experiment.
Catch Up with the other Golf DOE Posts:
Part 2: 5 Reasons Factorial Experiments Are So Successful
Part 3: Mulligan? How Many Runs Do You Need to Produce a Complete Data Set?
Part 4: ANCOVA and Blocking: 2 Vital Parts to DOE
Part 5: Concluding Our Golf DOE: Time to Quantify, Understand and Optimize