Learning Design of Experiments with Paper Helicopters and Minitab

This article was written by guest blogger Matthew Barsalou, Statistical Problem Resolution Master Black Belt and Engineering Quality Expert at BorgWarner Turbo Systems Engineering GmbH with over 15 years of experience in quality. 

The late statistician George E. P. Box, along with Soren Bisgaard and Conrad Fung, used a paper helicopter to teach statistics. The idea originated with Kip Rogers of Digital Equipment and is useful for demonstrating factorial designs. Decades after Box, Bisgaard and Fung’s publication, the DOE helicopter has become a useful staple of DOE training.

The paper helicopter provides a way to quickly explain basic DOE concepts. It also offers an easy-to-do experiment you can analyze using Minitab.

We can’t just declare we want a high-quality helicopter. Quality must be clearly defined.

To perform a DOE with a paper helicopter we need to identify the desired output, which would be our response variable.

A good helicopter is one which stays in the air for a longer time, so the response variable would be flight time as measured from the time the helicopter is dropped from a height of 2 meters until the time it hits the floor. Without defining the test conditions it could be possible that sample helicopters would be dropped from different heights, in which case our DOE results would be not be valid.

Test factors that influence flight time must also be identified. For the helicopter experiment, the factors could be paper type, rotor length, leg length, leg width and paper clip. The helicopter experiment levels are varied by using two different types of paper, using longer or shorter leg and rotor lengths and adding or removing a paper clip.

Here’s how to make the paper helicopters.

• Step 1: Cut the paper to a width of 5cm.
• Step 2: Cut the paper the length of paper rotor length plus leg length, and add 2 cm for the body.
• Step 3: Cut dotted lines at Leg A and Leg C. The length of each cut is 5 cm minus leg width divided by 2.
• Step 4: Fold leg A onto leg B.
• Step 5: Fold leg C onto leg B.
• Step 6: Fold rotor A and rotor B in opposite directions. They should form 90° to the body and be 180° away from each other.
• Step 7: For the paper clip version: Add a paper clip to the bottom of the leg

Table 1: Helicopter factors

Designing the Experiment

Statisticians and Six Sigma black belts should know how to set up and perform the calculations in a designed experiment by hand. However, computer programs make DOE a much simpler task, particularly for people who need to perform experiments only occasionally.

To create a fractional factorial design in Minitab Statistical Software, go to Stat > DOE > Factorial > Create Factorial Design where we can select the desired design.

For this experiment, we will use a 2-level factorial which can handle anywhere from 2-15 different factors. To select the desired design in Minitab, select 5 for the Number of factors, then click Designs to select the desired design and resolution level.

Resolution is the degree to which effects are aliased with other effects. In other words, aliased effects are mixed and can’t be estimated separately. This can also be referred to as confounding, and it results from not testing every possible combination of factors.

This is a disadvantage of a fractional factorial design. However, not testing every possible combination can be a significant advantage in time and expense over a full factorial design. If you’re not sure what resolution you should use, click on Display Available Designs… to see a list of designs and resolutions.

We typically use three levels of resolution: Resolution III, IV and V. There is no confounding of main effects with each other in these three resolution types; however, in a Resolution III design, main effects will be confounded with 2-factor interactions. This is a problem because 2-factor interactions are quite common in practice. Resolution IV designs do not have 2-factor confounding with main effects, but 2-factor interactions are aliased with other 2-factor interactions, and main effects are confounded with 3-factor interactions.

We try to use Resolution IV designs instead of Resolution III designs when possible because they have less aliasing, but still require fewer experimental runs than higher resolution experiments.

Resolution V designs have the added advantage that no 2-factor effects are confounded with other 2-factor effects; however, 2-factor effects are aliased with 3-factor effects, and main effects are aliased with 4-factor effects. This is a good thing, as 3-factor and above interactions are rarely significant in practice.

The confounding problem can be eliminated by performing a full factorial design. However, this requires more experimental runs, which might be prohibitive in terms of both time and money.

Looking at the Display Available Designs… option in Minitab, we can conduct a fractional factorial experiment using either a Resolution III or a Resolution V design for the 5-factor helicopter experiment. A Resolution III design would only need 8 runs, but because of the extreme confounding, the Resolution V design that requires 16 test runs is the better option. Click on Designs… and select the desired design.

As you set up the experiment, Minitab also asks for the number of blocks. Blocks are simply homogenous groupings of measurements that can be used to account for variation. The default value is one because, ideally, everything is homogenous.

The helicopter experiment will be set up so that there is only one experimental block. Each type of paper will come from the same source. The helicopters will all be built by the same person using the same scissors and ruler. If we used paper clips from two manufacturers or had some other potential causes for variation, then we would need separate blocks. Fortunately, this is not the case.

After you select your design, click Factors to enter the names and levels of the variables in your experiment. To change the name of a factor, type the name of the factor over the letter in the name field. The factor settings can also be renamed by replacing the default values of -1 and 1 with the actual factor levels.

When you’ve completed the dialog box, Minitab creates the experimental design and displays it in a Minitab worksheet. The Session Window above the worksheet provides a description of the selected design with the resulting alias structure.

In the resulting Minitab worksheet, the experimental results are entered into column C10. We can name the column “Flight time” because that is our experimental response variable.

A randomized run order is provided in the “RunOrder” column. Without randomization there is a risk that the experimental results will reflect unknown changes in the test system over time – such as if the scissors grow dull, resulting in slightly different cuts.

Minitab’s default setting for a designed experiment is one replicate. If you observe a lot of variation in the process or the resulting measurements, you can use Stat > DOE > Modify Design to add replicates to your design. Suppose the person making the helicopters had difficulty cutting a straight line so all edges are not uniform; the differences in results may reflect this variation. Replicating runs minimizes the effects of this kind of unanticipated variation.

Gathering the Experimental Data

Variability can have a major impact on experimental results, so take steps to reduce the variability. Have the same person make all helicopters and have them use the same pair of scissors and ruler. Drop the helicopters from a height of 2 meters, and identify the drop point clearly to ensure consistency. A higher or lower starting point would affect flight time, and this could throw off the results. The helicopters must also be held and released the same way, or variation in our data might be the effect of the release method and not the design of the helicopter.

The Minitab worksheet below contains the experimental results listed under “Flight time” in column C10.

Analyzing the Data

After running the experiment and entering the collected data in the Minitab worksheet, select DOE > Factorial > Analyze Factorial Design…

Significant factors are those that influence the response as they changed from one setting to another. When you click OK, Minitab provides an ANOVA table as well as a Pareto chart of effects, which make it very easy to identify significant factors.

In an ANOVA table, those factors with a p-value less than 0.05 are statistically significant. However, the ANOVA table for this model doesn’t include any p-values!

With all our factors included in the model, we have no degrees of freedom left for Error, and you need at least 1 degree of freedom to calculate p-values. But while we can’t accept this model based on the ANOVA results, we can use the normal plot or Pareto chart to identify factors and interactions that are not significant.

At this point, the experimenter would typically begin eliminating these factors, rerunning the analysis until only significant factors and interactions are left. This is usually referred to as “reducing the model.” As factors are removed from the model, additional degrees of freedom become available for the calculation of p-values. The number of models you need to evaluate depends on the number of factors in your analysis.

Reducing the model takes only one step with Minitab’s stepwise DOE tool. To use this feature, return to Analyze Factorial Design… (or hit Ctrl+E to open your most previous dialog box). Select C10 “Flight time” as the response, then click Stepwise…

The stepwise regression feature makes it simple and fast to select the optimal model for your data by automatically removing factors to find the model that best fits your data. You can choose from four stepwise analysis methods: Stepwise, Forward selection, and Backward elimination, and Forward Information Criteria. In Backward elimination, all factors are included in the initial analysis, and then non-significant factors are removed one by one. In Forward selection, we start with an empty model and search for significant terms. This can be a useful tool when you have a situation like we currently have: Too many terms with too few runs.

Regardless of the stepwise method you use, the model Minitab selects contains the same significant factors shown below:

To help you interpret your results, Minitab can also provide main effects and interaction plots. Select DOE > Factorial > Factorial Plots… Because we have already analyzed the results, Minitab automatically selects the factors used in our model:

Clicking OK gives us plots of the significant main effects and interactions. The main effects plot shows the results of changing from one setting to another for each factor. Be cautious with these. You can only directly interpret main effects that are *not* involved in a significant interaction. This is why we only select B and E, because the A*C interaction overrides the main effects.

The main effects plot shows we have longer flight times with the paper clip off, and longer rotor length. The interaction plot shows the interactions between the factors.

This interaction tells us about these two factors. With heavy paper, leg length DOEs not matter much. With light paper, leg length has a much larger impact. Here it shows us our longest flight is with light paper and short leg length.

Finally, we can use the Response Optimizer to find the combination of factor settings that will give us the longest flight time. Select Stat > DOE > Factorial > Response Optimizer… Our goal is to maximize the flight length.

The optimizer produces the following graph showing the optimal factor settings in red, and the predicted response for helicopters made with those settings in blue. Take note that factor D DOEs not appear, as our analysis did not flag it as significant. Because it has no statistically significant impact on flight time, we can choose the setting for Factor D based on other considerations, such as cost:

Conclusion

For the data we collected, our analysis with Minitab indicates the optimal helicopter settings are lighter paper, longer rotor length, shorter leg length, and no paperclip on the leg.

To design an even better helicopter, we could repeat the entire DOE using even lighter paper and longer helicopter blades. A 50 cm wing may be bigger, but that does not mean it will be better. You may be able to predict the ideal settings based on a DOE result, but you should always be cautious when extrapolating beyond the data set, or the result may be a crashing helicopter.