Many quality improvement projects require some form of experimentation on a process. A chemical engineer may wish to determine the settings for certain process variables to optimize a critical quality characteristic of the resulting product. A materials engineer may run a plastic injection molding process using different grades of raw material to determine which produces the least variability in breaking strength.
The deliberate changing of input process variables with the intention of studying their effect on output variables is referred to as a designed experiment. Typically, statisticians identify a designed experiment by describing two primary components:
In a previous article in Quality Progress, we illustrated the features of the split-plot design, how common the features are in industrial experimentation and how the practitioner can recognize this situation. We will now illustrate the proper analysis of this particular type of design structure.
Consider an experiment involving the water resistant property of wood. Two types of wood pretreatment (one and two) and four types of stain (one, two, three and four) have been selected as variables of interest. A graphical representation of this type of treatment design is shown in Figure 1.
Conducting this experiment in a completely randomized fashion would require eight wood panels for each full replicate of the design. Each wood panel would be randomly assigned a particular pretreatment and stain combination. But it turns out to be very difficult to apply the pretreatment to a small wood panel.
The easiest way would be to apply each of the pretreatment types (one and two) to an entire board, then cut each board into four smaller pieces and apply the four stain types to the smaller pieces. This is shown in Figure 2.
So how exactly will the experiment be conducted? For example, how many boards will be used for each treatment combination? How many replicates of each treatment combination will be run? In what order will the experimental runs be conducted? How many measurements will be made on each small piece? These decisions should be based on both statistical and practical considerations.
Suppose the experimenter has decided to run three replicates of the pretreatment factor. This results in six boards and 3 x 2 x 4 = 24 total observations. To produce an experimental run for this process, you must first pretreat a board. After one of the randomly selected pretreatments has been applied, the board is cut into four pieces and then stained using one of the four stains selected at random.
The reader should recognize this is a split-plot design for four reasons:
Because of these features, we would say the experimenter has run a 2 x 4 full factorial treatment structure within a split-plot design structure. Each of the six whole-plots (entire boards) has four subplots (smaller pieces of board), resulting in three replicates at the whole-plot level and six replicates at the subplot level.
The simplest experiment from a statistical analysis perspective is what’s called a completely randomized design structure. This, however, would require all 8 x 3 = 24 experimental runs to be conducted in a completely random order. For the experiment to be run in this way, each of the 24 runs would need to be a “true” experimental run. This would include a complete preparation and setup of the experimental materials and equipment.
As you can imagine, this experimental approach is not always efficient, practical or at times even possible to run. In many real experimental situations, a restriction is typically placed on the randomization of the runs. Such restriction, however, affects the statistical analysis.
An example illustrates the correct analysis of splitplot experiments. Consider the previously described experiment involving the water resistant property of wood. Two types of wood pretreatment (one and two) and four types of stain (one, two, three and four) have been selected as variables of interest.
A graphical representation of the experiment is shown in Figure 2 on p. 68 (for each pretreatment the stains have been randomly assigned to the four panels). Table 1 (p. 69) gives the design as it was carried out: First a randomly selected pretreatment is applied, then the wood is cut into four panels and the stains are applied in random order.
The null hypothesis for all factors is H0: There is no effect due to the factor. A test statistic is necessary to test this hypothesis. In this paper, the test statistics are all F-statistics, which are the ratio of the mean square (MS) for the factor of interest to the correct mean square error
Once the F-statistic has been calculated, a pvalue can be computed and used to test the null hypothesis (we typically reject H0 if the p-value < 0.05). The p-value is the probability the test statistic will take on a value at least as extreme as the observed value of the statistic, assuming the null hypothesis is true.
It is sometimes easier to think of the analysis of a split-plot experiment as two separate experiments corresponding to the two levels of the split-plot experiment: the whole-plot (WP) level and the subplot level.
Again, suppose the experiment is carried out using three replicates of the pretreatment factor. This involves six boards (three for pretreatment number one and three for pretreatment number two). For now, let’s focus on only these six boards (before they are cut and the stains are applied) and break down the degrees of freedom (df).
Because these six boards are randomly assigned a pretreatment level, this part of the experiment is essentially a completely randomized design with one 2-level factor (pretreatment) and three replicates. Therefore, there is 6 – 1 = 5 total df for this whole-plot level of the experiment.
Because the only factor has two levels, pretreatment has 1 df. This leaves 4 df for the error term at the whole-plot level. Notice how thinking of the experiment in this manner clearly shows the pretreatment variable has its own error term, “wholeplot error.” The split-plot design simply exploits the fact that each of the six pretreated boards can be cut into four pieces and another factor (stain) can also be studied.
Once all the data are collected, we could write the model as:
Average response = pretreatment factor + WP error
in which average response is the mean of the four different stain responses in each whole plot, and WP error is the error term for the whole-plot factor (pretreatment).
The whole-plot experimental error is estimated by examining the variability that occurs between the three whole plots within each of the two pretreatment settings. Using these six averages will yield the correct F-test for pretreatment (pretreatment is not significant with p = 0.115, as shown in Table 2, p. 69). However, the sums of squares will not be the same as the correct overall split-plot analysis (they will be off by a factor of 4 = the number of subplots in each whole-plot).
If the 24 pieces involving the four stains are incorrectly viewed as their own completely randomized experiment, then there would be 24 – 1 = 23 total df. This would involve 2 - 1= 1 df for pretreatment, 4 – 1 = 3 df for stain and (2 - 1)(4 - 1) = 3 df for the pretreatment by stain interaction. Therefore, there would be 23 – 7 = 16 df for error. The incorrect completely randomized model is:
Response = pretreatment + stain + pretreatment x stain interaction + error.
Notice, however, this analysis is incorrect because it does not remove the sums of squares and 4 df for whole-plot error discussed above (this is viewing the experiment only at the subplot level). The error term in this model is the sum of the whole-plot error and the subplot error.
When the whole-plot error is not removed from the completely randomized analysis, the error term used for testing the subplot factors is inflated. Therefore, the F-test for all terms in the model would use the wrong error term. This can result in F-tests that are insignificant for some subplot factors while overstating significance for the whole-plot factor.
Table 3 shows the analysis. Notice the pretreatment factor is incorrectly identified as significant (p = 0.002), while the stain factor is insignificant (p = 0.245). We have seen in the earlier whole-plot analysis that pretreatment is not significant, and we will see later in the correct split-plot analysis that stain is significant.
The split-plot model is:
Response = pretreatment + WP error + stain + pretreatment x stain interaction + SP error in which SP error is the error for the subplot factor (stain) and the whole-plot by subplot interaction (pretreatment x stain). To get the correct analysis of variance table with all sources of variation including the two error terms involves removing the sums of squares and df for the whole-plot error from the reported error term in the incorrect completely randomized analysis. This can be done manually, but then all F-tests and p-values will have to be generated manually as well. Fortunately, many software packages can be tricked to do this for you automatically by using a nested model:
Response = pretreatment + WP (pretreatment) + stain + pretreatment x stain interaction + SP error, in which WP is a variable that goes from one to six indicating each whole-plot and must be declared as a random factor.
Specifying the model in this way allows the creation of two separate estimates of experimental error, an ingredient of the split-plot design. The nested term WP (pretreatment) comes from the fact the whole plots are nested within pretreatment.
This term will be the correct error term for the pretreatment factor, and most software packages will correctly use this term for the F-test of pretreatment. The df will also be correctly calculated as 2(3 - 1) = 4 in which 2 represents the number of levels for the pretreatment factor and 3 represents the number of replicates at the pretreatment level of the experiment.
The other estimate of experimental error, called the subplot error, is estimated by examining the variation that occurs between the 12 pairs of experimental runs that have the same pretreatment and stain setting minus the whole-plot experimental error
The whole-plot experimental error is used to test the significance of the whole-plot factor, pretreatment. The subplot experimental error is used to test the significance of the subplot factor, stain and pretreatment by stain interaction. Therefore, the tests use a different mean square error in the denominator of the F-ratio.
Table 4 (p. 70) shows the F-statistic for the effect of pretreatment, the whole-plot factor, is:
Note the p-value of 0.231 indicates the interaction effect is not significant. Notice for both pretreatment and stain, these are different conclusions from the analysis assuming a completely randomized design.
Many experiments in industry involve two-level factors. In the wood experiment, the four stains could actually be a 22 in stain type and amount. All this does is add a little more structure to the experiment and the breakdown of the degrees of freedom.
For example, the previous 3 df for stain can now be broken down into 1 df for stain type, 1 df for amount and 1 df for the stain type by amount interaction. This is also true for the previous pretreatment by stain interaction, which is now 1 df for pretreatment by stain type interaction, 1 df for pretreatment by amount interaction and 1 df for the pretreatment by stain type by amount interaction.
Consider another example with one hard-tochange factor (Z), three easy-to-change factors (A, B, C) and all factors at two levels. The hard-to-change factor is replicated so there are four whole plots, each with eight subplots.
Table 5 gives the design as it was carried out: First a level for Z is randomly selected, then the eight combinations of A, B and C are carried out in random order. The correct and incorrect analyses are shown in Table 6. Notice the incorrect analysis indicates Z is significant, while the Z x A and A x B interactions are shown as not significant.
An astute reader can probably now surmise the split-plot framework can be expanded to even more complicated experiments. Several extensions that can be made to the split-plot scenario are:
The design and analysis of industrial experiments involves understanding not only the treatment structure but also the three principles of the design structure: randomization, replication and controlling for known sources of variation (typically through blocking).
The experimenter should be made aware of an important point about the experimental replication in a split-plot design. The effect of the whole-plot factor, which will have the least number of experimental replicates, is estimated less precisely than the subplot factors, which will have more experimental replicates. Thus, if allowed a choice when planning a split-plot experiment, the experimenter should try to put the most important factors at the subplot level.
Many practitioners of experimentation are beginning to incorporate the principles and methodology of designed experiments developed in the statistical literature over the last 75 years. The first experiments learned in typical statistical and quality methodology training courses are those with simple design structures, such as the completely randomized design.
In practice, however, the limitations and challenges of experimenting in the real world result in these simple experiments being the exception rather than the norm. Typically, an experiment will contain some form of a restriction on the randomization. We fear that more often than not, these features are not being incorporated into the planning and analysis of the experiment.
With the recent growth and interest in the use of the statistical sciences in today’s businesses, however, we expect the sophistication and understanding of experimentation will increase, and designs such as the split plot will become more readily recognized and properly analyzed.