A 2 proportion test helps you determine whether two population proportions are significantly different -- such as whether the proportion of men who support a candidate is different from the proportion of women who support the same candidate. It uses the null hypothesis that the difference between two population proportions equals some hypothesized value (H0: p1 - p2 = P0), and tests it against an alternative hypothesis, which can be either left-tailed, right-tailed, or two-tailed.
So, you’ve done a 2 proportions test, and you’ve noticed that the session output provided by Minitab Statistical Software lists two p-values. One p-value is based off a Z-statistic, and the other is coming from a Fisher’s Exact Test.
Which one should you use to interpret significance?
The 2 Proportions test always calculates a hypothesis test based on a normal approximation. This information is displayed on the line starting with “Test for difference...” The Z-test, however, is not accurate when the number of events (or nonevents) is less than 5.
This ”five successes rule” is related to these criteria:
1. N*P>5
2. N(1-P)>5
Where N is number of trials in sample and P is proportion of "successes in sample”
Because the binomial random variable is the sum of the independent random variables, each having the same distribution, we can apply the central limit theorem. Thus, the normal distribution can be used to approximate the binomial distribution when n is of an appropriate size (otherwise, the central limit theorem doesn't apply). If n is small and the probability of success is small, the actual binomial distribution is skewed to the right. In this case, the symmetrical normal curve will not provide a good approximation.
In the case where your null hypothesis states that P1 - P2 = 0, Minitab also calculates Fisher's exact test. The Fisher's exact test is always exactly accurate and reliable for any combination of sample sizes.
The Fisher's exact test is based off the hypergeometric distribution, and the factorials in its formulas can make it difficult to calculate the p-value. Fortunately, Minitab can handle such computations.
Students learning basic statistics might be more inclined to employ the normal approximation method to cut on computation. But, for small samples (#successes/failures < 5), the hypergeometric is not as hard to calculate. Therefore, in cases where the sample size is small, I would suggest that you use the Fisher test to interpret significance.