In Part I, Part II, and Part III we shared our experiment, the survey results, and the experimental results. To wrap things up, we're going to see if the survey results tied to the experimental results in any meaningful way...
First, we look at whether self-identified knowledge correlated to the total number of correct appraisals:
We have no evidence of a relationship (p = 0.795). So we'll look at the number correct by how much each participant usually spends:
Again, no evidence of a relationship (p = 0.559).
How about how many types each regularly buys?
There appears to be something here, but statistically we don't have evidence (p = 0.151). Perhaps a larger experiment might uncover something.
Remember Question #4 in our survey, which asked if participants felt they could identify certain wines by taste? Eight wines were included as choices, including the four wines used in the experiment. So did participants' responses to that question correlate to their ability? To test, I did a Chi-Square test in Minitab.
Here are the expected number of correct guesses for each wine type along with the observed number of correct guesses:
At a strict alpha level of 0.05, there is not statistical significance—but given the small experiment, the corresponding p-value of 0.069 would probably give me reason to investigate further. The largest contributor to the Chi-Square was Riesling, which few participants felt they could identify but as many were correct on as any other wine. It could be that participants underrated their ability, or it could be process of elimination (if you know the other three, then the wine you can't identify in the experiment must be the Riesling).
In the end, we found little evidence of any relationships between the survey questions and the experimental results. While a larger study could likely draw some conclusions, we've learned enough to say that any real, underlying relationships are not particularly strong and would have considerable variation around them.
The next time you plan on trying a new wine, try tasting the wine without being told ahead of time what type it is or even looking at it (sleep masks make great blindfolds)...you might find it to be a completely different experience!