---
title: "The Lady Tasting Beer: Evaluating a Go/No-Go Gage (Part II)"
description: "The Lady Tasting Beer: Evaluating a Go/No-Go Gage (Part II)"
image: https://blog.minitab.com/hubfs/B2BML%20-%20NEW%20FILE%20MANAGER%20STRUCTURE/Blog%20Images/Imported_Blog_Media/lady_tasting_beer_3-3.jpg
---
English
- [English](https://blog.minitab.com/en/?hsLang=en)
- [Français](https://blog.minitab.com/fr/?hsLang=en)
- [Deutsch](https://blog.minitab.com/de/?hsLang=en)
- [Português](https://blog.minitab.com/pt/?hsLang=en)
- [Español](https://blog.minitab.com/es/?hsLang=en)
- [日本語](https://blog.minitab.com/ja/?hsLang=en)
- [한국어](https://blog.minitab.com/ko/?hsLang=en)
- [中文（简体）](https://blog.minitab.com/zh/?hsLang=en)
- [ Partners](https://www.minitab.com/en-us/company/partners/)
- [ Contact](https://www.minitab.com/en-us/company/contact-us/)
- [ Talk to Minitab](https://www.minitab.com/en-us/talk-to-minitab/)
- [ My Account](https://licensing.minitab.com/account/manage)
[](https://www.minitab.com/en-us/)
- Products
Back
[Products](https://www.minitab.com/en-us/products/)
[All Products](https://www.minitab.com/en-us/products/)
- [Minitab Solution Center](https://www.minitab.com/en-us/products/minitab-solution-center/)
- [Minitab Statistical Software](https://www.minitab.com/en-us/products/minitab/)
- [Minitab Connect](https://www.minitab.com/en-us/products/connect/)
- [Minitab Model Ops](https://www.minitab.com/en-us/products/model-ops/)
- [Minitab Education Hub](https://www.minitab.com/en-us/products/education-hub/)
- [Minitab Engage](https://www.minitab.com/en-us/products/engage/)
- [Minitab Workspace](https://www.minitab.com/en-us/products/workspace/)
- [Real-Time SPC](https://www.minitab.com/en-us/products/real-time-spc/)
- [Prolink Data Collection & SPC](https://www.minitab.com/en-us/products/prolink)
- [Simul8 Discrete Event Simulation](https://info.minitab.com/en/new-to-simul8?hsLang=en)
- [SPM](https://www.minitab.com/en-us/products/spm/)
- Solutions
Back
[Solutions](https://www.minitab.com/en-us/solutions/)
[All Solutions](https://www.minitab.com/en-us/solutions/)
- [Analytics](https://www.minitab.com/en-us/solutions/analytics/)
- [Statistics & Predictive Analytics](https://www.minitab.com/en-us/solutions/analytics/statistical-analysis-predictive-analytics/)
- [Data Science & Machine Learning](https://www.minitab.com/en-us/solutions/analytics/data-science-machine-learning/)
- [Business Analytics & Intelligence](https://www.minitab.com/en-us/solutions/analytics/business-analytics-intelligence/)
- [Statistical Process Control](https://www.minitab.com/en-us/solutions/analytics/statistical-process-control/)
- [Quality Analytics](https://www.minitab.com/en-us/solutions/analytics/quality-analytics/)
- [Live Analytics](https://www.minitab.com/en-us/solutions/analytics/live-analytics/)
- [Reliability & Life Data Analysis](https://www.minitab.com/en-us/solutions/analytics/reliability/)
- [Discrete Event Simulation](https://info.minitab.com/en/new-to-simul8?hsLang=en)
- [Process Mining](https://info.minitab.com/en/new-to-simul8?hsLang=en)
- [Key Capabilities](https://www.minitab.com/en-us/solutions/key-capabilities/)
- [Continuous Improvement](https://www.minitab.com/en-us/solutions/key-capabilities/continuous-improvement/)
- [Data Integration & Data Prep](https://www.minitab.com/en-us/solutions/key-capabilities/data-integration-data-prep/)
- [Diagramming & Mind Mapping](https://www.minitab.com/en-us/solutions/key-capabilities/diagramming-mind-mapping/)
- [Digital Twins](https://info.minitab.com/en/new-to-simul8?hsLang=en)
- [Model Deployment & ML Ops](https://www.minitab.com/en-us/solutions/key-capabilities/model-deployment-ml-ops/)
- [Innovation & Project Management](https://www.minitab.com/en-us/solutions/key-capabilities/innovation-project-management/)
- [Process Excellence: Detect, Correct & Prevent](https://www.minitab.com/en-us/solutions/key-capabilities/process-excellence/)
- [Featured Industries](https://www.minitab.com/en-us/solutions/industry/)
- [Academic](https://www.minitab.com/en-us/solutions/industry/academic/)
- [Energy & Natural Resources](https://www.minitab.com/en-us/solutions/industry/energy-resources/)
- [Government & Public Sector](https://www.minitab.com/en-us/solutions/industry/government/)
- [Healthcare](https://www.minitab.com/en-us/solutions/industry/healthcare/)
- [Insurance](https://www.minitab.com/en-us/solutions/industry/insurance/)
- [Manufacturing & Industrial](https://www.minitab.com/en-us/solutions/industry/manufacturing-industrial/)
- [Services](https://www.minitab.com/en-us/solutions/industry/service-industries/)
- [Software & Technology](https://www.minitab.com/en-us/solutions/industry/software-technology/)
- [Construction](https://www.minitab.com/en-us/solutions/industry/construction/)
- [Featured Roles](https://www.minitab.com/en-us/solutions/role/)
- [Engineering](https://www.minitab.com/en-us/solutions/role/engineering/)
- [Business Analysis](https://www.minitab.com/en-us/solutions/role/business-analysts/)
- [Information Technology](https://www.minitab.com/en-us/solutions/role/information-technology/)
- [Supply Chain](https://www.minitab.com/en-us/solutions/role/supply-chain/)
- [Customer Service & Contact Center](https://www.minitab.com/en-us/solutions/role/customer-service-contact-center/)
- [Human Resources](https://www.minitab.com/en-us/solutions/role/human-resources/)
- [Marketing](https://www.minitab.com/en-us/solutions/role/marketing/)
- [Research and Development](https://www.minitab.com/en-us/solutions/role/research-and-development/)
- Resources & Services
Back
[Resources & Services](https://www.minitab.com/en-us/resources-services/)
[All Resources & Services](https://www.minitab.com/en-us/resources-services/)
- [Resources](https://www.minitab.com/en-us/resources-services/resources/)
- [Case Studies](https://www.minitab.com/en-us/resources-services/resources/case-studies/)
- [eBooks & White Papers](https://www.minitab.com/en-us/resources-services/resources/ebooks-and-white-papers/)
- [Blog](https://blog.minitab.com/blog/?hsLang=en)
- [Data Sets](https://www.minitab.com/en-us/resources-services/resources/data-set/)
- [Webinars & Events](https://www.minitab.com/en-us/resources-services/resources/events/)
- [Education Hub](https://www.minitab.com/en-us/resources-services/resources/education-hub/)
- [Services](https://www.minitab.com/en-us/resources-services/services/)
- [Training](https://www.minitab.com/en-us/resources-services/services/training/)
- [Deployment](https://www.minitab.com/en-us/resources-services/services/deployment/)
- [Consulting](https://www.minitab.com/en-us/resources-services/services/statistical-consulting/)
- [Self-Paced Learning](https://www.minitab.com/en-us/resources-services/services/self-paced-learning/)
- [Continuing Education](https://www.minitab.com/en-us/resources-services/services/continuing-education/)
- Support
Back
[Support](https://www.minitab.com/en-us/support/)
[Technical Support](https://www.minitab.com/en-us/support/)
- [Subscriptions & Activation](https://www.minitab.com/en-us/support/subscriptions-and-activation/)
- [Minitab Quick Start](https://www.minitab.com/en-us/support/quick-start/)
- [Training](https://www.minitab.com/en-us/resources-services/services/training/)
- [Installation Support](https://www.minitab.com/en-us/support/installation/)
- [Support Videos](https://www.minitab.com/en-us/support/videos/)
- [Support Documentation](https://www.minitab.com/en-us/support/documents/)
- [Software Updates](https://www.minitab.com/en-us/support/software-updates/)
- [Product Downloads](https://www.minitab.com/en-us/support/downloads/)
- [Support Policy](https://www.minitab.com/en-us/support/policy/)
- Company
Back
[Company](https://www.minitab.com/en-us/company/)
[Company](https://www.minitab.com/en-us/company/)
- [About Us](https://www.minitab.com/en-us/company/our-company/)
- [Leadership Team](https://www.minitab.com/en-us/company/leadership-team/)
- [Partners](https://www.minitab.com/en-us/company/partners/)
- [Careers](https://www.minitab.com/en-us/company/careers/)
- [Contact](https://www.minitab.com/en-us/company/contact-us/)
- [News](https://www.minitab.com/en-us/company/press-releases/)
[Try/Buy](https://www.minitab.com/en-us/try-buy/)
- [Statistics](https://blog.minitab.com/en/blog/tag/statistics)
- [Data Analysis](https://blog.minitab.com/en/blog/tag/data-analysis)
-
- [Quality Improvement](https://blog.minitab.com/en/blog/tag/quality-improvement)
- [Subscribe](https://info.minitab.com/blog/subscribe?hsLang=en)
## Notice
You are now leaving minitab.com.
Click Continue to proceed to:
Continue
Go Back
# The Lady Tasting Beer: Evaluating a Go/No-Go Gage (Part II)
Minitab Blog Editor | 4/9/2012
Topics: [Quality Improvement](https://blog.minitab.com/en/blog/tag/quality-improvement)
- [Share](https://www.linkedin.com/shareArticle?mini=true&url=https://blog.minitab.com/en/blog/statistics-and-quality-data-analysis/the-lady-tasting-beer-evaluating-a-gono-go-gage-part-ii)
- [Share](https://www.facebook.com/sharer.php?u=https://blog.minitab.com/en/blog/statistics-and-quality-data-analysis/the-lady-tasting-beer-evaluating-a-gono-go-gage-part-ii)
If your measurement system depends on subjective decision-making, it’s critical to evaluate its consistency and accuracy.
Lately,[I’ve been bottle-feeding my work colleagues skunky and sour beer](https://blog.minitab.com/blog/statistics-and-quality-data-analysis/the-lady-tasting-beer-evaluating-a-gono-go-gage-part-i?hsLang=en) to see how well they can identify defective grog.
Now it’s time to evaluate my (hic) go/no-go gage in Minitab using an Attribute Agreement Analysis.
How well did our appraisers do? Let's look at the results from the Minitab Assistant (**Assistant > Measurement Systems Analysis > Attribute Agreement Analysis**.)
## Accuracy of Appraisals
First, the accuracy of all the appraisals taken together:
Overall, our appraisers correctly identified the “good” and “bad” samples about 60% of the time. That's just slightly better than what you’d expect by pure chance (50%). There’s definitely room for improvement in this appraisal system!
Now let's break it down by appraiser:
Morgan appears to be the most accurate appraiser, discriminating good grog from bad 75% of the time. Tootsie was close at her heels, at 62.5%.
Monique and Jaenbe, on the other hand, have accuracy rates less than the average of all the appraisers (60%) and the same as that expected by pure chance (50%). To improve their accuracy rates, they've both volunteered for intensive retraining at Fred's Bar and Grill, *if* I continue to supply them with free *Stella Artois*. (Hmmmm...am I being had?)
So we know our measurement system has some problems with accuracy. But what types of errors are causing the problem?
## Misclassifications
When misclassifying a sample, there was a slightly greater tendency to mistake good beer for bad beer (43.8%), rather than vice-versa (37.5%), as shown in the chart below:
This makes me wonder about a possible carryover effect. For example, could the sourness from a previous vinegary beer sample be lingering on the palate of an appraiser when they taste a subsequent “good beer” sample?
Carryover can be a problem in appraisal studies. Even if it’s not a taste test, appraisers might recognize an item they appraised before and rate from memory. In that case, you want to ensure there’s an adequate time between appraisals to prevent that.
In this study, to help guard against carryover effect, I supplied appraisers with filtered water and gluten-free, salt-free rice cakes to clear their palates between samples. (It had to be something tasteless, like packing peanuts, to prevent introducing another source of taste bias.)
But most appraisers didn’t nibble their rice cakes or sip water between each sample. In hindsight, I should have made that a mandatory part of the appraisal process. (This shows why running a small experimental pilot study is a good idea—you can identify and address these types of issues *befor*e you do a more expensive, time-consuming full-fledged study.)
To evaluate the overall self-consistency of appraisers, look at the **Mixed ratings** result in the above chart. About 31% of the time, the same beer sample was rated differently in each of the two trials. When you obtain a high percentage of mixed ratings, it can indicate that appraisers may need additional training or that the standards themselves may not be clearly defined.
The bar charts of misclassifications, such as the **%Bad rated Good** (shown below), helps to clarify mixed ratings in your study.
It’s clear that the appraisers had a hard time discerning the sourness of the vinegar beer. I should have added more vinegar to allow them to more clearly differentiate the “good” from the “bad.” Based on this chart, I’ll take the rap for the fuzzy standards!
The **Appraiser Misclassification Rates** chart reveals some interesting patterns for each individual appraiser.
**Jaenbe** seems to mistake good beer for bad beer at about the same rate (50%) as she mistakes bad beer for good beer. Interestingly, although her ratings weren't very accurate, Jaenbe was the only appraiser who was completely self-consistent in her ratings (**% Rated both ways** chart). She seems to be marching to the beat of a different bartender.
**Monique** was less likely to mistakenly classify good beer as bad than the other appraisers. On the other hand, she also rated bad beer as good 75% of the time—more than any other appraiser. She might benefit from retraining to better recognize Things That Taste Vile. (Although if the beer always tastes good, regardless of whether it’s flat or spiked with vinegar, she must be a very happy and content person. So maybe we should just leave her be.)
**Morgan** correctly identified every sample of defective beer (the **% Bad rate Good** chart indicates 0). When she did err, she mistook good beer for bad. That supports a possible carryover effect. So I checked the raw data. Sure enough, her two misclassifications of “bad” for “good” actually came immediately after she had sampled a skunky beer. Does old, flat beer leave a lingering bad taste in Morgan’s mouth? If so, to paraphrase Marie Antoinette (or [Jean-Jacques Rousseau](http://en.wikipedia.org/wiki/Let_them_eat_cake)), “Let her eat rice cake!”
**Tootsie’s** performance was somewhere in the middle of the pack. She was more accurate than Monique and Jaenbe at picking out bad beer, but less accurate than Morgan. Her self-consistency was higher than Morgan and Monique's, but lower than Jaenbe’s.
## If You Don't Read Anything Else, Read This!
The Accuracy Report shows 95% confidence intervals for the accuracy rate of each appraiser. Here’s where the rubber meets the road, statistically speaking.
Yikes! Look at those humongous confidence intervals. You could dock a cruise ship in those babies. More formally stated, you can be 95% confident that:
- The accuracy rate of Jaenbe and Monique falls between 15% and 84%
- Morgan’s accuracy rate falls between 40% and 97%
- Tootsie’s accuracy rate falls between 25% and 92%
Remember, you can be 100% confident that any appraiser’s accuracy rate is between 0% and 100%, without performing any statistical analysis whatsoever. So if your confidence intervals come close to that span, your data doesn’t tell you much.
So, I really had no right to impugn the skills of the appraisers using their individual results. Based on these huge confidence intervals, they could all successfully sue me for libel and slander.
To obtain more conclusive results, I'd need to conduct a much larger study, with more appraisals, more trials, and clearer standards for “good” and “bad.”
Problem is, I can't afford to buy any more sixpacks of *Stella*. And my colleagues can’t drink 30 beer samples at work. And even if they could, the ensuing buzz they caught might introduce yet another source of bias into the appraisal process.
Ach! I think I could use a beer...
## You Might Also Like
[Do You Really Need That New Machine? How Minitab Can Help You Find Out](https://blog.minitab.com/en/blog/do-you-really-need-that-new-machine-how-minitab-can-help-you-find-out?hsLang=en)
[Quality Improvement](https://blog.minitab.com/en/blog/tag/quality-improvement) | 4 Minute Read
[Optimizing Quality Control Measures with Minitab's Predictive Analytics](https://blog.minitab.com/en/blog/optimizing-quality-control-with-minitab-predictive-analytics?hsLang=en)
[Lean Six Sigma](https://blog.minitab.com/en/blog/tag/lean-six-sigma) | 6 Minute Read
[Getting Real: A Simple Way to Assess Process Capability without Complex Assumptions](https://blog.minitab.com/en/blog/nonparametric-capability-analysis?hsLang=en)
[Quality Improvement](https://blog.minitab.com/en/blog/tag/quality-improvement) | 5 Minute Read
---
- [Careers](https://www.minitab.com/en-us/company/careers/)
- [Trust Center](https://www.minitab.com/en-us/trust-center/)
- [Suppliers](https://www.minitab.com/en-us/suppliers/)
- [Site Map](https://www.minitab.com/en-us/site-map/)
- [](https://www.linkedin.com/company/minitab/)
- [](https://www.facebook.com/Minitab/)
- [](https://www.youtube.com/user/MinitabInc)
- [](https://www.instagram.com/minitab/)
---
© 2025 Minitab, LLC. All Rights Reserved.
- [Terms of Use](https://www.minitab.com/en-us/legal/terms-of-use/)
- [Privacy Policy](https://www.minitab.com/en-us/legal/privacy-policy/)
- [Legal](https://www.minitab.com/en-us/legal/)
- Cookies Settings