Expanding the Role of Statistics to Areas Traditionally Dominated by Expert Judgment
In a previous post, I wrote about how the field of statistics is more important now than ever before due to the modern deluge of data. Because you’re reading Minitab's statistical blog, I’ll assume that we’re in agreement that statistics allows you to use data to understand reality. However, I’d also bet that you’re picturing important but “typical” statistical studies, such as studies where Six Sigma analysts determine which factors affect product quality. Or perhaps medical studies, like determining the effectiveness of flu shots.
In this post, I’m going to push that further, much further. So far that it might even make you feel a bit uncomfortable. It’s not that I want you to feel discomfort, but I hope you can find new areas to apply statistical analyses. I’m talking about areas that are considered human specialties: intuition, experience, professional judgment, decision-making, and even creativity.
Human Judgment Is Often Insufficient
Life is complicated and it forces people to process a large amount of information, often in real time, to make complex decisions. Humans have developed mental shortcuts, intuition, and accepted wisdom in order to avoid information overload. It’s often not possible to process all of the relevant variables, their interactions and correlations, weight them all properly, and avoid personal biases. People are also easily swayed by detailed stories, irrelevant information, and anecdotal evidence.
Studies in the cognitive and behavioral sciences have consistently shown that our brains are not up to the task. However, statistical analyses can solve these problems. Studies that date back to at least the 1950s have repeatedly shown that even simple statistical models can produce better predictions than expert judgment. Let's check out some examples that show how statistical analyses have outperformed experts.
Six Examples of Predictive Analytics and Statistical Decision-Making Tools
Predicting wine vintage prices
Orly Ashenfelter, a Princeton economist, fit regression models that predict the price of wine vintages. These models include predictors such as temperatures and rainfall, among others. Wine experts dismissed the regression models until the model’s predictions beat the experts to the punch on identifying several “vintages of the century.” (Use Regression in Minitab to Make Predictions)
Predicting cardiac risk in the emergency room
Cardiovascular complications pose one of the most significant risks to patients undergoing major surgery. Ideally, cardiologists identify the patients with a high risk before surgery so they can take steps to reduce the risk. Unfortunately, the risk of complications is notoriously hard to predict even for cardiologists. Lee Goldman, a cardiologist, used statistical methodology to create an easy-to-use checklist that estimates cardiac risk. Over the years, this simple tool has proven to outperform the assessment of emergency room doctors. It was originally developed in 1977 but it was not implemented until the mid-1990s.
Predicting the correct diagnosis at the point of care
Isabel is a computer-based clinical diagnostic support system (DSS). This system uses patient variables and symptoms to calculate the most probable diagnoses from over 11,000 possible diagnoses, a task that simply isn’t possible for humans. A study found that when key features from 50 challenging cases, reported in the New England Journal of Medicine, were entered into the system, it provided the final diagnosis in 48 cases (96%).
The Journal of the American Medical Association has concluded that misdiagnosis is increasing. An autopsy study found that 20% of fatal cases are diagnosed incorrectly! So you’d think that using software like this would be a good thing! However, a 2007 study found...well, the title of the article in the Journal of Medical Decision Making says it all: Patients derogate physicians who use a computer-assisted diagnostic aid. Physicians are aware of this perception and, as a result, many don’t use these valuable tools.
Predicting academic achievement for college admissions
Academic admissions counselors attempt to predict the future academic achievement of the candidates they review. However, decades of psychology studies have found that high school rank and standardized test scores have a higher correlation with their eventual academic achievement than expert judgment.
Are you wondering if a reasonable compromise might be to use the numeric measures in conjunction with human judgment? Researchers have studied this as well. They found that academic achievement is better predicted by standardized scores alone than by the scores plus expert judgment.
Predicting job performance to select job candidates
Human resource (HR) professionals try to predict the future job performance of the applicants. A survey of HR executives reveals that the traditional unstructured job interview is widely considered to be more effective than pencil-and-paper tests that assess aptitude, personality, and general mental ability.
However, a meta-analysis finds that these types of tests are consistently better predictors than the unstructured interviews. Scholarly surveys have found that HR professionals are aware of these persistent findings, but they don’t think the pattern applies to them!
Predicting movie profits to influence script writing
The last example comes from a recent New York Times article and it is a new one for me. You’d think that script writing would remain a creative activity. However, big money is riding on a movie’s success. Scripts are approved based on a prediction that the movie will make money, and I’m sure you see where this is heading based on the other examples. Statistician Vinny Bruzzese has developed a methodology to statistically analyze the story structure of the script in order to predict profitability. Based on the analysis, Bruzzese suggests alterations to the script that are designed to increase the predicted profits.
While the methodology is proprietary, it appears that Bruzzese maintains a very detailed movie database which he uses to create a statistical model that relates the characteristics of movies to their profits. Bruzzese hasn’t released data that would allow us to evaluate the results. However, he is reportedly expanding his business and some industry executives are saying that everyone will be doing this soon. Not surprisingly, this statistical intrusion into the creative world is causing much controversy among the creative people!
Pushing the Boundaries
I mentioned that some of these examples might make you feel uncomfortable. Does the idea that it’s better to select a job candidate based on a test score than an interview make you squirm? Perhaps you don’t like the idea that expert opinion actually lowers the reliability of predictions for academic achievement? How about the idea that your risk of complications may be determined by a checklist rather than a doctor’s assessment in the ER? Personally, I’m a writer, so the script analysis example is a bit scary!
If you do feel discomfort, you’re far from alone. In fact, there was great resistance to all of the statistical models in the examples because they intrude on human judgment. In some cases, the overwhelming results have changed minds. No one is talking about getting rid of Goldman’s cardiac risk assessment. But, that’s not always true.
After decades of research, many HR executives and admissions committees still do not understand that test scores are proven to outperform human judgment. And, patients still think that doctors who use computer tools are less competent than those who don’t. This occurs despite the fact that a meta-study, which assessed 100 studies, found that an overwhelming majority concluded that doctors who consult diagnosis predictions based on statistical analyses are correct more often than unaided doctors.
Unfortunately, it is more socially acceptable to rely on judgment rather than test scores, ratings, and formulas, even when the data suggest otherwise.
The Times They Are a-Changin’!
Don’t fret; there is a positive message behind all of this. Ultimately, this is a story about playing to our strengths. It turns out that we aren’t always as good as statistical models when it comes to predicting outcomes for complicated scenarios. However, the good news is that we excel at creating the statistical models and decision-making tools that produce better predictions! And, it has been powerful statistical software, like Minitab, that has been instrumental in unlocking this potential to produce the best predictions possible.
This blog is really a challenge to you. Find new and innovative ways to apply statistics. Push the envelope. Don’t let the old boundaries and discomfort hold you back!