This is part three in a series where we assess what information we can obtain from the various estimates of quarterly GDP growth using statistical analysis and a control chart. Read part one here and part two here. You can download the Minitab Statistical Software project file used in this series here.
Today, the U.S. Bureau of Economic Analysis (BEA) announced that the gross domestic product (GDP) grew at an estimated 2.8 percent for the fourth quarter of 2011. This is up from 1.8 in the third quarter. Can we draw any conclusions from this change? Before we can answer that, we need to assess the early estimates using an I-MR chart, like we did for the Latest, gold-standard, estimates.
Normally, you compare the Third estimate of the previous quarter to the Advance estimate of the current quarter, which I did above. However, we can’t graph this approach because it requires continually updating the chart, changing Advance estimates to Latest estimates as you move forward in time. However, assessing Third estimates is a good substitute. Among the early estimates, Third estimates utilize the most complete data, so they are theoretically the best-case scenario for the early estimates.
I-MR Chart of the Third GDP Estimates
In this I-MR chart, we are looking at the Third estimates for quarterly GDP growth:
At first glance, this I-MR chart looks similar to the previous one. However, there are fewer failed points and some patterns are different. (To compare this chart to the previous chart, click here and scroll down in the new window.)
First, there are no failed points in the MR chart, which indicates that the change between Third estimates contains no useful information. That’s an important point to ponder because it’s this change that news media often reports as, “the economy is heating up!” or “cooling down!” The change between quarters using the Third estimates has not provided a meaningful signal for the past 15 years.
So, off we go to the I-chart. Specifically, we’ll compare the 4 patterns in the I chart of Latest estimates to those in this I chart of Third estimates. Bold indicates a match!
1) The early estimates do not correctly show that this time frame is consistently above average. Instead, see how the early estimates are bouncing randomly above and below the average for a while? Eventually, you get to the failed point in Q2 2000, which failed test 6, with 4 out of 5 points more than 1 standard deviation from the center line on the same side. The I-MR chart of the Latest estimates detected this pattern 2 years earlier, in Q2 1998! The early estimates systematically under-estimate GDP growth for the first 13 quarters of this data set.
2) Moderately below-average GDP growth for an extended time. This time frame is still consistently below average for an extended time. The same point fails in this I-MR chart as the previous chart. The early estimates correctly show this pattern!
3) The early estimates do not correctly show that this time frame is randomly bouncing around the average. Instead, see how the early estimates are consistently above average? Q3 2003 even exceeds the upper control limit and 2 more points fail test 2 (9 points in a row on the same side of the center line). The early estimates systematically over-estimate GDP growth during this time.
4) Economic downturn. This time frame is still way down. This chart detects it one quarter later because the Third estimate for Q3 2008 doesn’t reflect the full extent of the true decline. Still, I’d say that the early estimates correctly show the pattern.
The Third estimates correctly identify only 2 of the 4 patterns that we see in the Latest estimates.
In real life, you wouldn’t just watch the Third estimates for 15 years. Eventually, you’d start replacing Third estimates with Latest estimates. However, you have to wait for those. And, as we’ve seen, you often need a number of Latest estimates (sometimes as many as 9!) to really see the true pattern. So, it takes time to get good estimates and enough of them. Inaccuracies occur when you try to determine what is occurring right now in the economy, or even in the recent past.
Out of curiosity, I ran an I-MR chart on the Advance estimates. It showed nearly the same picture as the Third estimates. The Advance MR chart incorrectly indicated a significant change between quarters, signaling that the economy was heating up when it actually wasn’t. However, the Advance I chart correctly identified and misidentified the same patterns as the Third estimates. Because the Advance and Third estimates tell the same story, it is OK that we can’t graph the more realistic switching from Advance estimates to Third estimates as they become available. There’s not much of a difference.
How the Early Estimates Are Different from the Latest Estimates
The early estimates missed some correct patterns but it’s not because the early estimates are far off on average. The Third estimates are an average of 0.33% greater (2.785% - 2.458%) than the Latest estimates, and the Advance estimates are only 0.24% greater. Yet, despite these miniscule differences, there seems to be enough noise to obscure the signals in the data. How does this work?
Random error is by definition, random. This means that the estimates should have errors that fall equally above and below the true value, which leaves the overall average unaffected. For instance, a +2% error for one estimate will be cancelled out by a -2% error in another. This produces a 0% change in the average but it can change the apparent relationship between data points.
When random error is higher, the overall average should remain unaffected but the picture becomes less accurate. The early GDP growth estimates exhibit this pattern: unaffected averages but missed some true patterns. Therefore, my hypothesis is that the earlier estimates simply have more random error than the Latest estimates.
Five Big Lessons
I wasn’t sure how this experiment would turn out. The time series plot bugged me but I wasn’t sure why at first. The early estimates weren’t off by that much, but something didn’t seem to be right. The control charts really clarified how the estimates were different. In the process, I’ve learned 5 big lessons:
- While the earlier estimates are close, on average, the noise causes them to miss some of the patterns that the latest estimates detect.
- For the early estimates, the difference from one quarter to the next has never been meaningful or predictive, not even during the recent severe downturn. Large changes during normal times are not unusual.
- Patterns take time to develop. And you need the Latest estimates to get the correct data.
- The economy is complex, and you’re not going to predict it with one estimate per quarter.
- If you look back historically, at many data points, using the Latest estimate, you do get a good sense of how the economy is doing . . . albeit well after the fact!
Back to the Present
So, let's get back to the two most recent estimates of GDP growth: Q3 2011 (1.8%) and Q4 2011 (2.8%), with a 1% change between them. I can tell you upfront that there won't be any startling revelations here. Instead, my goal is to provide the larger context of how the types of estimates differ and what sorts of changes are meaningful.
With that in mind, the MR Chart of Third Estimates shows us that we can expect an average absolute change of 1.866% between growth estimates. The current 1% difference is less than the average, and far less than the 6.097% that would signify a meaningful change. Additionally, the moving ranges of Third estimates have historically not provided meaningful signals.
This doesn't stop the news media because I'm reading current headlines like this one on CNNMoney: "U.S. economy picks up speed." The article says that it's a "major improvement," although it's less than the 3.2% that analysts expected, which sent stock futures tumbling. Phooey on all of that. Even 3.2% growth (1.4% change) would've told us nothing!
The change isn't telling us anything, so let's look at the recent past and see if we can detect a pattern. On the I chart after period 4, it seems to show a potential pattern of moderately below average for an extended time. However, and this is a big caveat, there is insufficient evidence at this point to trip the alarm. We also know that Third estimates don't always provide an accurate picture. The I-MR chart of Latest estimates (here) shows a more mixed picture, where half of the points are above average and half below. The picture here is somewhat better. Of course, there are also fewer Latest estimates available and they don't extend as far forward in time as the Third estimates.
To sum up, we don't really know where the economy is headed at the current time. This illustrates why it's generally easier to assess the state of the economy from a historical perspective. For example, recessions are always officially determined well after they started. After we build up more Latest estimates for this time frame, we'll have a better idea.
"It's tough to make predictions, especially about the future." — Yogi Berra