Measuring Up to Prove Accuracy to a Commission

I love talking to people who use data and analysis to improve processes and quality. As I've worked with customers to tell their stories, my definition of "quality" has expanded. In some cases, data has been used not just to improve quality in terms of reducing defects, but even to demonstrate to regulators that a company is already meeting or exceeding regulatory requirements. 

A few years ago, I spoke with some of the quality experts at a large energy company. This company's business included delivering natural gas to 1.2 million customers in a midwestern state.
Remote systems on about 370,000 meters in that state let the company take readings without requiring a technician to make a house visit, but the company faced complications—and tremendous expenses—when the state's Public Utilities Commission decided that remote readings could not be considered actual readings. The company would thus fail to meet a requirement that companies read meters at least once a year. 
About 43% of the company’s gas customers had meters located inside their homes; of those, 29% were equipped with remote readers. The company estimated that obtaining actual meter readings from all meters, including those equipped with remote readers, would cost between $8 and $12 million yearly, in addition to inconveniencing customers who would need to be on hand when readings were taken. 
The new requirement also theatened to delay an existing effort to improve meter accuracy. The company already had embarked on a project to install new automated meter reader (AMR) technology on all meters by 2011. In use throughout the United States, these devices use computerized technology to transmit gas usage information to company vehicles driving through neighborhoods. But until it could put the AMR technology in place, it needed to keep using the remote readers.
The company was so confident its remote readers were reliable, it launched a Six Sigma project to prove it, racing against the imminent regulatory deadlines. The project team needed to demonstrate conclusively that the remote devices were as accurate as reading the actual meter. About $8 to $12 million in cost savings depended on it, but more important factors were on the line, including customer confidence. Would this project prove that the technology used to bill them was accurate?
The team used Minitab Statistical Software’s ability to analyze data from a sample to make inferences about the total population with high levels of confidence. They analyzed data collected from 19,704 records, or 5.9% of the installed population. This data included both the remote and the actual meter readings collected from accounts between January 2005 and July 2006. Among the analyses they used was a 1 proportion test to prove the remote readings were accurate. Using Minitab’s Power and Sample Size tool, they further determined the size of their sample permitted the team to make estimates with an unusually high 99% confidence interval. That really helped the company make its case to the commission.
Minitab’s analysis helped prove that the remote reading devices were accurate. The team found that the overwhelming majority of  remote readings—13,454 records—were exact matches for the actual meter reading. Slightly less than 1,000 were within ± 1 Mcf, while 737 were within ± 3 Mcf of the actual reading. The team found that its Hexagram remote devices had a 1.8% defect rate, while the Read-O-Matic remote devices had a 9.5% to 21.4% defect rate. That sounds high—until you realize the overall defect rate accumulated over the life of the device, which ranged between 15 and almost 30 years. In short, the remote devices were accurate, and the company proved it with Minitab. 
The team recommended that the company ask the utility commission for a waiver on the requirement to obtain actual meter readings from in-home meters that were equipped with remote reading devices.
Swayed by the data, the utility commission granted the waiver, with stipulations: Within 5 years, all Hexagram devices needed to be replaced with AMR devices; Read-O-Matic remotes needed to be replaced within 2 years; and AMR devices also were to be installed on all other meters—as the company was already doing.
The waiver reduced the company's costs by approximately $7.8 million, and it also meant fewer customers needed to disrupt their schedules to accommodate in-home meter readings. The company also saw reductions in employee safety incidents, and was able to devote more resources to installing its AMRs. 
Not a bad conclusion for a hypothesis test


Name: Bob Quinn • Friday, October 26, 2012

Thank you for a great article! I think you've demonstrated:
- the power of good data
- how to use data to influence a regulatory office
- the power of a business case as a driver for change
- the power of using hypothesis testing to validate a process

I do wonder though what happened with the other 4,513 customer readings not mentioned? The math was not all disclosed in the article.I'd also love to see the cost per defect, the total capital investment to achieve the 7.8 million in savings and of course the msa on how they validated the readings as "accurate."

Thanks again for a thought provoking article.

blog comments powered by Disqus