It’s important for your Six Sigma initiative to know if your measurement system is effective. You need solid data to initiate your project and having a solid measurement system is key.
A computer disk drive manufacturer in the mid-1980s was experiencing a nagging problem with poor yields. The principle concern was that the sensitive magnetic medium coating the disks was in some way defective. As a result, the company implemented a collection of very demanding standards and a battery of stringent tests with the hope of detecting and removing media problems from the system.
At one point, design engineers at the company happened to notice some visual defects and spots in the magnetic coating on the disk. They concluded that this issue was the long-sought-after cause of their persistent yield problems.
The engineering department immediately requested that manufacturing implement a final visual inspection of each disk to be done at the end of the already-tedious test cycle. With the implementation of this new inspection, the disk reject rate jumped from 8 to 10 percent. At $30 a disk, the scrap bill neared $300,000 per month!
With no real improvement evident, engineering proposed to further tighten the specifications on the magnetic disk medium. With mounting assembly and scrap costs, manufacturing asked that an expert from engineering audit the test and inspection process one last time before tightening the specs again.
The engineering expert reviewed the entire test and inspection process. He then decided to run some experiments to validate the final visual inspection process on the disks. His first experiment was to send a bunch of previously rejected disks back through the final inspection process without the inspectors’ knowledge.
The results were so startling that he reran the experiment several times; each time the previously inspected disks were secretly sent back through the final visual inspection process, an additional 10 percent of the disks would be rejected!
Armed with this new insight, the engineer tried another test. This time he took a bunch of disks that had already passed the final inspection step and secretly reinserted them back into the inspection process. Even with these “passed” disks, the inspectors continued to find ten percent of the previously passed disks to be visually defective.
As a final confirmation, the engineer sent a collection of passed disks and a collection of failed disks into the final stages of the assembly process. At the end of the assembly process, the disks with the rejected medium actually had a slightly higher final performance yield than those disks passing the visual inspection.
Clearly, this company was living in a measurement system illusion. The visual inspection system they had added provided no benefit to the company but was costing over $300,000 per month in incorrectly rejected disk medium.
Consider a measurement system that categorizes items — whether it is a characteristic or a process — into categories of “pass” and “fail.” To study the effectiveness of this type of measurement system, follow these steps:
Set aside 15 to 30 samples of what is being measured.
You want these samples to represent the full range of variation that is typically encountered, with about half of the samples being “passes” and the other half “fails.”
Create a master standard by designating each of the samples as a “pass” or a “fail.”
Use a panel of experts or some standard that you know is absolutely correct to make these distinctions.
Pick two or three inspectors.
Have them review the sample items in a random order and record their conclusions — whether each item is a “pass” or a “fail.”
Have each inspector repeat her measurements of the samples after you mix the samples up into a new random order and record the repeated measurements.
Randomizing the samples before the second measurements is critical; each inspector’s second measurements must be fair, as if they were happening for the first time. You may need to wait for a day before performing the second measurements (or spin the inspectors around in place until they’re very dizzy — just kidding).
For each inspector, calculate (as a percentage) how often the first and second measurements agreed with each other.
This percentage is the repeatability for each inspector. You can also calculate an overall measurement system repeatability by averaging the repeatabilities of the individual inspectors.
The calculated repeatability for the individual inspectors needs to be as close to 100 percent as possible. Lower calculated individual repeatabilities mean that the inspectors aren’t consistent in distinguishing between good and bad items. Training helps inconsistent inspectors become consistent in their measurements.
For each of the sample items, calculate the percentage of the recorded measurements where each of the inspectors agreed with themselves and all inspectors agreed with each other.
This figure is the reproducibility for the measurement system. The calculated measurement system reproducibility tells you how precise the measurement system is over the long term — over different inspectors, different setups, and different environmental conditions.
You can also calculate the percent of the time individual inspectors and the group of inspectors agree with themselves and agree with the “master” standard created in Step 2.
This number tells you how consistently your measurement system detects what your experts have decided really is pass and fail.
As an example, a calculated 63 percent agreement between all inspectors for all samples with the “master” standard in a measurement system study means that the likelihood that this measurement system will correctly measure the items is 63 percent, and the chance of error is 37 percent. Clearly, the goal is to achieve a measurement system with as high an effectiveness as possible.
More sophisticated analysis tools are available for situations when an attribute measurement system has more than two categories. These tools, such as kappa analysis, can be found in advanced statistical analysis software such as Minitab and JMP.