NASA’s Office of Safety and Mission Assurance Nondestructive Evaluation (NDE) program recently published a Technical Manual, NASA DOEPOD [Design of Experiments for Probability of Detection] NDE Capabilities Data Book.
The NDE program works with a variety of inspection systems, including X-ray, ultrasound, thermal and eddy current, each of which plays a different role in the inspection process. Certified inspectors use these systems to look for critically sized flaws, cracks, damaged areas, etc.; however, there was no validated standard procedure that estimated the likelihood that these critically sized flaws would be detected given the typically used limited test data. The data book, TM-2015-218770, estimates the capabilities of these inspection systems — the likelihood of detection of critically sized flaws — using validated probability of detection methods.
When estimating the likelihood of detection of critically sized flaws it is important to recognize that estimates of probability of detection are specific to the inspection test method, type of flaws, materials and structures, as well as the capability of test personnel and environmental factors. For example, applications of test procedures in a space environment versus on a complex structure may yield quite different likelihoods of detection of critically sized flaws. Therefore, specific capabilities validation data must be provided, generated or documented by the user to support critical hardware design and use. The data book provides resource guidance and validated standard procedures that estimate the capabilities of inspection systems in any relevant environment.
By capturing NASA’s corporate knowledge, the book secures coverage for NASA with respect to accepting inspection capabilities of systems and personnel for many years. The program used a companion software, DOEPOD, to generate the complied list of current capabilities to serve as a baseline to which additional inspection capabilities for new inspection technologies and new structural- and failure-critical components with demanding inspection requirements can be added.
“The data book captures the corporate knowledge of all the NDE people that developed Apollo, the space shuttle and the space station, and when you use the DOEPOD software, you’re using the corporate knowledge of these people,” said Edward Generazio, NDE program manager. “It’s like they’re standing right behind you when you go to do your analysis, providing you with guidance. From my perspective, it will keep everyone at NASA safe, keep us out of a risky area.”
The software also could be used to qualify inspectors. Inspectors could be given samples and asked to identify the flaws, and the software would verify the findings. With new, junior employees joining the NASA workforce as previous generations retire, the software can help close the knowledge gap and keep everyone trained to the highest level.
“We’ve done well in flaw detection over the years, and we want to keep that going,” said Generazio.
The data book also has an electronic version, so the content is easily searchable. Users can search different methods, shapes and materials, and identify what they want to compare.
Background
“Years ago, let’s say the 70s, the inspection folks didn’t know the best way to qualify equipment for inspections,” explained Generazio.
Then, an article came out on the point estimate probability of detection method for qualifying inspection systems that used what became known as the 90/95 method for detection. This method statistically shows that a system is able to detect flaws of a certain size 90 percent of the time (if present) with 95 percent confidence. These numbers became the standard, although as years progressed, various methods were used to determine capabilities.
The military wanted a more efficient method, and developed the curve-fitting approach — a multi-parameter maximum likelihood estimation of probability of detection. This method assumes the probably of detection to have a specific functional form — which is rarely observed in practice and is contrary to the world’s physic-based understanding of ultrasound, electromagnetics, surface tension, etc. — where interactions between interrogating energy materials are quite complex, particularly in regions where wavelengths approach flaws dimensions.
Unlike the military, which has many aircraft, NASA has very few and therefore less data to use to generate these curves based on simple models
“We don’t want to rely on curve fitting for estimating our critical inspection capabilities, so I went back to the basics,” said Generazio.
The basics in this case is the 90/95 method, also known as a binomial point estimate method. Generazio encourages those using the curve-fitting method to validate that the curve generated is applicable for how it’s being used. The data book shows how NASA would have evaluated the same test results from specimens using the curve-fitting method for comparison, and there are dramatic differences.
In Generazio’s experience, curves generated using the curve-fitting method aren’t being validated when compared to the validated binomial based-methods of DOEPOD, and as a result, he wants to eliminate the use of curve fits at NASA. If critically sized flaws go unnoticed for a fracture critical component on a NASA project, the mission could be jeopardized.
“We have to go back to what works for us,” said Generazio. “It’s very important for NASA because we want to make sure we minimize the risk. The data book is really to lower risk. Someone should look at it and also use the binominal-based procedures in the book before they accept any multi-parameter-based number.”
Although the binomial method is used for all of NASA’s major systems and the multi-parameter approach isn’t used to qualify components, there is a push to adopt it. In fact, Generazio’s research showed that NASA has never accepted even 90/95, but instead always has used near 100 percent detection with 95 percent confidence.
“My goal in life is to get the folks promoting the use of multi-parameter curve fitting methods for estimate probability of detection on NASA systems to validate their methods and POD estimates before they are implemented so their results match up with NASA’s results,” Generazio continued.