One Software Assurance Research Program (SARP) project, “Toward a Guide for Software Defect Tracking, Modeling, and Analysis,” intends to connect software Reliability growth models to practical software defect tracking. SARP — hosted by NASA’s Independent Verification and Validation (IV&V) Program — is sponsoring five research projects in Fiscal Year 2020 aimed to benefit Software Assurance (SA) processes across the agency.

NASA-STD-8739.8A, Software Assurance Standard references software Quality metrics, such as defect density, but these metrics have limited use in practice. Also, NASA-STD-8739.8A requires reporting trends in software Quality metrics but does not provide a detailed approach. 

Ying Shi, Goddard Space Flight Center, and Lance Fiondella, University of Massachusetts Dartmouth, are leading a team to develop joint defect discovery/resolution models and defect life cycle models. Discovery/resolution models can predict how many defects will be discovered as well as the time until all of these defects are resolved. Defect life cycle models use statistics contained in the data from past NASA projects to characterize the stages of the defect tracking process in terms of the logical workflow and time between these stages. 

This more detailed modeling of the defect life cycle can provide insight into where process improvements may be possible. It will enable metrics to be more detailed than just mean time to resolution and practical inferences related to the time to resolve a defect as a function of attributes, such as build and severity. 

The following figure summarizes the team’s most recent analysis, illustrating the time at which defects were resolved for each pair of build found-built target pairs, where build target denotes the build by which the defect was targeted for resolution.

For example, the uppermost timeline 1.0.0-2.0.0 shows the time defects discovered during Build 1.0.0 were resolved relative to Build 2.0.0. Defects occurring after the end of the interval are those that were not removed according to schedule. 

Visualization offers a succinct view of defect tracking performance over an entire project. It can also serve as a roadmap for thorough review of defects that relate to the most pressing SA issues, such as missing or inconsistent requirements that were not resolved in a timely manner. Identifying underlying causes for such defects could capture systematic knowledge for process refinement and lessons learned for future projects instead of becoming lost to the organization with the progression of time as team members depart and memories fade. There is a potential to expand the defect visualization and underlying cause analysis to all of the current Goddard past and ongoing flight projects for future studies.

For more information on this project, contact Shi.

SARP Background

SARP is aligned to support discipline goals to improve how NASA performs SA activities. The research program is designed to provide NASA with greater knowledge about the SA practices, methods and tools needed to produce safe and reliable software.

SARP is designed to address fundamental SA problems in the field of Software Engineering, primarily as it relates to software safety, quality, IV&V, testability and reliability. It is intended to develop and transfer into practice SA technologies, methods and tools to support and improve the quality of the software produced by and for NASA, and to assist the agency in continuing its leadership in the development of safe, reliable and cost-effective software. Thus, by sponsoring forward-thinking research as well as addressing current needs, SARP helps assure that sufficient and appropriate software risk mitigation is applied to the software that controls and monitors NASA’s systems.