NASA Program Sponsors 4 Software Assurance Research Areas in FY21

NASA Program Sponsors 4 Software Assurance Research Areas in FY21

8-minute read

The NASA Software Assurance Research Program (SARP) is sponsoring four research projects in Fiscal Year (FY) 2021 to benefit Software Assurance processes across the agency. The research program is aligned to support agency goals to improve how NASA performs Software Assurance activities.

Each year, the Software Assurance Working Group (SAWG) identifies initiatives based on current needs in the Software Assurance community, collects research proposals, evaluates their intent, and awards those that best serve the SAWG objectives. Some research initiatives help the SAWG address immediate Software Assurance issues. In contrast, others may address longer-term needs, exploring where software development and Software Assurance need to be in the next five years or so.

“These research groups are working to improve current processes and develop new resources and tools to progress Software Assurance throughout the agency,” said Derek Roesch, SARP manager.

The following are overviews of each FY 2021 SARP project:

Augmenting Requirement Analysis Tool with Artificial Intelligence

Principal Investigator (PI): Chris Williams, TMC Technologies, NASA Independent Verification and Validation (IV&V) Software Assurance Tools Team

Co-PI: Michael Lemasters, TMC Technologies, NASA IV&V Software Assurance Tools Team

The primary goal of this research effort is to help prioritize requirement analysis through the application of Natural Language Processing (NLP) and Machine Learning (ML). The intent is not to replace the analyst but to help the analyst be as efficient as possible. A successful prediction model that could be implemented into a requirements assessment tool, such as the IV&V Facility-developed Analysis Tool Set (ATS), would help direct analysts to requirements identified by ML as potentially problematic.

Research shows that software problems identified in the requirements phase of the software development life cycle have significantly less impact on projects than those specified in later phases. Therefore, it is of mission-critical importance that Software Assurance professionals thoroughly scrutinize NASA software requirements to help identify problems as early as possible. Unfortunately, NASA frequently deals with systems of a size and complexity that make a size and complexity that make a manual inspection of the requirements unmanageable and error-prone. Commercial Off-the-Shelf (COTS) requirement management systems are often cost-prohibitive for a wide-scale deployment throughout the agency. More importantly, they typically only assist with the navigation, viewing, and management of requirements. This approach results in much of the requirements analysis effort being handled outside of the requirements management tool.

The ATS accepts input from most COTS requirement management systems and provides the IV&V Facility with increased flexibility and productivity across the NASA missions it supports. The ATS provides users with assistance in viewing requirements, managing the relationships between the requirements, and collaborating on their analysis. However, it does not provide any automated (or autonomous) assistance in prioritizing that analysis. This project will apply ML to the large ATS dataset to develop prediction models that would help prioritize requirements that need to be analyzed. These models would then be implemented as a feature within the ATS to help identify requirements that are likely to require further analysis.

This research will provide the following value and benefits to Software Assurance:

  • New automated requirements analysis capability could help NASA missions identify problems earlier in the requirements phase.
  • Automated requirements analysis techniques will help users prioritize manual requirements analysis based on identifying possible concerns, such as ambiguous requirement text.
  • Analysts can integrate the results of the ML research into the ATS to enable immediate use of such features in day-to-day analysis work at the IV&V Facility.
  • Research work and use of NLP with requirement text present opportunities for “hybrid” approaches, where ML may be applied in some cases and direct implementation of targeted text processing algorithms in other cases.
  • Research performed in this effort on the applicability of ML to requirements analysis could also be applied to other textual development data for Software Assurance analysis outside of the specific proposed implementation within the ATS.
  • The research will provide a foundation for implementing the concept of an overall “Quality Score” for individual requirements based on automated analysis results, Software Assurance assessment criteria (e.g., Quality Assurance checklist results), and possibly other requirements metrics.
  • Possible follow-on efforts could be targeted to help configure the ATS for use elsewhere within NASA for Software Assurance practitioners to take advantage of the automated requirements analysis capabilities.

Dependency Structure Matrix CAP Integration

PI: Chris Williams, IV&V Software Assurance Tools Team

Co-PI: Jerry Williams, IV&V Software Assurance Tools Team

This research intends to enhance the NASA IV&V Facility-developed Code Analysis Pipeline (CAP) system with Dependency Structure Matrix (DSM) capabilities. DSM representation of a software project’s dependencies provides an intuitive way for Software Assurance practitioners, such as IV&V analysts, to view the relationships between software components. Further, this effort will leverage file-level dependency information mapped to system-level components to allow analysts to consider the software project from an architectural perspective, not just as a collection of source files.

Analysts originally developed the CAP as a research effort to automate the execution of multiple static source code analysis tools to eliminate portions of the manual effort required by analysts, combine the results of these several tools in a single place and allow repeated executions tools as the software changes. Because the CAP is not just limited to running static analysis tools that search for software defects, Software Assurance practitioners updated it to run a dependency tool against the software under analysis. The CAP collects dependency information from this tool to display a DSM visualization to CAP users, which aids understanding of source code dependencies and cycle detection.

Mapping source code files to logical components in the system architecture will allow analysts to view dependencies between system components, not just source files. The CAP’s static analysis capabilities, coupled with the logical component structure and DSM capability, will allow users to identify system components that need focus due to high numbers of potential defects. For example, an increased number of possible defects in the source files related to a specific architectural component indicates a “hot spot” requiring additional scrutiny. DSM dependency data helps analysts understand the impact the defects in that component would have on other parts of the system.

This research will provide the following value and benefits to Software Assurance:

  • DSM capabilities will better understand a software project’s dependencies, and CAP DSMs will provide this view at an architectural level. These capabilities will help both developers and Software Assurance personnel better understand the system, identify architectural concerns (e.g., cyclic dependency), and assess the defect and change impacts.
  • Joining logical architectural components with software file dependency data, static analysis results, and source code metrics (e.g., complexity) will allow assurance personnel to identify and focus analysis on system components of highest concern, mitigate potential defects, and verify design and requirements traceability, as implemented.
  • Static analysis results, dependency information, and metrics collected through the CAP will aid analysts in assessing the quality attributes of a software system as part of a code quality risk assessment. The CAP could be further developed to implement this type of assessment directly in the tool to support assessment data collection, status dashboards, and reporting.
  • Because the CAP is an infrastructure that supports the automated execution of tools against software projects, it will continually evolve to add new capabilities. Software Assurance personnel who leverage this toolset will have access to that new capability as it evolves.

Advancing the Requirements Review Approach with NLP

PI: Mikael Lindvall, Fraunhofer Center Mid-Atlantic (CMA)

Co-PI: Ying Shi, NASA Goddard Space Flight Center, and Madeline Diep, Fraunhofer CMA

Requirements specification flaws are significant contributions to most software-related defects. Ambiguous, incomplete, vague, untestable, and missing requirements are typical problems with those specifications. Such often go undetected because requirements are specified to make them difficult to review and analyze manually.

This project aims to investigate how NLP, an ML-based approach for analyzing natural language, can facilitate the requirements review activity. Specifically, the group is exploring two applications of NLP:

  1. To create groups of requirements that can be examined together. Researchers conjecture that by analyzing a group of related requirements, they can identify requirement inconsistencies and incompleteness.
  2. To automatically identify requirements with complementary antonyms and detect whether the complementary requirements are present in the requirement document. Complementary antonyms are word pairs that have opposite meanings representing discrete states (e.g., open and close). These pairs typically generate a complete set of requirements, and the presence of one word, but not its complement, can indicate a missing requirement.

Researchers are developing a tool that integrates the NLP-based and complementary word analyses to find missing software requirements. They will apply their integrated approach to a set of NASA requirements to evaluate the efficacy of the solution set and identify its strength and weaknesses for future research direction.

Software Defect Proneness: Discovering the Metrics that Matter Most

PI: Katerina Goseva-Popstojanova, West Virginia University, Morgantown, West Virginia

Co-PI: Noble Nkwocha, NASA Katherine Johnson IV&V Facility, Fairmont, West Virginia

This SARP project investigates the defect proneness of NASA missions’ software using quantitative and qualitative methods. The goal is to discover the metrics that matter most for identifying and predicting defect-prone parts of the software systems and then use that information to efficiently and effectively conduct Software Assurance and improve software quality. The empirical work will be based on the flight software component of a large NASA mission.

The main tasks of this SARP’s project are as follows:

  1. Extract metrics from different software artifacts and pre-process the data to ensure data quality.
  2. Conduct quantitative analysis, including descriptive statistics to characterize the collected metrics and inferential statistics to quantify the level of correlation with the number of defects. In addition to the quantitative analysis based on the NASA mission under study, the project will include a meta-analysis for those research questions that published works have previously explored.
  3. Conduct qualitative analysis of the most defect-prone subsystems, which will provide in-depth insights into the reasons that lead to defect proneness of software systems.
  4. Use ML to predict defect proneness, which the developers and IV&V analysts could use to prioritize their Software Assurance efforts.

Identifying software metrics that are highly correlated with the number of defects will help developers and IV&V analysts focus their efforts on preventing, detecting, and eliminating software defects in the most effective ways, at the most effective time(s). The evidence-based findings of this project will also benefit other projects that undergo iterative development and must be sustained in engineering for a long time. In addition, the lessons learned from this SARP project will contribute toward improving the practical usefulness and quality of the data collected by NASA projects.

About SARP

SARP is a Headquarters Software Assurance program delegated to the IV&V Program’s Safety and Mission Assurance Support Office. It addresses fundamental Software Assurance problems in the field of Software Engineering. It helps NASA Software Assurance personnel stay current with new practices, methods, and tools to produce safe and reliable software. The researchers have a year to develop, analyze, test, and record findings, which they will share across the agency and present to the SAWG at the end of the year.

SARP directly supports

  • Improving the risk, issue and finding reporting from the NASA Software Assurance and software safety organizations
  • Adding value for Software Assurance and software safety activities
  • Demonstrating the importance of the NASA Software Assurance activities
  • Providing standard tools and services for Software Assurances activities on projects
  • Focusing Software Assurance activities on known software issues, including targeting Software Assurance and software safety research activities
  • Developing more efficient and automated methods for Software Assurance activities.

To find out more about current and past SARP initiatives, visit SARP’s NASA Engineering Network or the Software Assurance web page or contact Roesch.



Tim Crumbley

SA Technical Fellow

Learn more about SA Technical Fellow Tim Crumbley.

Read More

Guille del Carmen

Technical Discipline Team Lead

Learn more about SA Technical Discipline Team Lead Guille del Carmen.

Read More

Points of Contact

For details on contacting an SA Point of Contact (PoC), click below.

Find Your PoC

Software Assurance Working Group

The Software Assurance Working Group (SAWG) is a group of Software Assurance (SA) professionals from across NASA who work together to help formulate NASA SA policy, standards, training, guidance, briefings and other needed items. It is also a forum to share experiences, lessons learned and useful techniques. The SAWG provides a community that can provide assistance and support to individual practitioners.

The group meets twice a month (second and fourth Wednesdays), with the second meeting of the month devoted to supporting SA Technical Excellence efforts.

SA Working Group 2022 

Past Events

Event Date Description
4 Types of Peer Reviews
12/15/2021 Webinar covering similarities and differences between the four basic types of peer review  
Software Quality Risk Scoring Workshop Part I 6/8/2021 First day of the virtual Software Quality Risk Scoring Workshop.  
Software Quality Risk Scoring Workshop Part II 6/8/2021 First day of the virtual Software Quality Risk Scoring Workshop.  
Software Quality Risk Scoring Workshop Part III 6/9/2021 Second day of the virtual Software Quality Risk Scoring Workshop.  
Software Quality Risk Scoring Workshop Part IV 6/9/2021 Second day of the virtual Software Quality Risk Scoring Workshop.

NASA Software Assurance Program Goals

  1. Provide risk-based performance requirements that provide flexibility for the project Software Assurance and Software Safety activities.
  2. Improve the risk, issue and finding reporting from the NASA Software Assurance and Software Safety organizations.
  3. Add value for Software Assurance and Software Safety activities and demonstrate the importance of the NASA Software Assurance activities.
  4. Provide standard tools and services for Software Assurances activities on projects.
  5. Provide measurable Software Assurance process improvement.
  6. Improve the use of data and metrics on all NASA Software Assurance activities.
  7. Focus Software Assurance activities on known software issues, including targeting Software Assurance and Software Safety research activities.
  8. Develop more efficient and automated methods for Software Assurance activities.
  9. Establish a Software Assurance services and tool sharing capability.
  10. Improve Software Assurance training and training requirements in the Safety and Mission Assurance Technical Excellence Program and across the agency.

IV&V Program

NASA’s Independent Verification and Validation (IV&V) Program provides assurance that safety- and mission-critical systems and software will operate reliably, safely and securely. The NASA IV&V Program's primary location is the Katherine Johnson IV&V Facility in Fairmont, West Virginia. The IV&V Program provides the following services:

  • System and Software Assurance: Full life cycle IV&V and independent assessments for NASA’s highest profile missions. IV&V leads to higher quality products, reduced risk, greater insight, reduced cost and knowledge transfer.
  • Safety and Mission Assurance (SMA) Support: Support across the agency, in-line with the development project. Hazard Analysis, Software Assurance plan development, and standards development and evaluation.
  • Mission Protection Services: Vulnerability assessment and authorization, end-to-end full life cycle security risk assessment, FedRAMP 3PAO (cloud) services, security training, and security testing (penetration testing, code analysis and vulnerability scanning).
  • Software Development, Testing and Research: Independent testing, automation and virtualization enabled through IV&V’s Jon McBride Software Testing and Research Laboratory.
IV&V Program


Launch SATERN My STEP STEP Software Assurance Curriculum Guide 

SATERN Courses

Course Title Course Number Buttons
Intermediate Software Assurance SMA-SA-WBT-201 SMA-SA-WBT-201 Details
Introduction To Software Engineering SMA-SA-WBT-206 SMA-SA-WBT-206 Details
Intermediate Software Testing SMA-SA-WBT-301 SMA-SA-WBT-301 Details
Software Requirements, Development and Management SMA-SA-WBT-303 SMA-SA-WBT-303 Details
Software Safety For Practitioners SMA-SA-WBT-306 SMA-SA-WBT-306 Details
Software Processes and Metrics SMA-SA-WBT-402 SMA-SA-WBT-402 Details

Policy and Guidance




The purpose of the Software Assurance and Software Safety Standard is to define the requirements to implement a systematic approach to Software Assurance, software safety, and Independent Verification and Validation (IV&V) for software created, acquired, provided, or maintained by or for NASA. The Software Assurance and Software Safety Standard provides a basis for personnel to perform software assurance, software safety, and IV&V activities consistently throughout the life of the software, that is, from its conception, through creation to operations and maintenance, and until the software is retired.

See NASA-STD-8739.8 


NASA Software Engineering Handbook

This handbook provides users and practitioners with guidance material for implementing the requirements of NPR 7150.2, NASA Software Engineering Requirements and the implementation of the NASA Software Assurance and Software Safety requirements in NASA-STD-8739.8, Software Assurance Standard. The use of this handbook is intended to provide "best-in-class" guidance for the implementation of safe and reliable software in support of NASA projects. The handbook is a key component of an agencywide plan to work toward a continuous and sustained Software Engineering and Software Assurance process and product improvement.

See NASA-HDBK-2203

Additional Guidance

Policy Title Buttons Buttons
NASA-STD-8739.8 NASA Software Assurance Standard NASA-STD-8739.8 Details See NASA-STD-8739.8
NPD 7120.4 NASA Engineering and Program/Project Management Policy NPD-7120-4 Details See NPD 7120.4
NPR 7120.5 NASA Space Flight Program and Project Management Requirements NPR-7120-5 Details See NPR 7120.5
NPR 7123.1 Systems Engineering Processes and Requirements NPR-7123-1 Details See NPR 7123.1
NPR 7150.2 Software Engineering Requirements NPR-7150-2 Details See NPR 7150.2
NASA-GB-8719.13 NASA Software Safety Guidebook NASA-GB-8719-13 Details See NASA-GB-8719.13
NASA-STD-8739.9 NASA Software Formal Inspections Standard NASA-STD-8739-9 Details See NASA-STD-8739.9
NASA-HDBK-8739.23 Complex Electronics Handbook for Assurance Professionals NASA-HDBK-8739.23 Details See NASA-HDBK-8739.23
SSP 50038 Computer-Based Control System Safety Requirements SSP 50038 Details See SSP 50038


Software Assurance Research Program

The Software Assurance Research Program (SARP) — hosted by NASA’s Independent Verification and Validation (IV&V) Program — is designed to provide NASA with greater knowledge about the Software Assurance (SA) practices, methods and tools needed to produce safe and reliable software.

SARP is designed to address fundamental SA problems in the field of software engineering, primarily as it relates to software safety, quality, IV&V, testability and reliability. It is intended to develop and transfer into practice SA technologies, methods and tools to support and improve the quality of the software produced by and for NASA, and to assist the agency in continuing its leadership in the development of safe, reliable and cost-effective software. Thus, by sponsoring forward-thinking research as well as addressing current needs, SARP helps assure that sufficient and appropriate software risk mitigation is applied to the software that controls and monitors NASA’s systems.

In Fiscal Year 2021, SARP is sponsoring four research projects aimed to benefit Software Assurance processes across the agency:

  1. Augmenting Requirement Analysis Tool with Artificial Intelligence
  2. Dependency Structure Matrix CAP Integration
  3. Advancing the Requirements Review Approach with NLP
  4. Software Defect Proneness: Discovering the Metrics that Matter Most

Visit SARP Website Visit SARP NEN Website