NASA Program Sponsors 4 Software Assurance Research Areas in FY21
The NASA Software Assurance Research Program (SARP) is sponsoring four research projects in Fiscal Year (FY) 2021 to benefit Software Assurance processes across the agency. The research program is aligned to support agency goals to improve how NASA performs Software Assurance activities.
Each year, the Software Assurance Working Group (SAWG) identifies initiatives based on current needs in the Software Assurance community, collects research proposals, evaluates their intent, and awards those that best serve the SAWG objectives. Some research initiatives help the SAWG address immediate Software Assurance issues. In contrast, others may address longer-term needs, exploring where software development and Software Assurance need to be in the next five years or so.
“These research groups are working to improve current processes and develop new resources and tools to progress Software Assurance throughout the agency,” said Derek Roesch, SARP manager.
The following are overviews of each FY 2021 SARP project:
Augmenting Requirement Analysis Tool with Artificial Intelligence
Principal Investigator (PI): Chris Williams, TMC Technologies, NASA Independent Verification and Validation (IV&V) Software Assurance Tools Team
Co-PI: Michael Lemasters, TMC Technologies, NASA IV&V Software Assurance Tools Team
The primary goal of this research effort is to help prioritize requirement analysis through the application of Natural Language Processing (NLP) and Machine Learning (ML). The intent is not to replace the analyst but to help the analyst be as efficient as possible. A successful prediction model that could be implemented into a requirements assessment tool, such as the IV&V Facility-developed Analysis Tool Set (ATS), would help direct analysts to requirements identified by ML as potentially problematic.
Research shows that software problems identified in the requirements phase of the software development life cycle have significantly less impact on projects than those specified in later phases. Therefore, it is of mission-critical importance that Software Assurance professionals thoroughly scrutinize NASA software requirements to help identify problems as early as possible. Unfortunately, NASA frequently deals with systems of a size and complexity that make a size and complexity that make a manual inspection of the requirements unmanageable and error-prone. Commercial Off-the-Shelf (COTS) requirement management systems are often cost-prohibitive for a wide-scale deployment throughout the agency. More importantly, they typically only assist with the navigation, viewing, and management of requirements. This approach results in much of the requirements analysis effort being handled outside of the requirements management tool.
The ATS accepts input from most COTS requirement management systems and provides the IV&V Facility with increased flexibility and productivity across the NASA missions it supports. The ATS provides users with assistance in viewing requirements, managing the relationships between the requirements, and collaborating on their analysis. However, it does not provide any automated (or autonomous) assistance in prioritizing that analysis. This project will apply ML to the large ATS dataset to develop prediction models that would help prioritize requirements that need to be analyzed. These models would then be implemented as a feature within the ATS to help identify requirements that are likely to require further analysis.
This research will provide the following value and benefits to Software Assurance:
- New automated requirements analysis capability could help NASA missions identify problems earlier in the requirements phase.
- Automated requirements analysis techniques will help users prioritize manual requirements analysis based on identifying possible concerns, such as ambiguous requirement text.
- Analysts can integrate the results of the ML research into the ATS to enable immediate use of such features in day-to-day analysis work at the IV&V Facility.
- Research work and use of NLP with requirement text present opportunities for “hybrid” approaches, where ML may be applied in some cases and direct implementation of targeted text processing algorithms in other cases.
- Research performed in this effort on the applicability of ML to requirements analysis could also be applied to other textual development data for Software Assurance analysis outside of the specific proposed implementation within the ATS.
- The research will provide a foundation for implementing the concept of an overall “Quality Score” for individual requirements based on automated analysis results, Software Assurance assessment criteria (e.g., Quality Assurance checklist results), and possibly other requirements metrics.
- Possible follow-on efforts could be targeted to help configure the ATS for use elsewhere within NASA for Software Assurance practitioners to take advantage of the automated requirements analysis capabilities.
Dependency Structure Matrix CAP Integration
PI: Chris Williams, IV&V Software Assurance Tools Team
Co-PI: Jerry Williams, IV&V Software Assurance Tools Team
This research intends to enhance the NASA IV&V Facility-developed Code Analysis Pipeline (CAP) system with Dependency Structure Matrix (DSM) capabilities. DSM representation of a software project’s dependencies provides an intuitive way for Software Assurance practitioners, such as IV&V analysts, to view the relationships between software components. Further, this effort will leverage file-level dependency information mapped to system-level components to allow analysts to consider the software project from an architectural perspective, not just as a collection of source files.
Analysts originally developed the CAP as a research effort to automate the execution of multiple static source code analysis tools to eliminate portions of the manual effort required by analysts, combine the results of these several tools in a single place and allow repeated executions tools as the software changes. Because the CAP is not just limited to running static analysis tools that search for software defects, Software Assurance practitioners updated it to run a dependency tool against the software under analysis. The CAP collects dependency information from this tool to display a DSM visualization to CAP users, which aids understanding of source code dependencies and cycle detection.
Mapping source code files to logical components in the system architecture will allow analysts to view dependencies between system components, not just source files. The CAP’s static analysis capabilities, coupled with the logical component structure and DSM capability, will allow users to identify system components that need focus due to high numbers of potential defects. For example, an increased number of possible defects in the source files related to a specific architectural component indicates a “hot spot” requiring additional scrutiny. DSM dependency data helps analysts understand the impact the defects in that component would have on other parts of the system.
This research will provide the following value and benefits to Software Assurance:
- DSM capabilities will better understand a software project’s dependencies, and CAP DSMs will provide this view at an architectural level. These capabilities will help both developers and Software Assurance personnel better understand the system, identify architectural concerns (e.g., cyclic dependency), and assess the defect and change impacts.
- Joining logical architectural components with software file dependency data, static analysis results, and source code metrics (e.g., complexity) will allow assurance personnel to identify and focus analysis on system components of highest concern, mitigate potential defects, and verify design and requirements traceability, as implemented.
- Static analysis results, dependency information, and metrics collected through the CAP will aid analysts in assessing the quality attributes of a software system as part of a code quality risk assessment. The CAP could be further developed to implement this type of assessment directly in the tool to support assessment data collection, status dashboards, and reporting.
- Because the CAP is an infrastructure that supports the automated execution of tools against software projects, it will continually evolve to add new capabilities. Software Assurance personnel who leverage this toolset will have access to that new capability as it evolves.
Advancing the Requirements Review Approach with NLP
PI: Mikael Lindvall, Fraunhofer Center Mid-Atlantic (CMA)
Co-PI: Ying Shi, NASA Goddard Space Flight Center, and Madeline Diep, Fraunhofer CMA
Requirements specification flaws are significant contributions to most software-related defects. Ambiguous, incomplete, vague, untestable, and missing requirements are typical problems with those specifications. Such often go undetected because requirements are specified to make them difficult to review and analyze manually.
This project aims to investigate how NLP, an ML-based approach for analyzing natural language, can facilitate the requirements review activity. Specifically, the group is exploring two applications of NLP:
- To create groups of requirements that can be examined together. Researchers conjecture that by analyzing a group of related requirements, they can identify requirement inconsistencies and incompleteness.
- To automatically identify requirements with complementary antonyms and detect whether the complementary requirements are present in the requirement document. Complementary antonyms are word pairs that have opposite meanings representing discrete states (e.g., open and close). These pairs typically generate a complete set of requirements, and the presence of one word, but not its complement, can indicate a missing requirement.
Researchers are developing a tool that integrates the NLP-based and complementary word analyses to find missing software requirements. They will apply their integrated approach to a set of NASA requirements to evaluate the efficacy of the solution set and identify its strength and weaknesses for future research direction.
Software Defect Proneness: Discovering the Metrics that Matter Most
PI: Katerina Goseva-Popstojanova, West Virginia University, Morgantown, West Virginia
Co-PI: Noble Nkwocha, NASA Katherine Johnson IV&V Facility, Fairmont, West Virginia
This SARP project investigates the defect proneness of NASA missions’ software using quantitative and qualitative methods. The goal is to discover the metrics that matter most for identifying and predicting defect-prone parts of the software systems and then use that information to efficiently and effectively conduct Software Assurance and improve software quality. The empirical work will be based on the flight software component of a large NASA mission.
The main tasks of this SARP’s project are as follows:
- Extract metrics from different software artifacts and pre-process the data to ensure data quality.
- Conduct quantitative analysis, including descriptive statistics to characterize the collected metrics and inferential statistics to quantify the level of correlation with the number of defects. In addition to the quantitative analysis based on the NASA mission under study, the project will include a meta-analysis for those research questions that published works have previously explored.
- Conduct qualitative analysis of the most defect-prone subsystems, which will provide in-depth insights into the reasons that lead to defect proneness of software systems.
- Use ML to predict defect proneness, which the developers and IV&V analysts could use to prioritize their Software Assurance efforts.
Identifying software metrics that are highly correlated with the number of defects will help developers and IV&V analysts focus their efforts on preventing, detecting, and eliminating software defects in the most effective ways, at the most effective time(s). The evidence-based findings of this project will also benefit other projects that undergo iterative development and must be sustained in engineering for a long time. In addition, the lessons learned from this SARP project will contribute toward improving the practical usefulness and quality of the data collected by NASA projects.
About SARP
SARP is a Headquarters Software Assurance program delegated to the IV&V Program’s Safety and Mission Assurance Support Office. It addresses fundamental Software Assurance problems in the field of Software Engineering. It helps NASA Software Assurance personnel stay current with new practices, methods, and tools to produce safe and reliable software. The researchers have a year to develop, analyze, test, and record findings, which they will share across the agency and present to the SAWG at the end of the year.
SARP directly supports
- Improving the risk, issue and finding reporting from the NASA Software Assurance and software safety organizations
- Adding value for Software Assurance and software safety activities
- Demonstrating the importance of the NASA Software Assurance activities
- Providing standard tools and services for Software Assurances activities on projects
- Focusing Software Assurance activities on known software issues, including targeting Software Assurance and software safety research activities
- Developing more efficient and automated methods for Software Assurance activities.
To find out more about current and past SARP initiatives, visit SARP’s NASA Engineering Network or the Software Assurance web page or contact Roesch.