Researchers completed 10 NASA Software Assurance Research Program (SARP)-sponsored research projects aimed to benefit Software Assurance (SA) processes across the agency and presented the results to the Software Assurance Working Group (SAWG). After SAWG selected initiatives in 2017, researchers were given a year to develop, analyze, test, record findings and present results.
SARP is a Headquarters SA program that is delegated to the Independent Verification and Validation (IV&V) Program’s Safety and Mission Assurance Support Office. It is designed to address fundamental SA problems in the field of software engineering and stay current with new practices, methods and tools needed to produce safe and reliable software. Each year, the SAWG identifies initiatives based on current needs in the SA community, collects research proposals, evaluates their intent and awards those that would best serve the SAWG objectives. Some SA issues are immediate and others that SARP awards are longer term, exploring where software development and SA need to be in the next 5 years or so.
“SARP, in particular, is important because it gives the SA community across the agency a chance to explore issues they have, find ways they can improve SA and create new tools to do their jobs more efficiently,” said Scott Benton, SARP manager. “Before we put out a call for proposals, we meet with the SA community and the Office of Safety and Mission Assurance Software Assurance Technical Fellow to figure out what solutions we need, problems we’re having and challenges we see in the future that we want to get in front of.”
The following are overviews and results of each Fiscal Year (FY) 2018 SARP project:
Actionable Reliability Assurance
Reliable software, while desired, is often considered infeasible and this work helps bring a practical approach to performing software reliability. The Actionable Reliability Assurance (ARA) research proposed to extend and adapt the Jet Propulsion Laboratory’s Mission Design and Navigation (MDN) software group’s Reliability models and methods available to assurance practitioners across the agency.
Results: The research team drafted documentation of MDN’s Reliability assurance methodology, implemented guidelines for ARA and submitted a paper on validation of ARA. The paper “Actionable Analytics for Strategic Maintenance of Critical Software” was accepted for presentation at the 2018 International Conference of Engineering, Science and Mathematics Education.
Adapting SA Objectives Hierarchy to Model-Based Reuse
As NASA projects evolve to model-based approaches, NASA needs SA to be prepared with tools and processes that support the development of these approaches. This project is one of the steps the agency is taking to meet that challenge. The main objective of this project was to retrospectively investigate and characterize the extent of assurance that can be obtained using the Software Assurance Objectives Hierarchy (SOH) and Risk-Informed Safety Cases (RISC) concepts.
Results: Main results from the research include an assessment of the SOH; consideration of its applicability to the Validation and Verification (V&V) of flight software for the BioSentinel mission; and preliminary mappings between concepts of the SOH, RISC, and current NASA standards. The next step for researchers is to use these findings to specify tool requirements for supporting the use of SOH and RISC.
Assurance for Autonomy
Several NASA projects have various levels of autonomous software and as the agency explores deep space and the planets, more will be needed. There are unique issues with autonomous software that need to be understood in order to assure them. The goal of this project was to develop methods and associated tools for making informed decisions for assurance of autonomous software and operations using a modeling approach.
Results: Researchers found the following information through this project:
- How key areas of uncertainty and their corresponding SA hazards for each class contribute to the Risk Management of autonomous systems.
- The main focus of approaches for V&V of autonomy software is managing the risks of the autonomous software as an independent module.
- The main area of concern for autonomous operations is the difference in the assumptions and knowledge of the state of the world by the ground team and the autonomous system and the risk of overriding the decisions made by the autonomous software.
cFS Flight Software Verification & Validation
The core Flight System (cFS) architecture and framework approach to flight software is a platform and project-independent reusable software framework and set of reusable software applications suitable for reuse on any number of NASA flight projects and embedded software systems. The cFS has successfully been used on many missions, as reusability provides a stable approach and saves cost and time. This project aimed to establish if the open-source cFS software represents improved software Quality in addition to potentially controlling cost and schedule, identifying the reusable elements of V&V, and establishing a template for tailoring V&V of cFS.
Results: Researchers found the following information through this project:
- Static code analysis (SCA) of the open-source cFS-related code suggests that, with the exception of string processing, the cFS software is generally free of the kinds of potential bugs that SCA readily identifies. The source code generally holds that lead to understandable and maintainable software standards, while related warnings suggest a target for improvement.
- Multiple software projects are not performing structured verification of their cFS software, assuming it was done elsewhere and that was sufficient despite their different environments or usage.
- There are substantial variations between cFS instances, while “families” of similar software and hardware instances limit the level of novelty between all instances that hold the potential for longer-term full-compliance with best engineering practices and constrained costs.
- The reuse of software Class B and expansion of the user community to SW Class A safety-critical software are at least partially based on the perception of cFS success within SW Class B uses rather than detailed consideration of cFS assurance evidence, while others appear to be reaching nearly comprehensive V&V of their cFS instance.
- There are signs of unresolved problem reports not considered by potential future users. An announced draft change to agency-level instructions that removes the expectation of all future Off-The-Shelf (OTS) software users who review OTS software vendor-reported defects to ensure the defects do not impact the selected software components.
Flight Software and Security Qualification
Recently, there has been a significant increase in interest within NASA for utilizing an open-source flight software "stack" consisting of an open-source Real-Time Operating System (RTOS) coupled with mission-specific flight software that executes within Goddard Flight Research Center’s open-source core cFS.
The team worked to enhance the NASA Operational Simulator and Small Satellites (NOS3) and cFS Kick Start projects by expanding their scopes to include open-source RTOS considerations, qualification considerations and security considerations with onboardâflight software.
Results: This research focused on developing space mission security guidelines for applying to new missions, particularly Class D missions. The guidelines described security best practices, how to apply them to new missions and implement consultative committee for space data systems flight software security by utilizing a cryptography Library for cFS. The NoS3, newly open-sourced, was utilized as a framework for applying CryptoLib and the guidelines.
GPU-Accelerated Monte Carlo Simulations
The overall goal was to deliver and support a versatile, mission-generic, open-access independent test capability Graphic Processing Unit (GPU)-accelerated Monte Carlo numerical simulation infrastructure (inclusive of specialized hardware and software libraries) and to evaluate and showcase its test bed potential on the current Parker Solar Probe Guidance, Navigation and Control flight software.
Results: This research produced how-to tutorials and a comprehensive introduction on the NVIDIA CUDA programming language. The GPU software was installed and optimized, and CUDA software libraries were developed. The research team developed a GPU-accelerated Monte Carlo simulation infrastructure and is currently testing and evaluating. This infrastructure will soon be available for use by NASA missions to help detect where faults are most likely to occur.
Modeling Requirements for Autonomy
As the need for autonomous software increases, the use of models to help design and assure them are evolving. The agency needs to be prepared for a SA approach that helps understand and identify risks. The purpose of this project was to identify and evaluate modeling methods that can be used to better understand and reason about autonomy requirements.
Results: The group defined a three-tier modeling approach that models various aspects of autonomous system behavior such as goals, tasks, required data, data dependencies and agents. They also enumerated a number of heuristics that can be leveraged for checking completeness and consistency in the requirements.
Practical Software Reliability
Software reliability as part of the system reliability is a growing concern, and with the already overloaded SA personnel to take on this additional task, we need to make it more manageable. The goal of this project was to apply the Software Failure and Reliability Assessment Tool to NASA programs to demonstrate its potential to provide more detailed oversight of software Reliability at various stages of development and testing.
Results: The team implemented a customizable script to achieve a higher level of automation in order to enable the application of methods to assess software Reliability at multiple points throughout a program. To further encourage adoption, they will apply the approach to a historical NASA program and document how an ongoing program could benefit from periodic analysis.
SAPE Deployment
The Software Assurance Planning and Execution (SAPE) tool deployment project was carried over from FY18. The SAPE tool was created within SARP several years ago to help the SA personnel determine which SA requirements should be applied to a project and the level of effort needed to accomplish the tailored, risk-based approach for each project. The SA plans of action and tasks are recorded and the results at each project phase are captured and evaluated. SAPE maintains and follows the work on several projects and can be used for cross-project evaluations. The goal of this stage of the project was to finalize the SAPE tool for operational use; bring it into alignment with the updates to the SA and software safety standards; implement the tool on a Glenn Research Center-hosted server; provide maintenance and user support; and implement usability into the tool. The SAPE plugin for JIRA provides support for tracking SA tasks, audits and associated findings, as well as requirements compliance. It allows for automated creation of workflows using text-based template files, directly based on requirements documentation.
Results: This year, the system was moved to full production status in the Glenn data center with NAMS integration, single sign-on and multiple secure partitions for users from multiple NASA centers. The team integrated the new SA standards requirements into the workflow creation system and is currently updating the metrics dashboard.
Safety-Critical Requirements Analysis and Tool
Software safety criteria and requirements do not always flow directly and cleanly from the preliminary hazard analyses, but can show up in the software requirements in other ways. The goal of the Safety-Critical Requirements (SCR) analysis SARP project was to develop guidelines for identifying possible additional safety critical requirements and help find derived software safety requirements by applying Natural Language Process (NLP) and Machine Learning (ML) algorithm for SCR identification process. The project also helped define SA personnel roles and responsibilities during the analyses phase and beyond.
Results: This effort resulted in the creation of an Excel tool and the application of NLP and ML algorithms to improve the SCR analysis process. In addition, the effort produced detailed guidelines to support the analysis of SCR; the guidelines included recommended SCR analysis activities during the project life cycle as well as software safety roles and responsibilities.
SARP researchers are currently working on eight initiatives through FY19. To find out more about current and past SARP initiatives visit SARP’s NASA Engineering Network or contact Benton or SA Program Manager and Technical Fellow Martha Wetherholt.