In NASA software projects, requirements form the backbone of success, defining what the software must accomplish, how it should behave and the conditions under which it must operate. These requirements deliver mission-critical functionality and ensure software is developed safely, securely and with adherence to NASA’s stringent quality standards.

However, the mere existence of well-documented requirements does not guarantee proper implementation or validation; this is where objective evidence plays a critical role. Objective evidence ensures compliance and accountability across all software engineering requirements outlined in the NASA Software Engineering Handbook (SWEHB).

By embedding the process of identifying, documenting and validating objective evidence into every requirement [as illustrated with SWE-050 (see below)], NASA teams enable transparency and validation throughout the software lifecycle.

What Is Objective Evidence?

Objective evidence refers to tangible, unbiased and documented proof demonstrating that a requirement, activity or task has been appropriately implemented, reviewed and validated. It is independent of opinion, capturing facts through observations, records or artifacts produced by the Software Assurance (SA) and Software Safety teams. All software engineering requirements require identification of objective evidence, with specific examples provided for many of these instances in the SWEHB.

Forms of objective evidence vary based on the requirement being verified, and examples include

• Audit reports and checklists: Used to confirm compliance with specific safety or assurance processes, including tracking issues, risks and resolutions in tools such as Risk Logs.
• Artifact reviews: Documentation verifying the completion and implementation of deliverables such as Software Requirements Specifications (SRS), test procedures and code analysis reports.
• Traceability matrices: Relating requirements to their origin (e.g., mission goals, hazards, verification methods or interfaces) to demonstrate end-to-end implementation and validation.
• Performance metrics: Supporting quantitative analysis tied to requirements, such as system throughput, accuracy percentages or test coverage.
• Signatures and signoffs: Formal acknowledgments from SA personnel confirming that reviews and validations of critical artifacts have occurred.

All NASA software engineering requirements explicitly require objective evidence as outlined in the SWEHB, with guidance on specific examples relevant to each requirement.

Finding Objective Evidence – SWEHB Requirements

The SWEHB provides structured guidance for implementing requirements across various areas of software development and assurance, such as design, testing and risk management. For every requirement listed, the handbook includes descriptions of specific forms of objective evidence that need to be identified and documented. Below, we discuss practices for identifying objective evidence tied to SWEHB requirements.

SWE-050: Software Requirements

SWE-050 mandates that system-level software requirements must be verified, validated and traceable to their respective sources and mission objectives. To establish objective evidence for SWE-050, identify the following items:

• Requirement Verification Artifacts: Evidence of requirements being tested and validated via simulations, reviews or walkthroughs. This includes test cases and results indicating that the requirements behave as defined.
• Traceability Matrices: A documented chain linking each individual software requirement to higher-level system requirements or its corresponding hazard analysis.
• Review Logs: Meeting minutes and attendance records confirming SA involvement in requirement validation activities.
• Signatures and Approval Records: Objective proof of peer reviews conducted on requirement documents, with signoffs confirming validation.

SWE-020: Risk Management

This requirement emphasizes identifying, tracking and mitigating risks throughout software development. Objective evidence for SWE-020 includes

• Risk Logs: Records tracking risks identified during audits, risk assessments or SA reviews.
• Resolution Reports: Documentation of actions taken to address risks, including mitigations and residual risk justifications.
• Meeting Minutes and Follow-Ups: Record of risk management discussions during planning and milestone reviews.

SWE-074: Software Development Testing

SWE-074 focuses on rigorous testing practices to ensure that software conforms to requirements and functions as expected under operational conditions. Objective evidence includes

• Test Results: Reports from unit, integration and system-level tests, demonstrating that all testable requirements have been fully validated.
• Test Coverage Analysis: Metrics showing the percentage of requirements exercised by tests.
• Defect Reports: Issues found during testing, tracked and resolved, with resolutions documented as objective evidence.

SWE-140: Configuration Management

SWE-140 requires software Configuration Management (CM) processes to be established and followed. Objective evidence for SWE-140 includes

• CM Logs: Records of all configuration changes, including approvals, impact analyses and baselines.
• Audit Reports: Verifications of CM practices during compliance checks.
• Version Control Records: Documentation of source code changes, reviewers and associated modifications.

SWE-087: Peer Reviews

SWE-087 mandates peer reviews for all software work products. Objective evidence for this requirement includes

• Review Records: Details of issues identified and resolutions during peer reviews of artifacts such as requirements, design documents, and code.
• Meeting Documentation: Signoff and attendance records for reviews.
• Defect Metrics: Objective evidence of the defect discovery rate and resolution time during peer reviews.

Why Objective Evidence Matters

Deficient requirements—those that are ambiguous, incomplete, unverifiable or not consistently linked to validation methods—are a leading cause of software project failures, contributing to costly rework, safety risks and potentially catastrophic mission outcomes. Objective evidence helps safeguard NASA projects against these risks, ensuring that every requirement is properly implemented and verified.

Key Benefits:

• Traceability: Objective evidence enables direct links between requirements, their origins and their implementation, ensuring nothing is missed during validation.
• Safety and Security Assurance: Critical requirements related to system safety and cybersecurity can be confidently verified with documented proof, minimizing vulnerabilities.
• Change Management: Implementation of objective evidence allows teams to track and document the impact of evolving requirements, supporting better decision-making.
• Transparency and Accountability: Clear documentation of evidence builds confidence among stakeholders, demonstrating that compliance has been achieved.

Best Practices for Capturing Objective Evidence for SWEHB Requirements

Teams should follow these guidelines to ensure compliance and success across NASA software projects:

1. Draw from SWEHB guidance: Review the description of each requirement in the SWEHB, paying close attention to the examples of objective evidence provided (e.g., logs, metrics and validation artifacts).
2. Develop Traceability Matrices: These documents are essential for linking each requirement to its corresponding source, hazard or verification method. Tools like the Dynamic Object-Oriented Requirements System (DOORS) are highly effective for this process.
3. Incorporate SA Involvement: Actively include SA personnel in the creation, review and validation process, ensuring all findings are captured and documented.
4. Monitor Metrics: Track the effectiveness of requirement implementation through relevant metrics such as coverage percentage, defect discovery rates and resolution timelines.
5. Leverage Checklists: Develop checklists for objective evidence identification, tailored to specific requirements in the SWEHB, to ensure completeness.

The Payoff

Investing in thorough and well-documented objective evidence improves compliance with NASA’s software engineering standards and reduces risks across every aspect of project development. Incorporating objective evidence into each SWEHB requirement ensures

1. Risk Mitigation: Reduces rework and delays by validating and implementing requirements from the earliest phases of the project.
2. Enhanced Accountability: Provides transparency at all levels of development, assuring stakeholders that requirements are fully implemented and traceable.
3. Continued Success: Helps future-proof systems against defects, ensuring reliability for NASA’s long-term missions.

Learn More:

• Explore NASA Software Assurance resources.
• Review specific SWEHB guidance (such as SWE-050) for requirements