Preparing your product, your engineering plans, and yourself, for formal verification of requirements before entering a production phase is one of the most critical steps in product realization and is also one that our community can struggle to tailor correctly. We need to find ways to improve the quality and consistency of product verification planning while at the same time passing these skills on to our next generation of engineering leaders.

Systems Engineering Processes
Graphic adapted from the US DoD Systems Engineering Guidebook (red box is mine)

Product Verification Planning is Critical: Be prepared for product verification to consume a substantial portion of your engineering schedule and budget. Analysis published by the RAND Corporation in 2004 suggests defense system testing activities should account for around 15% of total development cost and that these costs are proportional to system complexity. Moreover, the RAND analysis suggests that development and operational testing could account for over 50% of development time (with substantial project-to-project variation). Growing emphasis in cyber-assurance, a specialized and complicated field, is likely to increase these proportions.

A Skill Not Taught in School: Product realization processes and test-forward thinking is not a skill we can assume is being taught in our universities, which are typically development-focused organizations. Intentional investment in mentoring and on-the-job training of these skills for newer-in-career engineers will increase our community’s ability to meet the increased demand for production output that US national security agencies are demanding.

Key Inputs to Product Verification: Plan to conduct product verification activities when the following, critical elements will be sufficiently mature.

  • Configuration Baseline: The in-development product passes basic operational testing and you don’t intend to make significant design change.
  • Requirements Baseline: The set of operational and non-operational product requirements have been captured with associated acceptance criteria. A Requirements Verification Traceability Matrix (RVTM) is a common form to capture this information and assess maturity.
  • Support Systems: Related systems and products needed to conduct verification testing, such as fixtures and test equipment, are available.

Verification Methods: Four methods are commonly used for product verification: Test, Analysis, Demonstration, and Inspection. While testing is likely the preferred method in many scenarios, sometimes the use of the other three methods are required due to test complexity and cost or are simply better fit for purpose.

  • Test: The methodical use of the product to obtain detailed, direct evidence of meeting requirements.
  • Analysis: The use of mathematical modeling or other analytical techniques to predict the product will meet requirements.
  • Demonstration: Showing that the use of the product meets requirements, distinct from the Test method by lacking the collection of detailed test data. For example, showing that the power button turns on/off a product is a form of Demonstration.
  • Inspection: A visual confirmation that the product meets requirements. Commonly used to verify required product features like color, shape, labeling, etc.

Recommendations for Professionals

  • At a minimum, ensure all product development plans have a placeholder for product verification wedged between design activities and production.
  • Budget 10-20% of your cost and 30-50% of your schedule to product verification. If the product is complex and will operate in complex environments, such as a vehicle or aircraft, you should be on the high end of these ranges, at a minimum.
  • If the system must be certified by a regulatory entity, such as the FAA or NHTSA, base your cost and schedule estimates on prior program performance.
  • If this is the first time your team has verified this type of product and it is complex or has certification requirements, find an experienced consultant that can provide advice and guidance.
  • Involve newer-in-career engineers on your team in verification test planning activities. Challenge them think about testing throughout development, not just at the end.

Recommendations for Students

  • Be thoughtful about data collection and analysis when you are in the lab, mentally connecting testing techniques to the technology you are working with.
  • On internships and within the first 90 days of your first job, identify the experienced engineers that always seems to be making decisions five steps ahead and ask them about their experience and approach to product verification.

Final Notes

I hope this short article makes you think about product verification and provides relevant recommendations and resources. I will be discussing Test Readiness Reviews, specifically applied to product verification activities, in my next post. Hope to see you then.

I’d love to hear any additional perspectives or advice for planning and estimating for product verification. Please send to me at chuck@criticalsigma.com.

Bonus Topic: Verification vs Validation

Verification is not the same as validation even those these terms are sometimes used interchangeably. According to the NASA Systems Engineering Handbook, “Verification of a product shows proof of compliance with requirements” while “Validation of a product shows that the product accomplishes the intended purpose in the intended environment”. While we all hope that our requirements could be so well-defined that verification implies validation, that is never the case. Always, always conduct validation testing in the field (or whatever your intended environment is).

On the flip side of the coin, I’ve seen programs structured with only field validation exercises, no lab-based verification testing. This too presents technical risk that you want to avoid. Establishing performance margin with statistical significance is extremely difficult with only field testing. Instead, be thoughtful about what can and cannot be testing in both lab and field testing environments to construct an effective and efficient Verification and Validation test plan for your product.

Useful Free References

NASA System Engineering Handbook
NASA Lessons Learned From Engineering Space Projects at the Marshall Space Flight Center
NASA Example Requirements Verification Matrix
RAND Test and Evaluation Trends and Costs for Aircraft and Guided Weapons
US DoD System Engineering Guidebook
US DoD National Defense Industrial Strategy