EVT - It's like TSA precheck for Verification & Validation

Picture this -- you've raced to the airport and found yourself at the end of a massive security line. You do the truffle shuffle with shoes off, belt off, keys out, laptop out, while the security agent barks at you about fluids in your carry-on bag, as if this is your first time ever flying.

You walk through the scanner and then remember, "Doh, I left the regular-sized toothpaste in my carry-on!" So, the security agent shuttles you and your bag aside, sifts through your personables, lectures you on the max volume of fluids, and finally sends you on your way.

But that's not all -- you then realize that you are in Terminal A, and your gate is in ... Terminal B. Gulp. And as luck would have it, Terminals A and B are completely disconnected. You'll need to trek to the other terminal, and wait in another unbearably long security line.

Your chances of getting there in time for your original flight -- zero. Many hours delayed, additional change fees incurred, and the call home to let your spouse know you won't be home for dinner.
As you proceed with the walk of shame back into the main ticketing area, you notice this other line, where people simply walk through -- shoes on, laptops stowed, chill security agents, and smiley, happy faces all around.

Yep, that's TSA pre-check. Those folks took a few extra steps up-front, which afforded them the ability to coast through security with convenience and confidence.
If you're a fan of TSA pre-check, then you should also be a fan of EVT.

What exactly is EVT?

EVT stands for Engineering Verification Testing. It's the work that is done before freezing your design and jumping into the formal Verification and Validation (V&V) process.
It's the step that finds the mistakes and failures before they turn into big headaches. It avoids the massive paper trail. It prevents you from re-running expensive validation tests after minor design tweaks have been implemented. And it saves you time in the long-run.

But, EVT is the step that most medical device companies skip in hopes of saving time. Unfortunately, the intended time saver almost always turns into a major time sink.
Here are 3 simple tips for completing EVT effectively and ensuring a smooth process through the V&V phase.

TIP #1
Earmark a specific place for EVT in your development process.

If you don't have a formalized product development process (PDP), start by putting one together. HERE's a link to a separate article on just that topic. A PDP will help you and your team align on approach, utilize resources effectively, and determine when "done" is actually done.

EVT should come after production equivalent parts are made but before freezing the design. The level of design control depends on your quality management system (QMS), but we establish EVT under "Engineering-Level" controls. This means that parts are numbered, entered into product data management (PDM), have dimensioned drawings / specifications, and referenced in Bills of Materials (BOM).

During EVT, there is more flexibility than you will have during the V&V process. For instance, supplier deficiencies during EVT may not need to run through NCMRs (Non-Conformance Material Reports). Additionally, deviations from formal V&V protocols may be acceptable while you are dialing in the testing process. Furthermore, EVT failures may not require full risk management review.

By contrast, once the design is frozen, these and other processes will be fairly rigid. When parts arrive out-of-spec, NCMRs will need to be issued. When the test plan needs to be revised due to equipment or sample problems, new protocols must be generated, reviewed, approved, and recorded in the Design History File (DHR). When V&V failures occur, the risk implications will need to be evaluated, documented, and mitigated appropriately.

The punchline here is that changes occurring during V&V will be much heavier. So, it's best to discover these issues during the more lightweight EVT process.

TIP #2
Toss the 3D printed parts.

3D printing is great. But (most of the time), parts created through 3D printing are not production intent. When entering EVT and the V&V process, it's really best to be using production grade materials and processes. Why is this?

For starters, the performance of your design will likely be different when using injection molded components rather than 3D printed parts. Structural rigidity, mechanism friction, and dimensional accuracy are often far different between molded and 3D printed parts. When conducting EVT, you really want to understand the actual performance of the product that you intend to release and commercialize.

Additionally, 3D printed parts tend to allow engineers to "cheat" on Design For Manufacturability and Assembly (DFMA). 3D printing allows for undercuts, zero draft, bosses and bores from multiple angles. So, most of the time, designs that are created with 3D printing as the fabrication method aren't up-to-snuff from when considering longer-term traditional manufacturing materials and processes. When these designs need to mature to meet reliability, aesthetic, and manufacturability requirements, there's often significant overhauling that needs to occur. And you don't want to do this type of overhauling post-design freeze.

The same is true of electronics. Electronic designs utilizing Arduino Printed Circuit Boards (or similar) are generally not ready for EVT. Their costs will be prohibitive when increasing volumes, and custom PCBs will eventually need to be created. This typically requires a redo on electrical safety testing and much of the hardware verification process.

TIP #3
Seek out failures (not wins).

Sometimes, teams that rush into V&V hold their breath and hope for the best. Unfortunately, that typically doesn't pan out well.

This may seem obvious by now, but EVT is the opportunity to fail. In fact, that's what you want to find -- the failures. EVT is to beat-up your prototypes, test them in worst case conditions, understand their thresholds of performance. When you drive EVT this way, you will have much greater visibility into the design and manufacturing issues that need to be addressed before getting into the V&V process. V&V should be a walk in the park compared to the abusive process of EVT.

So, before you jump straight into Verification & Validation, take the time to make sure you're ready. Run the EVT process, identify the failures, implement the design changes, and retest until you achieve the desired outcomes.

When you successfully get through EVT, you'll speed through V&V with confidence like the savvy travelers in the TSA Precheck line.