The journey to verification: four stages of device testing

1. Proof of concept testing

The purpose of early testing is to gauge which concepts are promising and should be developed further. When you embark on a medical device development you may have several ideas which need evaluating. They will likely be made up of rough, unrefined embodiments or prototypes where the output of testing could be simple and binary: either they ‘work’ or they don’t.

Concepts in this phase could be ‘looks-like’ and ‘works-like’ prototypes, or even demonstrators that may not resemble an integrated design but can prove it is feasible. Your concept’s development approach will depend on what you are trying to prove and the type of device you are aiming to produce.

At this early stage, documentation should be light. With the need for change and development agility, it is unlikely that you will want to write formal protocols. However, you will still need to document your test approach and the results for future reference and basic traceability. Depending on your project, testing could be qualitative rather than quantitative. If possible, you should consider whether the tests are performed by more than one individual. It is natural to have a preference or bias towards your own favoured concept, so getting someone independent involved in testing will bring valuable objectivity.

Top tips:

• Get an independent individual to test your concept from early on. It can be helpful to have several colleagues as your go-to people to sanity check and challenge your testing. If there are flaws in your concept, it is best to find them early on. Having other people testing your concept will increase the likelihood of discovering issues at this stage.
• Take as many photographs and videos of the testing as possible. In the absence of detailed methods, images can help set the scene and clarify what you’ve done, enabling you to review and share with others later.

Documents to be completed:

• Technical note including the test approach and summary of findings
• Image and video library of testing
• User feedback, if appropriate for the concept

2. Proof of principle testing

Once you have proven through the proof of concept stage that the fundamentals of your product are sound, you can now test your device’s performance against some early specifications. An example might be looking whether your device can deliver a formulation to observe how much and how fast this happens.

As you’re starting to think more about the detailed functions of the device, while still firmly in development, proof of principle testing involves a balance of qualitative and quantitative tests, with increased rigour in the test approaches. Even though you’re still in the development phases, you should be aware of the formalities to come, including design verification. This means that at this stage it’s important to get the right balance of agility and detail to keep your project on track while de-risking to avoid any unwanted surprises later.

Some questions you might want to ask yourself are:

• Can I rely on my test data?
  For example, if you are using a sensor to capture data, you need to be sure it’s accurate and precise. Using a calibrated sensor will give you some confidence that recorded data are accurate. Similarly, make sure you understand the influence custom test jigs and fixtures have on the measurements.
• Is the user influencing the performance?
  How much of what you are seeing is down to user technique or influence? If you are trying to assess the performance of a device away from user influences, consider removing the user as a variable e.g., automated, or semi-automated testing.
• How many samples should I be testing?
  If you are making decisions which influence the design or are analysing data against specification limits, even at this early stage, these shouldn’t be based on the performance of a single sample. Deciding on the number of samples that need to be tested at this stage needs to be pragmatic, and should be based on several factors, including: the availability of parts, how representative the parts are, or whether a well-controlled manufacturing process produces the parts.
• Could someone else repeat what I have done?
  A documented approach should be generated for this testing. Ideally, another competent operator should be able to repeat the test in the same way, and anyone looking at your work later should be able to understand what you did.
• Is what I’ve done traceable?
  Which sample did you test? Where did it come from (e.g., manufacturing process, materials of construction)? Where is it now? What was the design revision? The traceability of your parts and design is key to enabling you to investigate any unusual or interesting results.

At this point in the development, your testing and methods can be seen as an iterative step towards design verification. If done correctly, they can be used as the building blocks for later formal test methods and protocols.

During this phase you should aim to produce data and evidence to show that, in principle, based on the current embodiment of the design, your device is likely to meet its functional requirements. Achieving this will help de-risk your device’s progression to the next phase of development.

Top tips:

• Get input on your test approach from colleagues who are likely to be involved in your verification test program. Anything which can be done now and fed into later testing will make the development smoother. Seeking independent input can also be helpful to check that what you are planning to do has the right level of control and detail for this phase of the development.
• Anything which can be done to speed up testing, without compromising its quality, can also be useful. If a simple rig can be built which can speed up testing significantly, it is worth investing the time now, particularly if the rig may be useful in future testing phases.
• You might be establishing your device’s functional requirements. When doing this, having test approaches in mind is a good way of sanity checking that your key requirements are written to be clear, unambiguous, and measurable / ‘testable’.
• Preserve the independence between the design and testing teams. It’s even more important at this stage that someone independent is also assessing the device design.

Documents to be completed:

• (Draft) Product / Functional / Technical Requirements Specification
• Test methods or documented method statements
• Test report(s)
• Requirements for any rigs or fixtures

3. Pre-design verification testing

Now that you are further on in your development, design verification, and any testing work within that (DVT), is getting closer. You need to be thinking about your device’s performance in a formal way, particularly when more substantial data are collected, considering whether the device is likely to meet its functional requirements.

Design verification can be a long and expensive process which will determine your device’s success or failure. If any issues are found during formal design verification, putting them right can be costly. A good way to assess your device’s readiness for DV is to run a pre-design verification test program. Running pre-DVT is not mandatory, but it is advisable.

This is because pre-DVT:

• Gets you in the mindset of the coming formalities, including statistical sampling and analysis methods which will be required in DVT. Up until now you may have been focused on key device performance attributes during the ‘expected’ or even idealised use of a relatively small number of device samples. There may also be other requirements you have not tested until now, including robustness to environmental stresses. Does your device need to work after freefall or vibration tests? If developing a needle injection system or inhaler, the answer to that question is likely ‘yes’. Looking at all the device’s requirements may initiate some testing which has not yet been thought about
• Is one of the last opportunities to find and address issues with the device design – effectively one of the last, and most meaningful design de-risk exercises. Not everything is right the first time and what worked for prototypes may not be quite right for devices manufactured in volume. This may be the time when unforeseen issues pop up. The separation of pre-DV from the formalities of DV mean that you can still be reactive and address these issues. The impact of skipping this exercise could be significant; for example, if you have invested in pilot tooling, produced thousands of devices and started a formal DV test program, any issue with the test methods or the devices themselves – however minor – could prove to be very costly
• Can make accommodations for non-conforming samples. If you tested a device and it didn’t meet the test acceptance criteria, you can investigate and fix the issue, then re-do the test. This flexibility would simply not be present in DV
• Can help you make sure your test methods, fixtures and equipment are robust and reliable enough to be ready for test method validation. It provides an opportunity to demonstrate that someone new to the project or who has not been involved in the technical development can understand and execute them. This process can also familiarise the testing team with the test methods, paving the way for a smoother entry to DVT
• Can be a key project decision point, helping stakeholders feel more confident that the project should move forwards. More test data and experience of assessing your device can increase your confidence that the design and its implementation are of a high standard

Top tips:

• Ideally, pre-DVT should involve a product as close to the final embodiment as possible. However, natural development constraints may mean that this is not always possible. In these cases, it could still be possible to gather useful information from pre-DVT using devices made by alternative manufacturing processes. However, always be aware of the caveats that will need to be applied in these scenarios. Ask yourself: how confident am I in key functional test results from less representative samples?
• Imagine your test documentation will be audited, considering this from a more formal mindset. Would an auditor be happy with what has been written? Are they clear, unambiguous and easy to follow? Are they descriptive enough to be executed in the same way regardless of the operator? Organise peer reviews to ensure this is the case. Getting the quality of your documentation right means a smoother transition to DV and ensures mistakes are not carried through and picked up later when their impact may be greater.
• Ensure all measurement equipment have valid calibration certificates. If appropriate, complete IQ (Installation Qualification) and OQ (Operational Qualification) on equipment, jigs and fixtures to give additional confidence in their suitability.
• Have approved templates to-hand before writing test method and protocol documents. This helps build consistency between documents. It is much easier to amend and refine a single template than edit several methods individually.
• Think about and challenge yourself on data integrity. Make sure all recorded data meet regulatory expectations, captured in the ALCOA+ principles. The output data from your test approaches intended for DVT should be Attributable, Legible, Contemporaneous, Original, Accurate, Complete, Consistent, Enduring, and Available.
• Pre-DV doesn’t have to be done only once. It could be repeated for several reasons, including if test method or device issues are raised and addressed, or if more representative device samples become available. Iterating pre-DV alongside issue resolution and manufacturing evolution strengthens its value as a de-risk exercise.

Documents to be completed:

• Product / Technical / Functional Requirements Specification
• Pre-DV Test protocol, including a link to the product requirements, and test acceptance criteria
• Test methods (ideally matured to the point they are ready for validation)
• Pre-DV Test report(s)
• Requirements for any rigs and fixtures
• Original data records

4. Design verification

Now is the time to verify that you have made the device correctly and that your design outputs meet your design inputs. Design verification is a formal culmination of all the technical work undertaken as part of the project – the ‘icing on the cake’. The results will be submitted as part of your Technical File or Design History File and will influence whether your medical device makes it to market.

The bigger picture of design verification covers not just key functions and features, but also includes the assessment of your drug formulation, extractables and leachables, shelf-life studies, and biocompatibility. Some of this may be conducted by notified bodies and specialist test houses. Design verification is a large body of work of which testing may only be one part, so the testing aspects may be covered in a discrete DVT programme.

DVT will, certainly for most drug delivery devices, require the testing of many devices over several days, weeks, or months. Finding any issues with your device at this phase can lead to high-impact delays to the programme. If you have been testing the key aspects of your device’s design and function throughout its development, the risk of finding any issues during DVT will be significantly reduced.

Top tips:

• Before starting DVT, test methods must be validated. This means they must demonstrably output data which is repeatable and reproducible. The acceptance criteria for, and outcome of, test method validation must be documented. Test method validation may be performed alongside pre-DVT.
• Ensure all your DVT operators are trained in the methods and protocols they are expected to work with. Take the time to ensure operators have really understood the instructions and equipment, including demonstrating their competence where possible. Pre-DVT is another opportunity to ensure rigorous training.
• Ensure that all tests have clear acceptance criteria assigned. This can be documented in your test method and protocol. Is it clear to the reader what a test pass and fail result looks like? Set the goalposts clearly, including distinguishing between intra-device (repeatability) and inter-device (reproducibility) outcomes. Make sure to clearly describe any required statistical analyses.
• Make sure there is clear traceability in your DV reports, from the results reported, back to the source test data and methods. This will make it easier for a reviewer or auditor to follow the process from start to finish. A good report strikes a balance between being comprehensive and succinct. Ensure that just the right amount of detail is available to build a clear narrative and ensure the reader can understand the work without having to refer to too many other documents.
• Remember who you’re writing the report for; it is probably a regulator, so make sure you’re providing them with what they need. Check the latest regulatory expectations for technical reports. Your device may have performed brilliantly, but poor reporting, data integrity, or traceability may introduce unnecessary scrutiny and delays.

Documents to be completed:

• DV(T) Plan (if appropriate)
• DV(T) Protocol
• Test methods, validated
• Design verification (Test) reports, and a summary report if appropriate
• Original data records

Conclusion

Testing is an essential tool in any product’s development. It not only helps you make your device better, but also proves that it works as it should. Testing to prove your device’s features and functions throughout development will make achieving a successful product easier.

Of course, test approaches need to evolve alongside the device development, so the trick is to use a test approach with the right balance of agility and rigour for the stage you are at. This ensures your testing is a powerful de-risking tool, while not over-burdening the development phases. Looking ahead to what will be expected for design verification will help you dial up testing formalities at the right time, making the transition between phases and bodies of test work easier.

Testing should present an objective, independent challenge, so make sure to augment work done by the development team with independent operators. This ensures the design embodiment is not ‘rose tinted’ and checks that the device is suitable for a wider user group, not just the people who are invested in its development. Spotting performance issues and user challenges early on will mean less time and cost to resolve them.

If testing is carefully planned from the very beginning of your project, then design verification at the end will simply be a confirmation of what you already know, making for a smoother transition to market.

Team Consulting has been developing medical devices for over 35 years, from concept through to industrialisation. Our dedicated device testing team engages at all stages of your device development, bringing expertise and objectivity to ensure our test solutions have the right mix of established best practice, bespoke solutions, regulatory compliance, data, and documentation integrity. All these ingredients are essential for the successful completion of design verification.

If you would like to discuss your next device testing challenge, get in touch.