6 ms·

I've worked on medical devices. As a techie, what surprised me more is not the lack of trials for new devices. It's that for software (and other kinds of compon

I've worked on medical devices. As a techie, what surprised me more is not the lack of trials for new devices. It's that for software (and other kinds of components), it's often acceptable to have black box testing to certify a component as functional. This means basically saying "I pressed a button and the thing did X". That's the test. No need to understand how that works, or provide any more technical documentation or specs, or record system state or variables, or do anything else. Press button, it does X. That's good enough to be certified. The reason for this is simple enough: sometimes the thing you're using is proprietary and its manufacturer/vendor simply won't give you anything else to certify it. But this black box testing is even used for open source software and components. It's like a short cut you can use to cut through a lot of the testing you could have done. I imagine this will remain standard practice as the "AI" companies push their hallucinating dreck and unexplainable magic into the medical device space.

neandrake 3y ago
I work with software medical devices. Another aspect here aside from 510k is that it is required, or effectively required, to comply with something like IEC-62304 (developing software as a medical device), and ISO-13485 (quality management systems for medical devices). For the scenario you describe the piece that’s missing is risk analysis, a requirement. In preparation to release to market they must evaluate the probability and severity of the button not doing X or doing X incorrectly, and develop mitigations if the risk is unacceptable. What you ask - documentation and specs - exist at some level, but the manufacturer has to define what level is necessary for them. I could see an argument against the manufacturer deciding this for themselves, though it’s likely impractical to do so. For software medical devices that have hundreds of transitive dependencies it’s not feasible to go at the level you’re describing. Some management of dependencies is necessary but treating as a black box - with quality/test management and risk analysis of the black box - is what the current system defines as a reasonable trade-off. Again I could see arguments for changing this, though for many manufacturers the EU has instituted stricter regulatory in the past ~5 years which has been a bit painful but overall probably a good thing. Today one of the aspects of medical device development which is under tighter scrutiny is cybersecurity. It’s pretty painful right now. Previously there was not much related to cybersecurity required - obviously not ideal - but the pendulum has swung to the other end of the spectrum making it a significant burden. We’ll see, most of it is adopting new processes which is always painful and slows down progress at first. After the initial hump it should be eased into, and ultimately better for patient care and medical institutions in the long run.
- iancmceachern 3y ago
  Agree with you here. Wanted to echo " It's (cybersecurity) pretty painful right now". The FDA just implemented new requirements. Basically they require penetration testing on all new medical devices. The issue is they don't have the expertise in house to know the technical details, and they haven't defined the tests, etc. Additionally they're isn't yet an ecosystem of partners and service providers yet to provide and compete in providing those penetrstion testing services. Pragmatically what it means for folks trying to get a device cleared at the current moment: You need to send your device to the one and only penetration testing house that does this for the FDA now and let them try to physically hack into your medical device. You have to make it impossible and evident if someone tampers with it in any way. This is in addition to all the software security stuff we need. Imagine of you were making computer monitors. One day you are suddenly required to make it so a technical expert cannot open the monitor up using specialized tools.
  - seehafer 3y ago
    From a submission standpoint, as I write this, FDA seemingly cares more about cybersecurity than your medical device actually demonstrating safety and efficacy within intended use. The time that review teams have to review any given device has stayed the same, but fear-driven, heavy-handed cybersecurity regulations (which must be followed) have been added to the mix.
  - MichaelZuo 3y ago
    I'm more surprised that medical devices were not previously required to be tamper evident.
    - iancmceachern 3y ago
      They are, and it's generally good practice. The difference is now they spend much more time and effort on the penetration testing. An example. In the past it would be OK to use security bit screws for this. Yes you can buy the bits online, but it was at least one layer of perceived security. This doesn't fly anymore. The real challenge is they implemented these new requirements on devices that were already in the submission process. Also, these things aren't written down anywhere in standards etc. so you know them ahead of time when you design. You have to just wait until the penetration testing and find out. Ultimately the new rules aren't the challenge, it's the fact that you don't get to know them when you start and finish the design, you find out later.
jtwaleson 3y ago
It depends on the risk class according to IEC 62304. When you're developing Class C software you'll need acceptance criteria that goes into how you initialize variables etc.

neandrake 3y ago

I work with software medical devices. Another aspect here aside from 510k is that it is required, or effectively required, to comply with something like IEC-62304 (developing software as a medical device), and ISO-13485 (quality management systems for medical devices). For the scenario you describe the piece that’s missing is risk analysis, a requirement. In preparation to release to market they must evaluate the probability and severity of the button not doing X or doing X incorrectly, and develop mitigations if the risk is unacceptable. What you ask - documentation and specs - exist at some level, but the manufacturer has to define what level is necessary for them. I could see an argument against the manufacturer deciding this for themselves, though it’s likely impractical to do so. For software medical devices that have hundreds of transitive dependencies it’s not feasible to go at the level you’re describing. Some management of dependencies is necessary but treating as a black box - with quality/test management and risk analysis of the black box - is what the current system defines as a reasonable trade-off. Again I could see arguments for changing this, though for many manufacturers the EU has instituted stricter regulatory in the past ~5 years which has been a bit painful but overall probably a good thing. Today one of the aspects of medical device development which is under tighter scrutiny is cybersecurity. It’s pretty painful right now. Previously there was not much related to cybersecurity required - obviously not ideal - but the pendulum has swung to the other end of the spectrum making it a significant burden. We’ll see, most of it is adopting new processes which is always painful and slows down progress at first. After the initial hump it should be eased into, and ultimately better for patient care and medical institutions in the long run.

iancmceachern 3y ago
Agree with you here. Wanted to echo " It's (cybersecurity) pretty painful right now". The FDA just implemented new requirements. Basically they require penetration testing on all new medical devices. The issue is they don't have the expertise in house to know the technical details, and they haven't defined the tests, etc. Additionally they're isn't yet an ecosystem of partners and service providers yet to provide and compete in providing those penetrstion testing services. Pragmatically what it means for folks trying to get a device cleared at the current moment: You need to send your device to the one and only penetration testing house that does this for the FDA now and let them try to physically hack into your medical device. You have to make it impossible and evident if someone tampers with it in any way. This is in addition to all the software security stuff we need. Imagine of you were making computer monitors. One day you are suddenly required to make it so a technical expert cannot open the monitor up using specialized tools.
- seehafer 3y ago
  From a submission standpoint, as I write this, FDA seemingly cares more about cybersecurity than your medical device actually demonstrating safety and efficacy within intended use. The time that review teams have to review any given device has stayed the same, but fear-driven, heavy-handed cybersecurity regulations (which must be followed) have been added to the mix.
- MichaelZuo 3y ago
  I'm more surprised that medical devices were not previously required to be tamper evident.
  - iancmceachern 3y ago
    They are, and it's generally good practice. The difference is now they spend much more time and effort on the penetration testing. An example. In the past it would be OK to use security bit screws for this. Yes you can buy the bits online, but it was at least one layer of perceived security. This doesn't fly anymore. The real challenge is they implemented these new requirements on devices that were already in the submission process. Also, these things aren't written down anywhere in standards etc. so you know them ahead of time when you design. You have to just wait until the penetration testing and find out. Ultimately the new rules aren't the challenge, it's the fact that you don't get to know them when you start and finish the design, you find out later.

jtwaleson 3y ago

It depends on the risk class according to IEC 62304. When you're developing Class C software you'll need acceptance criteria that goes into how you initialize variables etc.