6 ms·

For starters, false positives are an inherent part of medical diagnosis. Interpretation and diagnosis is probabilistic based on ROC curves balancing different s

For starters, false positives are an inherent part of medical diagnosis. Interpretation and diagnosis is probabilistic based on ROC curves balancing different sensitivities and specificities balanced with disease prevalence as well as the significance of missed/delayed diagnosis. I totally get this and it's technical terminology. I think my takeaway from trying to explain why this isn't a big deal on this thread is that calling a tomo or mammo "positive" is just a recipe for confusion. I understand why it's done this way historically, but the idea of a mammo or tomo being "positive for cancer" makes no sense since there will almost always be either an ultrasound a biopsy or something else before a diagnosis is made. It's a test for whether a more invasive/expensive test is worth it, not a test for cancer. Curious what your opinion is on AI for DBT? I think there is great long term potential here, even more so than the potential that we have seen so far in mammo (such as https://www.nature.com/articles/s41586-019-1799-6 https://www.nature.com/articles/s41586-019-1799-6), because it's easier for AI to thoroughly look at things in 3d and spot new patterns that are not obvious to human eyes.

haldujai 4y ago
AI in medical imaging has been very limited and ineffective thus far. My institution pays for an expensive lesion detection / mammography CAD software package but it generates so many false positives I don’t use it in my practice. Overall having to review an extra n pseudolesions per scan results in decision fatigue and increases my miss rate, at least in my experience. There are several issues with AI when it comes to radiology that I can’t address with a single comment but given your question my biggest criticism would be that we don’t have a sensitivity problem in medicine. What we have is largely a specificity problem, but that’s inherent with imaging which is not a ground truth representation of a physical entity but rather how this tissue attenuates photons (for mammo). AI, like self driving cars, has to be an order of magnitude “safer” which generally translates into decreased specificity. What I mean by this is that I don’t miss a significant number of breast cancers, in fact it’s very very low. So an AI to improve my detection is pointless to me, what I really want is to recall less patients but AI won’t necessarily help with that because it’s inherent limitations of the imaging modality (scar and cancer look the same). I think you are in the right track that AI looking at information humans are not currently looking at is the future / next step. Not to replace a task already handled by physicians who have the ability to integrate disparate health records (I.e. does this patient have easier access to breast MRI or biopsy given location/insurance/biopsy schedule/MRI schedule/hospital resources and what are her personal goals of care to help me decide what to do with this lesion) that is currently silo’s and inaccessible to AI.
- dontreact 4y ago
  AI in medical imaging has been very limited and ineffective thus far. A) I think it's too early to make a call here. The first big paper for AI in medical imaging with modern methods came out in 2016, and we are just now getting large scale prospective studies where algorithms outperform specialists: https://www.thelancet.com/journals/landig/article/PIIS2589-7500(22)00017-6/fulltext https://www.thelancet.com/journals/landig/article/PIIS2589-7... My institution pays for an expensive lesion detection / mammography CAD software package but it generates so many false positives I don’t use it in my practice. Overall having to review an extra n pseudolesions per scan results in decision fatigue and increases my miss rate, at least in my experience. B) Totally agree that old CAD has not been effective, and this has created a huge trust gap for the new generation of technology. The new generation of technology using deep learning is also easier for anyone to create so it will take some time to separate the good from the bad. There are several issues with AI when it comes to radiology that I can’t address with a single comment but given your question my biggest criticism would be that we don’t have a sensitivity problem in medicine. What we have is largely a specificity problem, but that’s inherent with imaging which is not a ground truth representation of a physical entity but rather how this tissue attenuates photons (for mammo). C) This claim seems overly broad. If we can improve sensitivity it can certainly be a good thing. That being said, improving specificity in an assisted read scenario seems almost impossible in my experience, because if a doctor is convinced something is cancer, no output from an algorithm is going to get them to downgrade. We did an analysis on exactly what was going on when our algorithm would give a lower, non-recall score vs. a panel of radiologists, and we found that the most common situation was missing that a lesion was a scar. https://www.nature.com/articles/s41591-019-0447-x https://www.nature.com/articles/s41591-019-0447-x It turns out that radiologists will in general not look at a lesion from all different possible reconstructions in 3D, because it's too time consuming. But sometimes there is only one reconstruction/view where the scarring is obvious. AI obviously does not have this issue, so in principle it is certainly capable of improving on specificity and this has been seen in many peer reviewed retrospective studies. As I mentioned, the issue is how to make this happen in clinical practice where assisted read is the only option that will be tolerated for deployment for at least the next 5-10 years. In that time period, I think the only thing we can really hope for is a boost to sensitivity without impacting specificity. I believe eventually enough strong evidence (prospective observational multi site studies) will accumulate, and the algorithms will improve enough that for something like mammography, at least a portion of the reads can be done without a radiologist. This is probably the only way we will see improvements to specificity AND sensitivity in my opinion.

haldujai 4y ago

AI in medical imaging has been very limited and ineffective thus far. My institution pays for an expensive lesion detection / mammography CAD software package but it generates so many false positives I don’t use it in my practice. Overall having to review an extra n pseudolesions per scan results in decision fatigue and increases my miss rate, at least in my experience. There are several issues with AI when it comes to radiology that I can’t address with a single comment but given your question my biggest criticism would be that we don’t have a sensitivity problem in medicine. What we have is largely a specificity problem, but that’s inherent with imaging which is not a ground truth representation of a physical entity but rather how this tissue attenuates photons (for mammo). AI, like self driving cars, has to be an order of magnitude “safer” which generally translates into decreased specificity. What I mean by this is that I don’t miss a significant number of breast cancers, in fact it’s very very low. So an AI to improve my detection is pointless to me, what I really want is to recall less patients but AI won’t necessarily help with that because it’s inherent limitations of the imaging modality (scar and cancer look the same). I think you are in the right track that AI looking at information humans are not currently looking at is the future / next step. Not to replace a task already handled by physicians who have the ability to integrate disparate health records (I.e. does this patient have easier access to breast MRI or biopsy given location/insurance/biopsy schedule/MRI schedule/hospital resources and what are her personal goals of care to help me decide what to do with this lesion) that is currently silo’s and inaccessible to AI.

dontreact 4y ago
AI in medical imaging has been very limited and ineffective thus far. A) I think it's too early to make a call here. The first big paper for AI in medical imaging with modern methods came out in 2016, and we are just now getting large scale prospective studies where algorithms outperform specialists: https://www.thelancet.com/journals/landig/article/PIIS2589-7500(22)00017-6/fulltext https://www.thelancet.com/journals/landig/article/PIIS2589-7... My institution pays for an expensive lesion detection / mammography CAD software package but it generates so many false positives I don’t use it in my practice. Overall having to review an extra n pseudolesions per scan results in decision fatigue and increases my miss rate, at least in my experience. B) Totally agree that old CAD has not been effective, and this has created a huge trust gap for the new generation of technology. The new generation of technology using deep learning is also easier for anyone to create so it will take some time to separate the good from the bad. There are several issues with AI when it comes to radiology that I can’t address with a single comment but given your question my biggest criticism would be that we don’t have a sensitivity problem in medicine. What we have is largely a specificity problem, but that’s inherent with imaging which is not a ground truth representation of a physical entity but rather how this tissue attenuates photons (for mammo). C) This claim seems overly broad. If we can improve sensitivity it can certainly be a good thing. That being said, improving specificity in an assisted read scenario seems almost impossible in my experience, because if a doctor is convinced something is cancer, no output from an algorithm is going to get them to downgrade. We did an analysis on exactly what was going on when our algorithm would give a lower, non-recall score vs. a panel of radiologists, and we found that the most common situation was missing that a lesion was a scar. https://www.nature.com/articles/s41591-019-0447-x https://www.nature.com/articles/s41591-019-0447-x It turns out that radiologists will in general not look at a lesion from all different possible reconstructions in 3D, because it's too time consuming. But sometimes there is only one reconstruction/view where the scarring is obvious. AI obviously does not have this issue, so in principle it is certainly capable of improving on specificity and this has been seen in many peer reviewed retrospective studies. As I mentioned, the issue is how to make this happen in clinical practice where assisted read is the only option that will be tolerated for deployment for at least the next 5-10 years. In that time period, I think the only thing we can really hope for is a boost to sensitivity without impacting specificity. I believe eventually enough strong evidence (prospective observational multi site studies) will accumulate, and the algorithms will improve enough that for something like mammography, at least a portion of the reads can be done without a radiologist. This is probably the only way we will see improvements to specificity AND sensitivity in my opinion.