6 ms·

There are several key points that get left out in AI radiology conversations such as this one: 1) Mammograms are not interpreted in a vacuum. In fact mammogram

There are several key points that get left out in AI radiology conversations such as this one: 1) Mammograms are not interpreted in a vacuum. In fact mammograms are usually the first in a long line of tests before a breast cancer or other diagnosis is ultimately made. In fact, it's probably more accurate to refer to mammography as a screening exam for which patients need a biopsy rather than a diagnostic test for cancer (there are rare exceptions, but overall this point holds). 2) Speaking frankly as a radiologist myself, tests like mammograms aren't even that good in terms of overall diagnosis. Thats why ultrasound, tomosynthesis and MRI are often used as supporting evidence and/or alternative exams. 3) There is controversy over the overall utility of mammograms, particularly in the screening context. Radiologists more than anyone would like the sensitivity and specificity of these studies to be higher. It strikes me that the people that push these "radiology is ripe for disruption" or "AI outperforms radiologists" hyperbolic arguments are clearly people that have never seen the inside of a clinic. I'm sure they love this rhetoric though when pitching to VCs or sitting around the conference table coming up with 'breakthrough ideas' to turn into power-points for the other administrators.

ggm 7y ago
I wish there was a way to repost this to all most every medical breakthrough ml story.
cameronfraser 7y ago
It could be useful as a tool to help a radiologist do their job better though. I think many of these techniques described in ML papers will be used to enable people to be better at their jobs rather than replace them. At least until there is AGI at least.
- neuro_image3 7y ago
  I wouldn't dispute this (if they finally put something together that isn't horrendously cumbersome, time-consuming and hard to use) but this doesn't justify the 'AI is about to replace radiology' crap I seam to see every time some academic group publishes an AI/ML paper.
  - cameronfraser 7y ago
    I don't think this paper makes that claim, you might be thinking of media interpretations of papers which is usually the one making bogus claims. It says that it outperforms 5 out of 5 people, but that is in this specific context. They aren't necessarily claims meant to be generalized that much.
    - neuro_image3 7y ago
      I see that type of hyperbolic claim several times in the comments. I wasn't just commenting on the abstract presented. I was commenting on the comments I see here, as well as comments I see related to similar papers all the time. I also interact with AI/ML researchers all the time. Most of them are typically some combination of: 1. Poorly informed about the appropriate context and utility of medical imaging. 2. Trying as hard as they can to push AI/ML as the most important technology in medicine today. 3. Pursuing a very task-specific project which they claim is massively generalizable in some (incorrect) way.
      - StreamBright 7y ago
        Usually they do not understand that your workflow if not revolving around pattern recognition on pictures. The best use of ML in pattern recognition for radiology is ordering images. You would get the the images sorted by the likeliness of having something unusual.
- StreamBright 7y ago
  What do you think how far AGI is?
  - cameronfraser 7y ago
    a long long time
    - StreamBright 7y ago
      I guess over 100 years at the very least.
ekianjo 7y ago
If I remember correctly for some women a mammograph can be completely unreadable while much clearer for other's. That seems to make it a very poor method for diagnosis, despite it being recommended almost everywhere.
StreamBright 7y ago
The only thing I see as potential area of improvement (i do not like the word disruption) in radiology is workflow optimization to reduce the time you spend on administrative tasks. I would be interested to hear your opinion on this.
- neuro_image3 7y ago
  Absolutely, that's an area where various software innovations could be extremely helpful. In fact, the area of medicine most in need of 'disruption' imho is healthcare enterprise software. Doctors are literally killing themselves because the interfaces they have to deal with on a daily basis are so appallingly poor. Of course, the solution is less technological than political as it wouldn't take much to come up with better software than the legacy alternatives but you'd have a very hard time getting past the entrenched relationship interests of crony bureaucrats that run hospital administrations.
1e-9 7y ago
Your comments do not seem to be addressing this particular study, but rather seem to be directed at the plethora of poorly-designed/over-hyped ML papers that are published on a regular basis. While that is an issue, this particular paper made no claims about disruption and it had nothing to do with diagnostic performance. It was a screening study that appears to make sound comparisons to the performance of five reasonably-qualified radiologists on screening images.
- wdevanny 7y ago
  I think their comment is a very needed bucket of ice water on the multitude of other comments in this thread that are making claims about disruption and diagnostic performance.
  - 1e-9 7y ago
    Can you point to a particular comment in this thread making a claim about disruption or diagnostic performance?
    - neuro_image3 7y ago
      The next main root comment. 'Of the major specialties, it seams that radiology is the most in danger of disruption'. I wonder how long people think about these sorts of claims before posting them. Do they really think an AI is more likely to appropriately interpret an MRI scan (and all the anatomic, physics and pathophysiological data therein contained) in the context of a specific clinical work-up more easily than triage patients the way a family practitioner or ER doctor does?
      - jeffreygoesto 7y ago
        AI is not "interpret"ing. It is merely using a fitted distribution. Just because there are many degrees of freedom fitted does not make it magic.
      - TeMPOraL 7y ago
jlg23 7y ago
> 1) Mammograms are not interpreted in a vacuum. In fact mammograms are usually the first in a long line of tests before a breast cancer or other diagnosis is ultimately made. The paper specifically talks about mammography, it does not claim to replace a complete diagnosis. > 2) Speaking frankly as a radiologist myself, tests like mammograms aren't even that good in terms of overall diagnosis. Thats why ultrasound, tomosynthesis and MRI are often used as supporting evidence and/or alternative exams. From the abstract: "2) successfully extends to digital breast tomosynthesis" > 3) There is controversy over the overall utility of mammograms, particularly in the screening context. > It strikes me that the people that push these "radiology is ripe for disruption" [...] The paper, which I just skimmed over, does not read hyperbolic, for that we'll have to wait for popsci journalists. OTOH, if one leaves the 1st world context, any type of successful diagnosis automation in medicine is a blessing for areas where you simply don't have enough trained medical staff.
- neuro_image3 7y ago
  I wasn't just commenting on the abstract presented. I was commenting on the comments I see here, as well as comments I see related to similar papers all the time. I also interact with AI/ML researchers all the time. Most of them are typically some combination of: 1. Poorly informed about the appropriate context and utility of medical imaging. 2. Trying as hard as they can to push AI/ML as the most important technology in medicine today. 3. Pursuing a very task-specific project which they claim is massively generalizable in some (incorrect) way.
  - kilburn 7y ago
    There's one point that often comes up when I chat with my MD friends: All of them agree that more information is not strictly better while diagnosing. In fact, most support that unneeded information is actually worse because it confounds the issue. My engineering mind just can't come up to terms with this. Why wouldn't you collect all information you possibly can? You can always ignore irrelevant data you have, but you cannot consider data that you don't have! The closest I've been to rationalizing this: diagnosing is a stochastic process so complex (and with the search space so large) that the random noise in extra data is likely to point you towards wrong directions. Plus you can always collect more data afterwards if your initial diagnosis turns out to be wrong. This is of course very simplified, but it makes sense. However, I just can't turn off my inner voice from screaming "more data is always better". I guess that's why I'm not an MD :)
llsf 7y ago
Don't you think though that it would help you? I see this ML treatment of the images as a resolution improvement. It would help doctors to see better, maybe to not miss the detail that could be missed.
1_over_n 7y ago
We are working in the breast cancer space now looking at breast cancer and ultrasound (not just from a screening / diagnostic perspective - also treatment planning for medical oncologists and treatment response planning). We don't use deep learning - we use Biophysical models. We hate using the term "AI". This is a very challenging discipline to explain to VCs. Also, speaking to point 2 here - the "value" of building tools for ultrasound is often dismissed by VCs because "ultrasound isn't used for screening or diagnosis". This is an insane perspective from our position when we are practically based within hospitals, collaborating strongly with radiologists and medical oncologists who work with ultrasound on a daily basis. We are very embedded within the hospital and look to understand the clinicians workflow and decision making processes first, as well as understanding what's possible given the hurdles involved in data access (which can still be tricky even when you are through IRBs and ethics). We have found that telling VCs the reality about working with hospitals and doctors can often limit their excitement about your company prospects. Our success to date has largely been as a result of doctors and hospitals who believe in us, see the value in what we are doing. They have put time and effort into collaborating because they are impressed with what we have been able to do results wise by bootstrapping as a small team, rather than as a VC funded shiny startup. In a weird way i would say that at it's best times medtech can be one of the "purest" industries to work in. By this i mean ultimately your technology works or it doesnt (at least from the medical communities perspective - again VCs are a different story). There are obviously exceptions to this (Theranos anyone) and there are issues around the 510K process but on the whole there is a big price to pay for making unsubstantiated claims (say compared to aspirational lifestyle marketing).

ggm 7y ago

I wish there was a way to repost this to all most every medical breakthrough ml story.

cameronfraser 7y ago

It could be useful as a tool to help a radiologist do their job better though. I think many of these techniques described in ML papers will be used to enable people to be better at their jobs rather than replace them. At least until there is AGI at least.

neuro_image3 7y ago
I wouldn't dispute this (if they finally put something together that isn't horrendously cumbersome, time-consuming and hard to use) but this doesn't justify the 'AI is about to replace radiology' crap I seam to see every time some academic group publishes an AI/ML paper.
- cameronfraser 7y ago
  I don't think this paper makes that claim, you might be thinking of media interpretations of papers which is usually the one making bogus claims. It says that it outperforms 5 out of 5 people, but that is in this specific context. They aren't necessarily claims meant to be generalized that much.
  - neuro_image3 7y ago
    I see that type of hyperbolic claim several times in the comments. I wasn't just commenting on the abstract presented. I was commenting on the comments I see here, as well as comments I see related to similar papers all the time. I also interact with AI/ML researchers all the time. Most of them are typically some combination of: 1. Poorly informed about the appropriate context and utility of medical imaging. 2. Trying as hard as they can to push AI/ML as the most important technology in medicine today. 3. Pursuing a very task-specific project which they claim is massively generalizable in some (incorrect) way.
    - StreamBright 7y ago
      Usually they do not understand that your workflow if not revolving around pattern recognition on pictures. The best use of ML in pattern recognition for radiology is ordering images. You would get the the images sorted by the likeliness of having something unusual.
StreamBright 7y ago
What do you think how far AGI is?
- cameronfraser 7y ago
  a long long time
  - StreamBright 7y ago
    I guess over 100 years at the very least.

ekianjo 7y ago

If I remember correctly for some women a mammograph can be completely unreadable while much clearer for other's. That seems to make it a very poor method for diagnosis, despite it being recommended almost everywhere.

StreamBright 7y ago

The only thing I see as potential area of improvement (i do not like the word disruption) in radiology is workflow optimization to reduce the time you spend on administrative tasks. I would be interested to hear your opinion on this.

neuro_image3 7y ago
Absolutely, that's an area where various software innovations could be extremely helpful. In fact, the area of medicine most in need of 'disruption' imho is healthcare enterprise software. Doctors are literally killing themselves because the interfaces they have to deal with on a daily basis are so appallingly poor. Of course, the solution is less technological than political as it wouldn't take much to come up with better software than the legacy alternatives but you'd have a very hard time getting past the entrenched relationship interests of crony bureaucrats that run hospital administrations.

1e-9 7y ago

Your comments do not seem to be addressing this particular study, but rather seem to be directed at the plethora of poorly-designed/over-hyped ML papers that are published on a regular basis. While that is an issue, this particular paper made no claims about disruption and it had nothing to do with diagnostic performance. It was a screening study that appears to make sound comparisons to the performance of five reasonably-qualified radiologists on screening images.

wdevanny 7y ago
I think their comment is a very needed bucket of ice water on the multitude of other comments in this thread that are making claims about disruption and diagnostic performance.
- 1e-9 7y ago
  Can you point to a particular comment in this thread making a claim about disruption or diagnostic performance?
  - neuro_image3 7y ago
    The next main root comment. 'Of the major specialties, it seams that radiology is the most in danger of disruption'. I wonder how long people think about these sorts of claims before posting them. Do they really think an AI is more likely to appropriately interpret an MRI scan (and all the anatomic, physics and pathophysiological data therein contained) in the context of a specific clinical work-up more easily than triage patients the way a family practitioner or ER doctor does?
    - jeffreygoesto 7y ago
      AI is not "interpret"ing. It is merely using a fitted distribution. Just because there are many degrees of freedom fitted does not make it magic.
    - TeMPOraL 7y ago

jlg23 7y ago

> 1) Mammograms are not interpreted in a vacuum. In fact mammograms are usually the first in a long line of tests before a breast cancer or other diagnosis is ultimately made. The paper specifically talks about mammography, it does not claim to replace a complete diagnosis. > 2) Speaking frankly as a radiologist myself, tests like mammograms aren't even that good in terms of overall diagnosis. Thats why ultrasound, tomosynthesis and MRI are often used as supporting evidence and/or alternative exams. From the abstract: "2) successfully extends to digital breast tomosynthesis" > 3) There is controversy over the overall utility of mammograms, particularly in the screening context. > It strikes me that the people that push these "radiology is ripe for disruption" [...] The paper, which I just skimmed over, does not read hyperbolic, for that we'll have to wait for popsci journalists. OTOH, if one leaves the 1st world context, any type of successful diagnosis automation in medicine is a blessing for areas where you simply don't have enough trained medical staff.

neuro_image3 7y ago
I wasn't just commenting on the abstract presented. I was commenting on the comments I see here, as well as comments I see related to similar papers all the time. I also interact with AI/ML researchers all the time. Most of them are typically some combination of: 1. Poorly informed about the appropriate context and utility of medical imaging. 2. Trying as hard as they can to push AI/ML as the most important technology in medicine today. 3. Pursuing a very task-specific project which they claim is massively generalizable in some (incorrect) way.
- kilburn 7y ago
  There's one point that often comes up when I chat with my MD friends: All of them agree that more information is not strictly better while diagnosing. In fact, most support that unneeded information is actually worse because it confounds the issue. My engineering mind just can't come up to terms with this. Why wouldn't you collect all information you possibly can? You can always ignore irrelevant data you have, but you cannot consider data that you don't have! The closest I've been to rationalizing this: diagnosing is a stochastic process so complex (and with the search space so large) that the random noise in extra data is likely to point you towards wrong directions. Plus you can always collect more data afterwards if your initial diagnosis turns out to be wrong. This is of course very simplified, but it makes sense. However, I just can't turn off my inner voice from screaming "more data is always better". I guess that's why I'm not an MD :)

llsf 7y ago

Don't you think though that it would help you? I see this ML treatment of the images as a resolution improvement. It would help doctors to see better, maybe to not miss the detail that could be missed.

1_over_n 7y ago

We are working in the breast cancer space now looking at breast cancer and ultrasound (not just from a screening / diagnostic perspective - also treatment planning for medical oncologists and treatment response planning). We don't use deep learning - we use Biophysical models. We hate using the term "AI". This is a very challenging discipline to explain to VCs. Also, speaking to point 2 here - the "value" of building tools for ultrasound is often dismissed by VCs because "ultrasound isn't used for screening or diagnosis". This is an insane perspective from our position when we are practically based within hospitals, collaborating strongly with radiologists and medical oncologists who work with ultrasound on a daily basis. We are very embedded within the hospital and look to understand the clinicians workflow and decision making processes first, as well as understanding what's possible given the hurdles involved in data access (which can still be tricky even when you are through IRBs and ethics). We have found that telling VCs the reality about working with hospitals and doctors can often limit their excitement about your company prospects. Our success to date has largely been as a result of doctors and hospitals who believe in us, see the value in what we are doing. They have put time and effort into collaborating because they are impressed with what we have been able to do results wise by bootstrapping as a small team, rather than as a VC funded shiny startup. In a weird way i would say that at it's best times medtech can be one of the "purest" industries to work in. By this i mean ultimately your technology works or it doesnt (at least from the medical communities perspective - again VCs are a different story). There are obviously exceptions to this (Theranos anyone) and there are issues around the 510K process but on the whole there is a big price to pay for making unsubstantiated claims (say compared to aspirational lifestyle marketing).