7 ms·

Exactly: what if my radiologist doesn’t know my particular rare condition? I would feel comfortable removing the human diagnostician. Let’s have actual human d

by srge 3y ago

Exactly: what if my radiologist doesn’t know my particular rare condition? I would feel comfortable removing the human diagnostician. Let’s have actual human doctors acting as researchers working to improve the AI diagnostics.

haldujai 3y ago
My perspective as both a radiologist and CS/AI researcher so exactly what you're proposing: 1. We don't practice in imaginary vacuums, it's easy* to identify that something looks abnormal and then refer to clinical resources/other physicians for rare diseases with specific questions in mind (i.e. recognizing the nuanced imaging finding and referring to a resource to assist). As a rare disease example, tumors of the eyeball/orbit are very rare, but detecting them is not. If I open the case and see one I can refer to StatDx to help me narrow my differential knowing what imaging features I'm looking for. This reduces my misdiagnosis rate (which as an aside is ~2-5% for radiologists, "major" clinically significant that impact morbidity or mortality ~2-10% of those depending on the study). 2. Rare disease are hard to diagnose, and would likely also be hard for an AI. Imaging appearances are not unique to the vast majority of diseases, especially what we cal the "weird and wonderful". Pelvic tuberculosis, endometriosis, advanced cervical cancer and advanced rectal cancer can look identical/nearly identical on MRI and the clinical portion as well as additional testing helps us get to the diagnosis. We don't have to diagnose everything based on a single imaging test, nor should we given: 3. Diagnosis is a tradeoff of sensitivity and specificity. You can't have both. Let's consider adrenal gland tumors. Statistically these are going to be benign, there is no specific imaging feature to tell a small adrenocortical carcinoma (ACC) ~1 in 1 million incidence from an adrenal adenoma (99% of adrenal lesions). We also can't tell them apart with a biopsy under a microscope. If you're unlucky enough to get an ACC you're basically shit out of luck as the only options we have are to recommend adrenal surgery (and their complications which can be death) to optimize sensitivity, or assume that it's benign and optimize for specificity considering disease prevalence and risk of overdiagnosis. In practice, we just use a cutoff of 4cm. I'm not sure how an AI would solve this, especially as there isn't a large enough training set. MD Anderson has the most experience of any center and they've had ~600 cases in 40+ years which as you can imagine encompasses a very heterogeneous imaging set (we didn't have multidetector CT or 3T abdominal MRI 20+ years ago). Overall, AI can and should help radiologists and as someone involved in this field I can't envision a world where we can safely remove the human diagnostician element from the mix, given that it's a spectrum of grey not black/white labelling as it is for object detection. We've had attempts with mammography and stroke AI and it's still horrendously inaccurate compared to what I expect out of a resident radiologist let alone an experience staff physician.
- visarga 3y ago
  A great post. I can attest the same about information extraction from semi-structured documents. The situation is far from full autonomy. Can't do anything without human in the loop, not even with the latest AI. I am seeing this trend - everyone can explain at length why AI is "not quite up there" in their own field, but believes it's "near AGI" in other fields. We find it hard to imagine future difficulties AI will have to face in general, we can only do that in our own field where we have learned from direct experience.
  - haldujai 3y ago
    Exactly this! I chose radiology given my background thinking I could "easily build radiology AI systems" and help our struggling system. Then I became a radiologist and quickly discovered how hard this is. Something "as simple" as NER and entity-linking on radiology reports is damn near impossible at the moment (even with SOTA LLMs which have made it easier but still not accurate enough for production use).
    - chaxor 3y ago
      NER, Entity linking, and relationship extraction definitely seem to be 'low hanging fruit' due to LLM improvements, but one of the big problems is that they really need a completely different architecture to limit the decoder vocab if using a decoder transformer for producing the set of sequences in relation extraction with specific entity ids. A Longformer with full attention to input sequence, and sliding window attention to a large dictionary could be a decent way to find tune a system like this, but there are few that try it. Unfortunately there's a lot of stupidity going around right now in thinking the answer is just to 'pRoOoMpT tHe LLm RiGhT', but that will always be exceedingly wasteful such that processing terabytes of files will be prohibitively expensive, and there's no guarantee the system will always restrict to the specific vocab and structure desired. The images in radiology definitely make these types of things harder, and the sparsity is an enormous issue. However, working with some projects in this area, I don't think it's as impenetrable as a lot of radiologists in AI suggest. The main thing needed in the field is adoption of better techniques and architectures to deal with these problems.

by srge 3y ago

haldujai 3y ago

My perspective as both a radiologist and CS/AI researcher so exactly what you're proposing: 1. We don't practice in imaginary vacuums, it's easy* to identify that something looks abnormal and then refer to clinical resources/other physicians for rare diseases with specific questions in mind (i.e. recognizing the nuanced imaging finding and referring to a resource to assist). As a rare disease example, tumors of the eyeball/orbit are very rare, but detecting them is not. If I open the case and see one I can refer to StatDx to help me narrow my differential knowing what imaging features I'm looking for. This reduces my misdiagnosis rate (which as an aside is ~2-5% for radiologists, "major" clinically significant that impact morbidity or mortality ~2-10% of those depending on the study). 2. Rare disease are hard to diagnose, and would likely also be hard for an AI. Imaging appearances are not unique to the vast majority of diseases, especially what we cal the "weird and wonderful". Pelvic tuberculosis, endometriosis, advanced cervical cancer and advanced rectal cancer can look identical/nearly identical on MRI and the clinical portion as well as additional testing helps us get to the diagnosis. We don't have to diagnose everything based on a single imaging test, nor should we given: 3. Diagnosis is a tradeoff of sensitivity and specificity. You can't have both. Let's consider adrenal gland tumors. Statistically these are going to be benign, there is no specific imaging feature to tell a small adrenocortical carcinoma (ACC) ~1 in 1 million incidence from an adrenal adenoma (99% of adrenal lesions). We also can't tell them apart with a biopsy under a microscope. If you're unlucky enough to get an ACC you're basically shit out of luck as the only options we have are to recommend adrenal surgery (and their complications which can be death) to optimize sensitivity, or assume that it's benign and optimize for specificity considering disease prevalence and risk of overdiagnosis. In practice, we just use a cutoff of 4cm. I'm not sure how an AI would solve this, especially as there isn't a large enough training set. MD Anderson has the most experience of any center and they've had ~600 cases in 40+ years which as you can imagine encompasses a very heterogeneous imaging set (we didn't have multidetector CT or 3T abdominal MRI 20+ years ago). Overall, AI can and should help radiologists and as someone involved in this field I can't envision a world where we can safely remove the human diagnostician element from the mix, given that it's a spectrum of grey not black/white labelling as it is for object detection. We've had attempts with mammography and stroke AI and it's still horrendously inaccurate compared to what I expect out of a resident radiologist let alone an experience staff physician.

visarga 3y ago
A great post. I can attest the same about information extraction from semi-structured documents. The situation is far from full autonomy. Can't do anything without human in the loop, not even with the latest AI. I am seeing this trend - everyone can explain at length why AI is "not quite up there" in their own field, but believes it's "near AGI" in other fields. We find it hard to imagine future difficulties AI will have to face in general, we can only do that in our own field where we have learned from direct experience.
- haldujai 3y ago
  Exactly this! I chose radiology given my background thinking I could "easily build radiology AI systems" and help our struggling system. Then I became a radiologist and quickly discovered how hard this is. Something "as simple" as NER and entity-linking on radiology reports is damn near impossible at the moment (even with SOTA LLMs which have made it easier but still not accurate enough for production use).
  - chaxor 3y ago
    NER, Entity linking, and relationship extraction definitely seem to be 'low hanging fruit' due to LLM improvements, but one of the big problems is that they really need a completely different architecture to limit the decoder vocab if using a decoder transformer for producing the set of sequences in relation extraction with specific entity ids. A Longformer with full attention to input sequence, and sliding window attention to a large dictionary could be a decent way to find tune a system like this, but there are few that try it. Unfortunately there's a lot of stupidity going around right now in thinking the answer is just to 'pRoOoMpT tHe LLm RiGhT', but that will always be exceedingly wasteful such that processing terabytes of files will be prohibitively expensive, and there's no guarantee the system will always restrict to the specific vocab and structure desired. The images in radiology definitely make these types of things harder, and the sparsity is an enormous issue. However, working with some projects in this area, I don't think it's as impenetrable as a lot of radiologists in AI suggest. The main thing needed in the field is adoption of better techniques and architectures to deal with these problems.