9 ms·
Error by AI scribe during medical appointment leaves patient devastated
- deleted 26d ago[deleted]
- newsomix9xl 1mo agoNot only did the AI apparently hallucinate psilocybin consumption, but the doctor then hallucinated internal bleeding as a risk, which is not a known risk of such mushrooms. But this seems a bit click baity. You can challenge your medical record, and presumably an AI transcription service would be in there. Unless .au is special in that way.
- Herodotus38 28d agoThe doctor probably didn’t even read it and there’s a good chance the AI program wrote the bit about bleeding too. Source: am physician who has tried these programs and doesn’t use them.
- myhf 28d agoIt seems like this entire category of program is designed to make the same work take more time and more human attention. I can understand the appeal in the tech industry, where the increased costs can be deferred until the financial situation changes. But healthcare does not work like that.
- Herodotus38 28d agoI think you are right, These scribe programs can be used to help save time up front, at the cost that you have to spend the time review at the end to review. As others have said, it’s easy to skip the review part when you are distracted, and if most notes are fine with no edits needed you start to think you can get away without reviewing closely.
- roosterIllusi0n 27d agoWhy would anyone want a raw trascription in a medical record. There should be a summary and any suggestions, but the physician would still have to write the note, even if they are simply editing the AI summary. The physician would still be doing all the other charting where they enter conditions and problems.
- LoganDark 28d agoPsychedelics do tend to cause dehydration as the body tries to filter them out, and if you don't have a steady fluid intake you can dry out pretty badly before it's done. (Source: I take LSD frequently) I'd be hesitant to connect that to internal bleeding right off the bat though. Definitely would not trust claims that internal bleeding is a first-order effect of the drug.
- superkuh 28d agoA friend recently had a drug test added to their blood draw for an unrelated health check at a hospital. When she questioned the nurse about it turned out the doctors make notes that are transcribed by AI. And in this case the AI had mistakenly transcribed that she was on fentanyl patches. "the nurse who was trying to be very nice said in a less sweet tone that the doctor is supposed to review the notes so this does not happen. And that she is very glad I caught it and the doctor will be glad too. I do not know if this was something Very Serious or an ongoing problem or what." Given this threads story is the second incident of this I've heard in a week it seems like it is a common error with very serious consequences.
- flowerthoughts 28d ago> the doctor will be glad too. So even the doctor forgot what they had talked about, and assumed the transcription was right? Crikey.
- jamiequint 28d agoThe important question with any AI-related news story is what is the counterfactual. Sure, it sucks if your self driving car gets in a crash or your AI scribe incorrectly transcribes something to your medical record. This is news now. What isn't news is humans getting into crashes or doctors making poor medical decisions as a result of low quality or missing notes.
- kamranjon 28d agoI think if a real human doctor completely fabricated a drug usage history for one of their patients, that would also be news.
- paulv 28d agoExactly. When doctors make mistakes, they can be held accountable -- their malpractice insurance rates go up, their licenses are subject to suspension or revocation, they can go to jail (eg if they are pill mills) or they/the practice get a bad review. When AI makes mistakes, what happens? How is it held accountable?
- ars 28d agoDoctors make mistakes pretty much non-stop, and they are rarely held accountable because the system as a whole works OK. AI makes less mistakes than the doctors do (I read a lot of medical notes).
- ButlerianJihad 28d agofnord
- inigyou 28d agoYou know they'd have to be your roommate for that to amount to a positive test, right? They, correctly, assumed that your excuse for having a cannabis-user's dosage of cannabis in you is that you are a cannabis user, not because your neighbor two houses away smokes it.
- notabotiswear 28d agoYes. Perhaps pushing sophisticated models that require validation, on users who most definitely won't (and the whole premise of the model itself is that it spares them work) was a bad idea...
- bpodgursky 28d agoValidation will measure accuracy, and even if you require and get > 99.9% accuracy (and they should! don't get me wrong), it won't change that there will be anecdotes of patients getting wrong results given that millions of people go to the doctor a year. In fact, there will be MORE of those stories because it is shocking and strange, and clickworthy! The readers demand stories! But the reality is that there is no world where a story like this meaningfully informs the public about the accuracy of the services in question and the tradeoffs involved.
- NitpickLawyer 28d ago> even if you require and get > 99.9% accuracy [...] there will be anecdotes of patients getting wrong results given that millions of people go to the doctor a year. This is known as the long tail problem in ML, and it's a reality in almost every field. It's also why we don't officially have self driving cars, despite there being thousands of videos out there with cars driving autonomously for hours without any errors. But every now and then, there will be cases where the system doesn't work. There are two interesting aspects here. One, we don't actually have quantitative data on how often this happens in the same field when human errors don't get caught. In a perfect world we'd have that, plus a long study for the "AI" systems, and we'd get to compare the two. Secondly, even if we'd have that data, people would still act out against "the machine" in the (ideally fewer) cases where it errors out, compared to a doctor. It's part human nature, part (manufactured) rage against the machines. I also agree with your second paragraph. Case in point, when a waymo hit a cat, we got a shit ton of articles, riled up communities, and so on. Or every time that other car hits something we get plenty of press, even if some of the "accidents" are fender benders that likely wouldn't get reported otherwise.
- 28d ago
- lmpdev 28d agoI think we need to keep in mind that medical and pharmacy mistakes are still extremely common Recently my partner received a prescription with instructions that were over the LD50 (we caught it as it was obviously too high). We reported it but it was likely the case of the clerk simply hitting the wrong button and not double checking the resulting sticker I’d be interested to see the AI transcription failure rate compared with existing medical/pharmacy rates
- nradov 28d agoEHRs and other electronic prescription applications pretty much all have automated alerts for inappropriate dosages (like mg versus mcg) as well as allergies and harmful drug-drug interactions. The prescribing clinician likely had to dismiss at least one alert in making that error.
- globular-toast 28d agoThe pharmacist should have caught this. This is the reason doctors don't give drugs themselves and pharmacy is a completely separate, trained profession. Multiple things have to go wrong for you to get the wrong drugs. If that happened then something is very wrong with your pharmacy.
- unselect5917 28d ago"preventable medical errors" kill between 270,000 - 400,000 Americans every year. It was listed on this wikipedia page until July 11, 2020. https://en.wikipedia.org/wiki/Preventable_causes_of_death https://en.wikipedia.org/wiki/Preventable_causes_of_death I know because I frequently cited it in arguments with citizen disarmament advocates before it was edited out.
- red75prime 28d agoIt's easy to bring down the hammer of scope insensitivity by vividly describing a single person's struggle. The question is how AI affects error rates.
- cwillu 28d agoThe question is how AI affects error rates and error recovery rates! My experience dealing with ai-mediated processes is that the error recovery paths often simply don't exist, presumably because eliminating the personel that dealt with oddball and errors was the supposed benefit of having the AI deal with it in the first place.
- a2ff6eeb0 28d agoYeah, but also, if we can make AI do the same work cheaper than doctors, we can probably reduce the wages of doctors and save a bunch of money, which will both increase profits for health companies and reduce costs for patients. At some point, sufficient savings is better for patients than more doctors.
- UpsideDownRide 28d agoYou mean increase the profits AND increase the costs....
- inigyou 28d agoWhy would costs for patients decrease
- tdeck 28d agoThis response always comes up but I think it's not the whole story and gives "AI" a pass. If I report a bug in (for example) Slack that loses messages and cost me a lot of time and headaches, is it an appropriate response to say "but Teams has even more bugs" or "sure, but if you had that conversation face to face you might miss something too"?
- 28d ago
- s1artibartfast 28d agoAI makes errors, humans make errors. The relevant information is the rate for each. There are estimated hundreds of millions of medical transcription errors per year without AI. 42.4% of finalized medical notes still contained at least one error. Prior to AI it would be about voice to text. Prior to that it would be some transcriber in India. Prior to that it would be the doctor themselves
- senectus1 28d agogarbage in, garbage out. but with AI you get to add "hallucinations" to the mix.
- s1artibartfast 28d agoHumans are prone to hallucination and confabulation. Humans are notoriously bad at saying "I dont know" and wildly pattern match.
- senectus1 28d agothe fact that humans can say "I dont know" is a super power. AI will too often not say "I dont know" and just make shit up instead.
- cwillu 28d agoPrior to AI, you could get a person on the phone or in-person with the capacity to correct errors.
- s1artibartfast 28d agoCould you? Do you have a lot of experience before and after?
- cwillu 28d agoYes. Most recently, various major gig apps have stopped even offering any sort of human voice support, leading to cancelled orders and similar because the ai you have to use can't perform the discretionary actions to fix problems that the humans could do in back in march this year. However, the ai sounds positively heartbroken that it can't help, and the response latency is really good, so I guess that somehow makes up for the actual service being broken, right?
- pizzly 28d agoThe other side is that I have witnessed doctors making mistakes personally on medical files. They are overworked and only see you for 15 minutes and in a rush to kick you out and so of course they make errors. It would be interesting to compare whether transcription software or humans make the most mistakes. Guessing the answer will be very nuanced such as transcription software failing for accents not trained on, how over worked the clinic is etc.
- tdeck 28d agoThere's another option, which is to have the consultation recorded and then transcribed by a third person, or at least have the transcript reviewed while listening to the recording. I think for medical situations it's more than reasonable to put in the labor to ensure correctness.
- selestify 28d agoCost of medical care is already sky high (in the US). Are you sure you want to introduce even more administrative overhead?
- tdeck 28d agoHow were people doing this a few years ago before generative AI? Did the cost of care go down significantly and I just missed it?
- bpodgursky 28d agoThink about what this literally means. There are a million doctors in the US. Doctors spend their day consulting patients, so, you need a million transcription reviewers. Let's say you pay each of them $50k/yr (aka rock bottom US entry-level wage). That's $50 billion a year, and a million people pulled out of the labor force. Is that worth it? Is that really the best thing you can do with $50 billion dollars? In real life, tradeoffs exist.
- nradov 28d ago
- Incipient 28d agoI am not an AI evangelist by any means, but it does serve a purpose when used correctly. I feel like if you asked any half-intelligent AI "please review these notes and flag points we should review for correctness, or check with the patient?" I'm pretty sure it would pick up a huge chunk of issues?
- edoceo 28d agoJust prompt with "make no mistakes". (/s for clarity)
- crote 28d agoIt's a medical file, those rarely contain idle chit-chat. Everything should be reviewed for correctness.
- nradov 28d ago[dead]
- phil21 28d ago> Everything should be reviewed for correctness. Have you ever actually read your notes on file? This is basically living in a fantasy world. Errors are routine. One of my notes in my current medical file states I injured my shoulder playing for a NFL team. I have never played football, and certainly not at a professional level. Fixing it is sort of like trying to fix your credit report - you supposedly get it done, and then 6mo later the same error pops back up again. Ironically the reason this note exists is very similar to a way an AI scribe would misinterpret a conversation.
- insensible 28d agoWhen using an LLM for details like this I keep it out of the mode of generating prose as much and as long as possible. I get better results when I work with bullet points. I think this helps me keep the right frame of mind when reading the responses. It’s easier to scan the facts or points for accuracy. And I think it might help the LLM to be less creative with those points, but I don’t rely on that.
- kioleanu 28d agoI am shocked by the comments in this thread ranging from “yeah but humans also make mistakes” to “yeah but how many mistakes does an AI do compared to a human”. Neither of those is the point here. We have doctors that have employed AI as a software to help them and that software is flawed. I am a software developer and if I write some piece of medical software with bugs that are so blatant in it, I'd see hell up to potentially being sued into oblivion
- tdeck 28d agoSomehow the standards we have for every other kind of automation go out the window for AI. If it turned out an LLM embezzled funds and spent them at an internet casino, we would have folks in the comments explaining that what really matters is the embezzlement rate compared to humans doing the same job.
- Paria_Stark 28d agoThe cargo cult around AI on this website is making me second guess a career choice that has always been obvious to me. In addition to not enjoying my work as much as what I used to because it's become babysitting an superpowered AI toddler, I now have to deal with this kind of opinion online.
- inigyou 28d agoThe meta-economy losing touch with reality is a symptom of late stage capitalism. It used to matter whether a technology could solve a problem, then it started to also matter whether this fact could be explained to savvy investors, and now it only matters whether it can be explained to stupid investors because they are the ones with the money. You used to have to sell a product people wanted to get their money, but now customers don't have any money because it's all with billionaires so those are the only people you have to please.
- pineaux 28d agoIt has been called late stage capitalism since the 1920s
- arjie 28d agoThe classic whisper model will give you text in silent moments too. A lot of these guys just wrapped that. You can’t just do the minimal thing. Well I guess you can and you’ll get sales but it won’t correctly solve the problem. My cousin used one of these apps with patient consent and said she then has to listen to the audio and rewrite it all. Stopped for that reason. Wasn’t even a time saver. Strangely, I think Robin really just mistimed this. They stopped just as the state of the art came out and with their human in the loop transcription they might have been quite useful.
- sgt 28d agoI wonder if this app in question is Heidi Health? A lot of doctors in Australia use it.
- Izkata 28d agoThe issues were found and no one was hurt so they're not to the level of Therac-25, but it really feels like a matter of when, not if.
- eelioss 28d agoI am surprised the doctors do not have to review a bit the transcription. I do have to review my code before deployment
- somenameforme 28d agoWhat people have to do, and what people do do, are two different things.
- Gigachad 28d agoMy coworkers do not review the slop code they put up. Review is the easiest thing to shortcut since reviewing and finding nothing looks the same as not reviewing at all.
- Arn_Thor 28d agoYou can optimize work. But you can't optimize AWAY work. If you depend on human attention, expertise and judgement, as we do in medicine, you need that trained doctor to have the time to look at the transcript and think carefully about the contents. If we use AI as a way to help doctors spend even less time on each patient's case, we're going at it the wrong way. It shouldn't help cut costs, it should help improve the quality of care.
- junofan 28d agoThey’re just gonna have to think a lot faster. Scuttlebutt is we’re going to 12k patient panels, up from about 2–3k. Better to prep for what’s coming.
- drivebyhooting 28d agoThis isn’t a problem of you use actually capable models instead of wrapping whatever open source junk and pretend you have an AI product.
- inigyou 28d agoFalse.
- hypfer 28d agoI would be curious as to which (kind of) model they've used there, how the tooling and prompts and all look like, etc. I bet that to make the business case viable (or rather profitable), it's probably something small and cheap. Bigger and better models don't come with any guarantees as to correctness either, but they do push down the probability of something as wrong as this happening by orders of magnitude. That and processes. Even a smol dumb model can throw a report at both parties in the end where both need to sign off on it. Which should also scale better if both do, because the patient doesn't get fatigued because it's not happening many times per day. Or you maybe mirror what humans do and ask for active confirmation the less plausible something sounds. Hmm. __ Point being that I wouldn't necessarily blame it on the tech itself, but rather the (probably) startup, the culture and the fact that no one is going to jail here. Your honor, the floats are innocent. They were simply forced to do this by the evil startup founder.
- tanelih 28d ago> There's evidence that for the first few weeks of using AI, [people] check things pretty carefully and then at a point they stop checking because they assume it will be right. I think this is an issue people often overlook / does not get enough weight in the discussion. I do wonder that, when such profound technologies (such as AI, social media) are rolled out, should they be subjected to studies from the "human perspective" for a longer period of time. However, this might be impossible in the current system.
- simondanisch 28d agoAm I the only one who is ALSO shocked by the diagnosis, that microdosing is supposed to cause bleeding around the kidneys? Quick research shows there's no way that psilocybin could cause this especially when microdosing... This makes it so much worse since it wasn't just a transcript error, it was also that the doctor didn't even read the diagnosis?
- janalsncm 28d agoI had never heard of that before either but I found this: https://www.sciencedirect.com/science/article/pii/S2211753918301490 https://www.sciencedirect.com/science/article/pii/S221175391... This one is kidney failure not bleeding around the kidneys but it still surprised me. Microdosing definitely makes it even less likely of course.
- HaloZero 28d agoAny doctors in the room, I'm curious, would this kind of thing show up in an after visit summary report? I've seen that my doctor sends them but they seem AI generated and curious if it's just a summary of the transcript they made.
- cik 28d agoThere's a clear difference between defense, mission critical software, and slip. Medical cannot tolerate bugs of this nature (it's literally illegal in many a jurisdiction), which accounts for part of the cost, the approach, and testing cycles. Can we get there in the future - perhaps. But the accuracy needs to be much higher. I remember in the 90s when my father's clinic tried both Softvoice and Dragon. Comparing it to his receptionist typing out his notes, the accuracy was in the 80s - hence rejected. Just last week an AI startup here in Israel was kicked out of their HMO partner for having a transcription accuracy of only 92% in a mixed mode these where a doctor, patient, and caregiver are all in the room (think doctor + mom + kid) For the curious - trained medical receptions have and maintain accuracy rates when measures of 97%. This is just transcription. Hallucinating entire events is another, far worse thing.
- jdthedisciple 28d agoThere could be a good technical solution to at least minimize the risk: Instead of relying on the one-shot transcription, have the system double check key facts by asking the patient for confirmation: "Can you confirm you've you taken psychoactive drugs before?" – "yes/no" Etc
- jeffrallen 28d agoSigh. The error was made by the humans. Full stop.
- yard2010 28d agoIt's quite ironic that an AI hallucinated magic mushrooms use.
- sshine 28d agoI sit in an office space where some people work on customer support voice agents, and the experience is terrible. I’ve just had to call a support number for the first time in forever, and I’m met with a robot who perfectly understands my request (I speak very slowly and clearly), and proceeds to completely reject a 6 digit code, insisting it is 4 digits, and taking awkward silence to a new extreme (30-60 seconds between each interaction). I’ve worked on speech-to-text and have read transcripts between two people talking where everything the one person says comes out like complete gibberish because they’re on the wrong side of a single-directional microphone. A human would interject either during the meeting, or at least while transcribing. Putting it in production now is gambling your authenticity for cost saving, hoping your shitty voice bot is only as bad as the rest. Hopefully with this oncoming mass unemployment caused by AI automation we will have some hands free to provide some genuine service instead of this ridiculous nonsense.
- harikp2002 26d ago[flagged]