6 ms·

This reads like internet fiction to me. Very vague and short.

spacebacon 5mo ago
[flagged]
abrookewood 5mo ago
Given the subject matter, it would be highly unethical to reveal the name of the company before verifying it was indeed fixed. I'd be wary of getting sued.
- croemer 5mo ago
  Company should just take down the whole thing. One vuln might be fixed but how many others might be there.
- mijoharas 5mo ago
  The first time I stumbled onto a big security vulnerability (exposed stripe/aws/play store keys. I was poking around an API a web app was using, and instead of hitting /api/v1, if you just hit /api it served them. I wasn't trying to do anything malicious), the very first thing I did was contacted a security researcher friend to ask about covering my ass while performing responsible disclosure. You hear too much about people being persecuted for trying to point out security vulnerabilities. (Guess they haven't heard about "don't shoot the messenger"). (It turned out fine after finally managing to speak with someone. Had to ring up customer service and say "look, here are the last digits of your stripe private key. Please speak with an engineer". Figuring out how to talk with someone was the difficult thing)
yawniek 5mo ago
fwiw i know tobias and its very very unlikely he made this up. my guess its intentionally vague to not leak any information about the culprit which i guess is fair.
- spacebacon 5mo ago
  It’s unlikely any LLM tasked with a prompt involving medical records did not automatically address separation of concerns. The type of data involved is worst case scenario. One JS file is also worst case scenario. This is why it may feel manufactured. If it is true, they truly deserve to be put on blast.
  - moooo99 5mo ago
    I can 100% imagine prompts that would even feel natural that would never hint at any medical background of the data being processed. Could be as simple as using customer instead of patient.
- rausr 5mo ago
  heh, I know that username. I came to the same conclusion. (I hope all is well with you, Yannick)
- BrissyCoder 5mo ago
  Okay. If it's real I apologize. But in any case it's so lacking in detail and so brief as to make it so uninteresting that it might as well be fake. > Somebody "vibecodes" medical app/system. The app was insecure. Personal info leaked. Okay cool.
  - kuboble 5mo ago
    Is really weird to me that this is your reception. It's a rarely updated personal blog, not a daily tabloid story.
    - BrissyCoder 5mo ago
      It's pure bs. If you read that blog post and think "this definitely happened", let alone "wow - this is interesting" then I have a monorail to sell you. > Technical Background > The entire application was a single HTML file with all JavaScript, CSS, and structure written inline. The backend was a managed database service with zero access control configured, no row-level security, nothing. All "access control" logic lived in the JavaScript on the client side, meaning the data was literally one curl command away from anyone who looked. > All audio recordings were sent directly to external AI APIs for transcription and summarization. > There was more, but this is already enough to get the idea. Hmmmm... interesting, now that I have the "Technical Background" I for sure know that this medical app was 100% vibe coded by a Medical Practice in the Real World and exists! (TM)
sixhobbits 5mo ago
yeah keeping it vague makes sense to protect the place if it's still online but the whole thing doesn't really make sense? The timelines mentioned are weird - he spoke to them before they built it? Or after? It's not that clear, he mentions they mentioned watching a video. > The entire application was a single HTML file with all JavaScript, CSS, and structure written inline. This is not my experience of how agents tend to build at all. I often _ask_ them to do that, but their tendency is to use a lot of files and structure > They even added a feature to record conversations during appointments So they have the front-desk laptop in the doctor's room? Or they were recording conversations anyway and now they for feed them into the system afterwards? > All "access control" logic lived in the JavaScript on the client side, meaning the data was literally one curl command away from anyone who looked. Also definitely not the normal way an agent would build something - security flaws yes, but this sounds more like someone who just learnt coding or the most upvoted post of all time on r/programmerhorror, not really AI. Overall I'm skeptical of the claims made in this article until I see stronger evidence (not that I'm supporting using slop for a medical system in general).
- fzzzy 5mo ago
  The single file thing makes perfect sense if it was built as an artifact in one of the big provider's webui.
- mewpmewp2 5mo ago
  I don't know what to make of the article. First I thought it seems like a made up LinkedIn story, it seems too crazy while talking about it in such a casual manner. Ultimately I don't know, maybe it was vague for a specific reason. I guess one thing I'd find odd is that whoever developed it, that they didn't run and get stuck with CORS issues, if everything was done client side to those services and that they managed to get API keys, subscription stuff everywhere while still making mistakes like this. And no mention of leaked api keys and creds which UI side there must have been, right? > Everything that could go wrong, did go wrong. Then this claim seems a bit too much, since what could have gone more wrong is malicious actors discovering it, right? Did they? Maybe I have trouble believing that a medical professional could be that careless and naive in such a way, but anything could happen. I guess another thought is... If they built it why would they share the URL to the author? Was author like "Ooh cool, let me check that out", and they just gave the url without auth? Because if it worked as it was supposed to it should have just shown a login screen right? That's the weirdest part to me, I suppose.
rubzah 5mo ago
I assure you that these kinds of things are happening right now.
watwut 5mo ago
Short writing is just a good writing. Also, it was not vague, it just omitted identifying information.
samuel 5mo ago
I don't think it's the case, but it would be very funny that this would end being AI generated clickbait.
drkiz75 5mo ago
Agreed. It’s right there at plausibly deniable just short of falsifying facts you could look up.
camillomiller 5mo ago
It’s Germany, you want to be as generic as you can because libel, privacy and similar laws are pretty strong here
no_shadowban_9 5mo ago
[dead]

[flagged]

Given the subject matter, it would be highly unethical to reveal the name of the company before verifying it was indeed fixed. I'd be wary of getting sued.

croemer 5mo ago
Company should just take down the whole thing. One vuln might be fixed but how many others might be there.
mijoharas 5mo ago
The first time I stumbled onto a big security vulnerability (exposed stripe/aws/play store keys. I was poking around an API a web app was using, and instead of hitting /api/v1, if you just hit /api it served them. I wasn't trying to do anything malicious), the very first thing I did was contacted a security researcher friend to ask about covering my ass while performing responsible disclosure. You hear too much about people being persecuted for trying to point out security vulnerabilities. (Guess they haven't heard about "don't shoot the messenger"). (It turned out fine after finally managing to speak with someone. Had to ring up customer service and say "look, here are the last digits of your stripe private key. Please speak with an engineer". Figuring out how to talk with someone was the difficult thing)

yawniek 5mo ago

fwiw i know tobias and its very very unlikely he made this up. my guess its intentionally vague to not leak any information about the culprit which i guess is fair.

spacebacon 5mo ago
It’s unlikely any LLM tasked with a prompt involving medical records did not automatically address separation of concerns. The type of data involved is worst case scenario. One JS file is also worst case scenario. This is why it may feel manufactured. If it is true, they truly deserve to be put on blast.
- moooo99 5mo ago
  I can 100% imagine prompts that would even feel natural that would never hint at any medical background of the data being processed. Could be as simple as using customer instead of patient.
rausr 5mo ago
heh, I know that username. I came to the same conclusion. (I hope all is well with you, Yannick)
BrissyCoder 5mo ago
Okay. If it's real I apologize. But in any case it's so lacking in detail and so brief as to make it so uninteresting that it might as well be fake. > Somebody "vibecodes" medical app/system. The app was insecure. Personal info leaked. Okay cool.
- kuboble 5mo ago
  Is really weird to me that this is your reception. It's a rarely updated personal blog, not a daily tabloid story.
  - BrissyCoder 5mo ago
    It's pure bs. If you read that blog post and think "this definitely happened", let alone "wow - this is interesting" then I have a monorail to sell you. > Technical Background > The entire application was a single HTML file with all JavaScript, CSS, and structure written inline. The backend was a managed database service with zero access control configured, no row-level security, nothing. All "access control" logic lived in the JavaScript on the client side, meaning the data was literally one curl command away from anyone who looked. > All audio recordings were sent directly to external AI APIs for transcription and summarization. > There was more, but this is already enough to get the idea. Hmmmm... interesting, now that I have the "Technical Background" I for sure know that this medical app was 100% vibe coded by a Medical Practice in the Real World and exists! (TM)

sixhobbits 5mo ago

yeah keeping it vague makes sense to protect the place if it's still online but the whole thing doesn't really make sense? The timelines mentioned are weird - he spoke to them before they built it? Or after? It's not that clear, he mentions they mentioned watching a video. > The entire application was a single HTML file with all JavaScript, CSS, and structure written inline. This is not my experience of how agents tend to build at all. I often _ask_ them to do that, but their tendency is to use a lot of files and structure > They even added a feature to record conversations during appointments So they have the front-desk laptop in the doctor's room? Or they were recording conversations anyway and now they for feed them into the system afterwards? > All "access control" logic lived in the JavaScript on the client side, meaning the data was literally one curl command away from anyone who looked. Also definitely not the normal way an agent would build something - security flaws yes, but this sounds more like someone who just learnt coding or the most upvoted post of all time on r/programmerhorror, not really AI. Overall I'm skeptical of the claims made in this article until I see stronger evidence (not that I'm supporting using slop for a medical system in general).

fzzzy 5mo ago
The single file thing makes perfect sense if it was built as an artifact in one of the big provider's webui.
mewpmewp2 5mo ago
I don't know what to make of the article. First I thought it seems like a made up LinkedIn story, it seems too crazy while talking about it in such a casual manner. Ultimately I don't know, maybe it was vague for a specific reason. I guess one thing I'd find odd is that whoever developed it, that they didn't run and get stuck with CORS issues, if everything was done client side to those services and that they managed to get API keys, subscription stuff everywhere while still making mistakes like this. And no mention of leaked api keys and creds which UI side there must have been, right? > Everything that could go wrong, did go wrong. Then this claim seems a bit too much, since what could have gone more wrong is malicious actors discovering it, right? Did they? Maybe I have trouble believing that a medical professional could be that careless and naive in such a way, but anything could happen. I guess another thought is... If they built it why would they share the URL to the author? Was author like "Ooh cool, let me check that out", and they just gave the url without auth? Because if it worked as it was supposed to it should have just shown a login screen right? That's the weirdest part to me, I suppose.

rubzah 5mo ago

I assure you that these kinds of things are happening right now.

watwut 5mo ago

Short writing is just a good writing. Also, it was not vague, it just omitted identifying information.

samuel 5mo ago

I don't think it's the case, but it would be very funny that this would end being AI generated clickbait.

drkiz75 5mo ago

Agreed. It’s right there at plausibly deniable just short of falsifying facts you could look up.

camillomiller 5mo ago

It’s Germany, you want to be as generic as you can because libel, privacy and similar laws are pretty strong here

no_shadowban_9 5mo ago

[dead]