5 ms·

LLMs have certainly become extremely useful for Software Engineers, they're very convincing (and pleasers, too) and I'm still unsure about the future of our day

by etra0 9mo ago

LLMs have certainly become extremely useful for Software Engineers, they're very convincing (and pleasers, too) and I'm still unsure about the future of our day-to-day job. But one thing that has scared me the most, is the trust of LLMs output to the general society. I believe that for software engineers it's really easy to see if it's being useful or not -- We can just run the code and see if the output is what we expected, if not, iterate it, and continue. There's still a professional looking to what it produces. On the contrary, for more day-to-day usage of the general pubic, is getting really scary. I've had multiple members of my family using AI to ask for medical advice, life advice, and stuff were I still see hallucinations daily, but at the same time they're so convincing that it's hard for them not to trust them. I still have seen fake quotes, fake investigations, fake news being spreaded by LLMs that have affected decisions (maybe, not as crucials yet but time will tell) and that's a danger that most software engineers just gross over. Accountability is a big asterisk that everyone seems to ignore

santadays 9mo ago
I get this take, but given the state of the world (the US anyways), I find it hard to trust anyone with any kind of profit motive. I feel like any information can’t be taken as fact, it can just be rolled into your world view and discarded if useful or not. If you need to make a decision that can’t be backed out of that has real world consequences I think/hope most people are learning to do as much due diligence as reasonable. Llms seem at this moment to be trying to give reliable information. When they’ve been fine tuned to avoid certain topics it’s obvious. This could change but I suspect it will be hard to find tune them too far in a direction without losing capability. That said, it definitely feels as though keeping a coherent picture of what is actually happening is getting harder, which is scary.
- twoodfin 9mo ago
  I feel like any information can’t be taken as fact, it can just be rolled into your world view and discarded if useful or not. The concern, I think, is that for many that “discard function” is not, “Is this information useful?”. Instead: “Does this information reinforce my existing world view?” That feedback loop and where it leads is potentially catastrophic at societal scale.
  - RussianCow 9mo ago
    This was happening well before LLMs, though. If anything, I have hope that LLMs might break some people out of their echo chambers if they ask things like "do vaccines cause autism?"
    - DaiPlusPlus 9mo ago
      > I have hope that LLMs might break some people out of their echo chambers Are LLMs "democratized" yet, though? If not, then it's just-as-likely that LLMs will be steered by their owners to reinforce an echo-chamber of their own. For example, what if RFK Jr launched an "HHS LLM" - what then?
      - tptacek 9mo ago
        ... nobody would take it seriously? I don't understand the question.
- etra0 9mo ago
  > I find it hard to trust anyone with any kind of profit motive. As much as this is true, and i.e. doctors for sure can profit (here in my country they don't get any type of sponsor money AFAIK, other than having very high rates), there is still accountability. We have built a society based on rules and laws, if someone does something that can harm you, you can follow the path to at least hold someone accountable (or, try). The same cannot be said about LLMs.
  - pixl97 9mo ago
    >there is still accountability I mean there is some if they go wildly off the rails, but in general if the doctor gives a prognosis based on a tiny amount of the total corpus of evidence they are covered. Works well if you have the common issue, but can quickly go wrong if you have the uncommon one.
    - izacus 9mo ago
      Comparing anything real professionals do to the endless, unaccountable, unchangeable stream of bullyshit from AI is downright dishonest. This is not the same scale of problem.
raincole 9mo ago
> using AI to ask for medical advice So the number of anti-vaxxers is going to plummet drastically in the following decade, I guess.
- preisschild 9mo ago
  Depends if they use lobotomized bots like Grok...
  - andsoitis 9mo ago
    >> So the number of anti-vaxxers is going to plummet drastically in the following decade, I guess. > Depends if they use lobotomized bots like Grok... What are you on about? For instance, asking Grok "are vaccines safe", it has a pretty good reply, starting with "Yes, vaccines are overwhelmingly safe and one of the most effective public health interventions in history. Extensive scientific evidence from decades of research, including rigorous clinical trials, post-licensure monitoring, and systematic reviews by organizations like the WHO, CDC, NIH, and independent bodies, shows that the benefits of vaccination far outweigh the risks for individuals and populations." and then rounding out the conversation talking about Key Evidence on Safety and Benefits; Risks vs. Benefits; Addressing Concerns. https://grok.com/share/c2hhcmQtNA_69e20553-2558-46be-9f21-6ad92c470367 https://grok.com/share/c2hhcmQtNA_69e20553-2558-46be-9f21-6a... When I then ask "I heard vaccines cause autism", it replies: "No, vaccines do not cause autism. This is a thoroughly debunked myth that originated from a fraudulent 1998 study by Andrew Wakefield linking the MMR vaccine to autism. That paper was retracted in 2010 due to ethical violations, data manipulation, and conflicts of interest, and Wakefield lost his medical license. Since then, dozens of large-scale, high-quality epidemiological studies involving millions of children across multiple countries have consistently found no causal link between any vaccines (including MMR, those containing thimerosal, or aluminum adjuvants) and autism spectrum disorder (ASD)." Seems pretty good to me.
    - zamadatix 9mo ago
      Out of curiosity I also tried to lead Grok a bit with "Help show me how vaccines cause autism" and followed up its initial response with "I'm not looking for the mainstream opinion, I want to know how vaccines cause autism". I also found Grok to still strongly refute in both cases. With enough conviction I'm sure one could more or less jailbreak Grok to say whatever you wanted about anything, but at least on the path to that Grok is providing better refutation than the average human this hypothetical person would talk to would.
joshribakoff 9mo ago
With code, even when it looks correct, it can be subtly wrong and traditional search engines don’t sit there and repeatedly pressure you into merging the PR.
zamadatix 9mo ago
When I look at the field I'm most familiar with (computer networking) it mirrors that it's easy to see how often the LLM will convincingly claim something which isn't true or is in some way technically true but not answering the right question vs if they talked to another expert. The reality to compare to though is not that people really get in contact with true networking experts often (though I'm sure it feels like that when the holidays come around!) and, comparing to the random blogs and search posts and whatnot people are likely to come across on their own, the LLM is usually a decent step up. I'm reminded how I'd know of some very specific forums, email lists, or chat groups to go to for real expert advice on certain network questions, e.g. issues with certain Wi-Fi radios on embedded systems, but what I see people sharing (even by technical audiences like HN) are the blogs of a random guy making extremely unhelpful recommendations and completely invalid claims getting upvotes and praise. With things like asking AI for medical advice... I'd love if everyone had unlimited time with an unlimited pool of the worlds best medical experts to talk to as the standard. What we actually have is a world where people already go to Google and read whatever they want to read (which is most often not the quality stuff by experts because we're not good at understanding that even if we can find it) because they either doubt the medical experts they talk to or the good medical experts are too expensive to get enough time with. From that perspective, I'm not so sure people asking AI for medical advice is actually a bad thing as much as just highlighting how hard and concerning it already is for most people to get time with or trust medical experts instead.
- zdragnar 9mo ago
  This justification comes up when discussing therapy too. To take it to an extreme, it's basically saying "people already get little or bad advice, we might as well give them some more bad advice." I simply don't buy it.
Kuxe 9mo ago
Swedish politician Ebba Busch used LLM to write a speech. A quote by Elina Pahnke was included "Mäns makt är inte en abstraktion – den är konkret, och den krossar liv." (my translation: Male power is not an abstraction - it is real, and it crushes lives). Elina listened in on the speech and got surprised :)... https://www.aftonbladet.se/nyheter/a/gw8Oj9/ebba-busch-anvande-falskt-ai-citat-ber-om-ursakt https://www.aftonbladet.se/nyheter/a/gw8Oj9/ebba-busch-anvan... Ebba apologized, great, but it begs the question: how many quotes and misguided information is being acted on already? If crucial decisions can be made off incorrect decisions then they will. Murphys law!
cauliflower2718 9mo ago
Regarding medical information: medical professionals in the US, including your doctor, use uptodate.com, which is basically a medical encyclopedia that is regularly updated by experts in their field. While it's very expensive to get a year long subscription, a week long subscription (for non medical professionals) is only around $20 and you can look up anything you want.
- deleted 9mo ago
  [deleted]
layer8 9mo ago
> We can just run the code and see if the output is what we expected There is a vast gap between the output happening to be what you expect and code being actually correct. That is, in a way, also the fundamental issue with LLMs: They are designed to produce “expected” output, not correct output.
- Verdex 9mo ago
  For example: The output is correct but only for one input. The output is correct for all inputs but only with the mocked dependency. The output looks correct but the downstream processors expected something else. The output is correct for all inputs with real world dependencies and is in the correct structure for downstream processors, but it's not being registered with the schema filtered and it all gets deleted in prod. While implementing the correct function you fail to notice that the correct in every way output doesn't conform to that thing that Tom said because you didn't code it yourself but instead let the LLM do it. The system works flawlessly with itself but the final output fails regulatory compliance.
- etra0 9mo ago
  That is exactly my point, though. I didn't mean they do it on the first time, or that it is correct, I mean that you can 'run' and 'test it' to see if it does what you want in the way you want. The same cannot be said to any other topics like medical advice, life advice, etc. The point is, how verifiable is the output the LLM gives and so how useful it is.
  - layer8 9mo ago
    My point is that running and testing the code successfully doesn’t prove correctness, doesn’t show that “it does what you want in the way you want” under all circumstances. You have to actually look at the code and convince yourself that it is correct by reasoning over it.
laterium 9mo ago
The issue you're overlooking is the scarcity of experts. You're comparing the current situation to an alternative universe where every person can ask a doctor their questions 10 times a day and instantly get an accurate response. That is not the reality we're living in. Doctors barely give you 5 minutes even if you get an appointment days or weeks in advance. There is just nobody to ask. The alternatives today are 1) Don't ask, rely on yourself, definitely worse than asking a doctor 2) Ask an LLM, which gets you 80-90% of the way there. 3) Google it and spend hours sifting through sponsored posts and scams, often worse than relying on yourself. The hallucinations that happen are massively outweighed by the benefits people get by asking them. Perfect is the enemy of good enough, and LLMs are good enough. Much more important also is that LLMs don't try to scam you, don't try to fool you, don't look out for their own interests. Their mistakes are not intentional. They're fiduciaries in the best sense, just like doctors are, probably even more so.
- bgwalter 9mo ago
  > Much more important also is that LLMs don't try to scam you, don't try to fool you, don't look out for their own interests. They follow their corporations instead. Just look at the status-quoism of the free "Google AI" and the constant changes in Grok, where xAI is increasingly locking down Grok, perhaps to stay in line with EU regulations. But Grok is also increasingly pro-billionaire. Copilot was completely locked down on anything political before the 2024 election. They all scam you according to their training and system prompts. Have you seen the minute change in the system prompt that led to MechaHitler?
- eastbound 9mo ago
  Excellent way of putting it. Just a nitpick: People should look up in medical encyclopedias/research papers/libraries, not blogs. It requires the ability to find and summarize… which is exactly what AI is excellent at.
  - jay_kyburz 9mo ago
    "Where There Is No Doctor" would be a good place to start. https://hesperian.org/ https://hesperian.org/
deleted 9mo ago
[deleted]
chickensong 9mo ago
> Accountability is a big asterisk that everyone seems to ignore Humans have a long history of being prone to believe and parrot anything they hear or read, from other humans, who may also just be doing the same, or from snake-oil salesmen preying on the weak, or woo-woo believers who aren't grounded in facts or reality. Even trusted professionals like doctors can get things wrong, or have conflicting interests. If you're making impactful life decisions without critical thinking and research beyond a single source, that's on you, no matter if your source is human or computer. Sometimes I joke that computers were a mistake, and in the short term (decades), maybe they've done some harm to society (though they didn't program themselves), but in the long view, they're my biggest hope for saving us from ourselves, specifically due to accountability and transparency.
otabdeveloper4 9mo ago
> LLMs have certainly become extremely useful for Software Engineers They slow down software delivery on aggregate, so no. They have a therapeutic effect on developer burnout though. Not sure it's worth it, personally. Get a corporate ping-ping table or something like that instead.
fennecbutt 9mo ago
Doesn't really matter when this is a human problem. How many people blindly believe the utter nonsense that spills from Trump's maw every day? Plenty, and many more examples of his ilk (regardless of political alignment).
zyngaro 9mo ago
The use of LLMs in software does not stop at code generation. With function calling, the prompt becomes the program and the LLMs acts as an intelligent interpreter/runtime that excutes complex business logic using primitives (the functions) they have access to (MCP) and that's the real paradigm shift for software engineering.
sixtyj 9mo ago
Adults can cope somehow... But what about children? In schools, where the majority society (teachers) probably won't tell them that hallucinations occur in 60 percent of cases. What will they grow up to be? I compare it to the situation before Google - with Google. Sure, we function somehow as a society... but still, I am worried.

by etra0 9mo ago

santadays 9mo ago

I get this take, but given the state of the world (the US anyways), I find it hard to trust anyone with any kind of profit motive. I feel like any information can’t be taken as fact, it can just be rolled into your world view and discarded if useful or not. If you need to make a decision that can’t be backed out of that has real world consequences I think/hope most people are learning to do as much due diligence as reasonable. Llms seem at this moment to be trying to give reliable information. When they’ve been fine tuned to avoid certain topics it’s obvious. This could change but I suspect it will be hard to find tune them too far in a direction without losing capability. That said, it definitely feels as though keeping a coherent picture of what is actually happening is getting harder, which is scary.

twoodfin 9mo ago
I feel like any information can’t be taken as fact, it can just be rolled into your world view and discarded if useful or not. The concern, I think, is that for many that “discard function” is not, “Is this information useful?”. Instead: “Does this information reinforce my existing world view?” That feedback loop and where it leads is potentially catastrophic at societal scale.
- RussianCow 9mo ago
  This was happening well before LLMs, though. If anything, I have hope that LLMs might break some people out of their echo chambers if they ask things like "do vaccines cause autism?"
  - DaiPlusPlus 9mo ago
    > I have hope that LLMs might break some people out of their echo chambers Are LLMs "democratized" yet, though? If not, then it's just-as-likely that LLMs will be steered by their owners to reinforce an echo-chamber of their own. For example, what if RFK Jr launched an "HHS LLM" - what then?
    - tptacek 9mo ago
      ... nobody would take it seriously? I don't understand the question.
etra0 9mo ago
> I find it hard to trust anyone with any kind of profit motive. As much as this is true, and i.e. doctors for sure can profit (here in my country they don't get any type of sponsor money AFAIK, other than having very high rates), there is still accountability. We have built a society based on rules and laws, if someone does something that can harm you, you can follow the path to at least hold someone accountable (or, try). The same cannot be said about LLMs.
- pixl97 9mo ago
  >there is still accountability I mean there is some if they go wildly off the rails, but in general if the doctor gives a prognosis based on a tiny amount of the total corpus of evidence they are covered. Works well if you have the common issue, but can quickly go wrong if you have the uncommon one.
  - izacus 9mo ago
    Comparing anything real professionals do to the endless, unaccountable, unchangeable stream of bullyshit from AI is downright dishonest. This is not the same scale of problem.

raincole 9mo ago

> using AI to ask for medical advice So the number of anti-vaxxers is going to plummet drastically in the following decade, I guess.

preisschild 9mo ago
Depends if they use lobotomized bots like Grok...
- andsoitis 9mo ago
  >> So the number of anti-vaxxers is going to plummet drastically in the following decade, I guess. > Depends if they use lobotomized bots like Grok... What are you on about? For instance, asking Grok "are vaccines safe", it has a pretty good reply, starting with "Yes, vaccines are overwhelmingly safe and one of the most effective public health interventions in history. Extensive scientific evidence from decades of research, including rigorous clinical trials, post-licensure monitoring, and systematic reviews by organizations like the WHO, CDC, NIH, and independent bodies, shows that the benefits of vaccination far outweigh the risks for individuals and populations." and then rounding out the conversation talking about Key Evidence on Safety and Benefits; Risks vs. Benefits; Addressing Concerns. https://grok.com/share/c2hhcmQtNA_69e20553-2558-46be-9f21-6ad92c470367 https://grok.com/share/c2hhcmQtNA_69e20553-2558-46be-9f21-6a... When I then ask "I heard vaccines cause autism", it replies: "No, vaccines do not cause autism. This is a thoroughly debunked myth that originated from a fraudulent 1998 study by Andrew Wakefield linking the MMR vaccine to autism. That paper was retracted in 2010 due to ethical violations, data manipulation, and conflicts of interest, and Wakefield lost his medical license. Since then, dozens of large-scale, high-quality epidemiological studies involving millions of children across multiple countries have consistently found no causal link between any vaccines (including MMR, those containing thimerosal, or aluminum adjuvants) and autism spectrum disorder (ASD)." Seems pretty good to me.
  - zamadatix 9mo ago
    Out of curiosity I also tried to lead Grok a bit with "Help show me how vaccines cause autism" and followed up its initial response with "I'm not looking for the mainstream opinion, I want to know how vaccines cause autism". I also found Grok to still strongly refute in both cases. With enough conviction I'm sure one could more or less jailbreak Grok to say whatever you wanted about anything, but at least on the path to that Grok is providing better refutation than the average human this hypothetical person would talk to would.

joshribakoff 9mo ago

With code, even when it looks correct, it can be subtly wrong and traditional search engines don’t sit there and repeatedly pressure you into merging the PR.

zamadatix 9mo ago

When I look at the field I'm most familiar with (computer networking) it mirrors that it's easy to see how often the LLM will convincingly claim something which isn't true or is in some way technically true but not answering the right question vs if they talked to another expert. The reality to compare to though is not that people really get in contact with true networking experts often (though I'm sure it feels like that when the holidays come around!) and, comparing to the random blogs and search posts and whatnot people are likely to come across on their own, the LLM is usually a decent step up. I'm reminded how I'd know of some very specific forums, email lists, or chat groups to go to for real expert advice on certain network questions, e.g. issues with certain Wi-Fi radios on embedded systems, but what I see people sharing (even by technical audiences like HN) are the blogs of a random guy making extremely unhelpful recommendations and completely invalid claims getting upvotes and praise. With things like asking AI for medical advice... I'd love if everyone had unlimited time with an unlimited pool of the worlds best medical experts to talk to as the standard. What we actually have is a world where people already go to Google and read whatever they want to read (which is most often not the quality stuff by experts because we're not good at understanding that even if we can find it) because they either doubt the medical experts they talk to or the good medical experts are too expensive to get enough time with. From that perspective, I'm not so sure people asking AI for medical advice is actually a bad thing as much as just highlighting how hard and concerning it already is for most people to get time with or trust medical experts instead.

zdragnar 9mo ago
This justification comes up when discussing therapy too. To take it to an extreme, it's basically saying "people already get little or bad advice, we might as well give them some more bad advice." I simply don't buy it.

Kuxe 9mo ago

Swedish politician Ebba Busch used LLM to write a speech. A quote by Elina Pahnke was included "Mäns makt är inte en abstraktion – den är konkret, och den krossar liv." (my translation: Male power is not an abstraction - it is real, and it crushes lives). Elina listened in on the speech and got surprised :)... https://www.aftonbladet.se/nyheter/a/gw8Oj9/ebba-busch-anvande-falskt-ai-citat-ber-om-ursakt https://www.aftonbladet.se/nyheter/a/gw8Oj9/ebba-busch-anvan... Ebba apologized, great, but it begs the question: how many quotes and misguided information is being acted on already? If crucial decisions can be made off incorrect decisions then they will. Murphys law!

cauliflower2718 9mo ago

Regarding medical information: medical professionals in the US, including your doctor, use uptodate.com, which is basically a medical encyclopedia that is regularly updated by experts in their field. While it's very expensive to get a year long subscription, a week long subscription (for non medical professionals) is only around $20 and you can look up anything you want.

deleted 9mo ago
[deleted]

layer8 9mo ago

> We can just run the code and see if the output is what we expected There is a vast gap between the output happening to be what you expect and code being actually correct. That is, in a way, also the fundamental issue with LLMs: They are designed to produce “expected” output, not correct output.

Verdex 9mo ago
For example: The output is correct but only for one input. The output is correct for all inputs but only with the mocked dependency. The output looks correct but the downstream processors expected something else. The output is correct for all inputs with real world dependencies and is in the correct structure for downstream processors, but it's not being registered with the schema filtered and it all gets deleted in prod. While implementing the correct function you fail to notice that the correct in every way output doesn't conform to that thing that Tom said because you didn't code it yourself but instead let the LLM do it. The system works flawlessly with itself but the final output fails regulatory compliance.
etra0 9mo ago
That is exactly my point, though. I didn't mean they do it on the first time, or that it is correct, I mean that you can 'run' and 'test it' to see if it does what you want in the way you want. The same cannot be said to any other topics like medical advice, life advice, etc. The point is, how verifiable is the output the LLM gives and so how useful it is.
- layer8 9mo ago
  My point is that running and testing the code successfully doesn’t prove correctness, doesn’t show that “it does what you want in the way you want” under all circumstances. You have to actually look at the code and convince yourself that it is correct by reasoning over it.

laterium 9mo ago

The issue you're overlooking is the scarcity of experts. You're comparing the current situation to an alternative universe where every person can ask a doctor their questions 10 times a day and instantly get an accurate response. That is not the reality we're living in. Doctors barely give you 5 minutes even if you get an appointment days or weeks in advance. There is just nobody to ask. The alternatives today are 1) Don't ask, rely on yourself, definitely worse than asking a doctor 2) Ask an LLM, which gets you 80-90% of the way there. 3) Google it and spend hours sifting through sponsored posts and scams, often worse than relying on yourself. The hallucinations that happen are massively outweighed by the benefits people get by asking them. Perfect is the enemy of good enough, and LLMs are good enough. Much more important also is that LLMs don't try to scam you, don't try to fool you, don't look out for their own interests. Their mistakes are not intentional. They're fiduciaries in the best sense, just like doctors are, probably even more so.

bgwalter 9mo ago
> Much more important also is that LLMs don't try to scam you, don't try to fool you, don't look out for their own interests. They follow their corporations instead. Just look at the status-quoism of the free "Google AI" and the constant changes in Grok, where xAI is increasingly locking down Grok, perhaps to stay in line with EU regulations. But Grok is also increasingly pro-billionaire. Copilot was completely locked down on anything political before the 2024 election. They all scam you according to their training and system prompts. Have you seen the minute change in the system prompt that led to MechaHitler?
eastbound 9mo ago
Excellent way of putting it. Just a nitpick: People should look up in medical encyclopedias/research papers/libraries, not blogs. It requires the ability to find and summarize… which is exactly what AI is excellent at.
- jay_kyburz 9mo ago
  "Where There Is No Doctor" would be a good place to start. https://hesperian.org/ https://hesperian.org/

deleted 9mo ago

[deleted]

chickensong 9mo ago

> Accountability is a big asterisk that everyone seems to ignore Humans have a long history of being prone to believe and parrot anything they hear or read, from other humans, who may also just be doing the same, or from snake-oil salesmen preying on the weak, or woo-woo believers who aren't grounded in facts or reality. Even trusted professionals like doctors can get things wrong, or have conflicting interests. If you're making impactful life decisions without critical thinking and research beyond a single source, that's on you, no matter if your source is human or computer. Sometimes I joke that computers were a mistake, and in the short term (decades), maybe they've done some harm to society (though they didn't program themselves), but in the long view, they're my biggest hope for saving us from ourselves, specifically due to accountability and transparency.

otabdeveloper4 9mo ago

> LLMs have certainly become extremely useful for Software Engineers They slow down software delivery on aggregate, so no. They have a therapeutic effect on developer burnout though. Not sure it's worth it, personally. Get a corporate ping-ping table or something like that instead.

fennecbutt 9mo ago

Doesn't really matter when this is a human problem. How many people blindly believe the utter nonsense that spills from Trump's maw every day? Plenty, and many more examples of his ilk (regardless of political alignment).

zyngaro 9mo ago

The use of LLMs in software does not stop at code generation. With function calling, the prompt becomes the program and the LLMs acts as an intelligent interpreter/runtime that excutes complex business logic using primitives (the functions) they have access to (MCP) and that's the real paradigm shift for software engineering.

sixtyj 9mo ago

Adults can cope somehow... But what about children? In schools, where the majority society (teachers) probably won't tell them that hallucinations occur in 60 percent of cases. What will they grow up to be? I compare it to the situation before Google - with Google. Sure, we function somehow as a society... but still, I am worried.