7 ms·

Okay I guess I've just had a different experience entirely. Maybe I'm jaded by hallucinations. The code ChatGPT generates is often bad in ways that are hard to

Okay I guess I've just had a different experience entirely. Maybe I'm jaded by hallucinations. The code ChatGPT generates is often bad in ways that are hard to detect. If you are not an experienced software engineer, the defects could be impossible to detect, until you/ChatGPT has gone and exposed all your customers to bad actors, or crash at runtime, or do something terribly incorrect. As far as other thought work goes, I am not consulting ChatGPT over, say, a dietician or a doctor. The hallucination risk is too high. Producing an answer is the not the same as producing a correct answer.

TeMPOraL 2y ago
My experience actually agrees with you. It's just that the set of use cases that either: - Are hard (or boring) to do, but easy to evaluate - for me, e.g. writing code, OCR, ideation; or - Don't require a perfectly correct answer, but more of a starting point or map of the problem space; or - Are very subjective, or creative, with there being no single correct answer, is surprisingly large. It covers pretty much everything, but not everything for everyone at the same time.
sumedh 2y ago
> The code ChatGPT generates is often bad in ways that are hard to detect. Does it work though, yes it does. There are many human coders who write bad code and life goes.
deleted 2y ago
[deleted]
ignoramous 2y ago
> Okay I guess I've just had a different experience entirely. I've seen both the good and the bad. I really like the good parts. Most recently, Claude Sonnet 3.5 fixed a math error in my code (I prompted it to check for it from a well-written bug report, and it did it fix it ever so perfectly). These days, it is pretty much second nature for me to pull up a new file & prompt Copilot to complete writing the entire code from my comment trails. I don't think I've seen as much change in my coding behaviour since Borland Turbo C -> NetBeans.
nvarsj 2y ago
I agree. I've just seen it hallucinate too many things that on the surface seem very plausible but are complete fabrications. Basically my trust is near 0 for anything chatgpt, etc. spits out. My latest challenge is dealing with people that trust chatgp to be infallible, and just quote the garbage to make themselves look like they know what they are talking about.
- gwervc 2y ago
  > things that on the surface seem very plausible but are complete fabrications LLMs are language model, it's crazy people expect them to be correct in anything beyond surface level language.
  - nvarsj 2y ago
    Yeah, I was probably being a bit too harsh in my original comment. I do find them useful, you just have to be wary of the output.
itsoktocry 2y ago
If your procees is asking it to "write me all this code", then you slap it in production, you're going to have a bad time. But there's intermediate ground. >I am not consulting ChatGPT over, say, a dietician or doctor Do you know any doctors, by chance? You have way more faith in experts than I do.
- kortilla 2y ago
  ChatGPT is just statistically associating what it’s observed online. I wouldn’t take dietary advice from the mean output of Reddit with more trust than an expert.
  - interstice 2y ago
    Doctors can be associating what they’ve learned, often with heavy biases from hypochondriacs and not enough time per patient to really consider the options. I’ve had multiple friends get seriously ill before a doctor took their symptoms seriously, and this is a country with decent healthcare by all accounts. Human biases are bad too.
    - beachy 2y ago
      > Doctors can be associating what they’ve learned, often with heavy biases from hypochondriacs So true. And it's hard to question a doctor's advice, because of their aura of authority, whereas it's easy to do further validation of an LLMs diagnosis. I had to change doctor recently when moving towns. It was only when chancing on a good doctor that I realised how bad my old doctor was - a nice guy but cruising to retirement. And my experience with cardiologists has been the same. Happy to get medical advice from an LLM though I'd certainly want prescriptions and action plans vetted by a human.
      - throwaway2037 2y ago
        > It was only when chancing on a good doctor that I realised how bad my old doctor was How did you determine the new doctor is "good"?
CuriouslyC 2y ago
Easy answer. Ask ChatGPT to write testable code, and tests for the code, then just verify the tests. If the tests don't work, have ChatGPT use the test output to rewrite the code until it does. If you can't have ChatGPT write testable code because of your architecture, you have other problems. People with bad process and bad architecture saying AI is bad because it doesn't work well with their dumpster fire systems, 100% facepalm.
- aleph_minus_one 2y ago
  > If you can't have ChatGPT write testable code because of your architecture, you have other problems. There exist lots of reasons why code is hard to test automatically that have nothing to do with the architecture of the code, but with the domain for which the code is written and runs.
ern 2y ago
I recently needed to help a downstream team with a problem with an Android app. I never did mobile app dev before, but I was able to spin up a POC (having not coded in Java for 22 years) and solve the problem with the help of ChatGPT 4.0. Sure I probably would have been able to do it without ChatGPT, but it was so much easier to have something to bounce ideas off-of. A safety net, if you will. The hallucination risk was irrelevant: it did hallucinate a little early on. I told it it was a hallucinating, and we moved onto a different way of solving the problem. It was easy enough to verify it was working as expected.
- gexla 2y ago
  Seems to me this is the equivalent of fast retrieval and piecing together from a huge amount of examples in the data. This might take far more time if you were to do this yourself. That's a plus for the tools. In other words, a massively expensive (for the service provider) auto-complete. But try to do something much more simple but has much fewer examples (a typical case is something which has bad documentation) in the data, and it falls apart. I even tried to use Perplexity to create a dead simple CLI command, and it hallucinated an answer (looking at the docs, it misused the parameter, and may have picked up on someone who gave an incorrect answer in the data.)
dmix 2y ago
It's already gotten significantly better and faster in a few yrs. Maybe LLMs will hit a wall in the next 5yrs but even if it does it's still extremely useful and there are always other ways to optimize the current technology where this is already a major development for society.
jakderrida 2y ago
>The code ChatGPT generates is often bad in ways that are hard to detect. If you are not an experienced software engineer, the defects could be impossible to detect I keep hearing this, but it's incorrect. While I only know R, which is obviously a simple language, I would never type out all my code and go without testing to ensure it does what I intended before using it regularly. So I can't imagine someone that knows a more complex language just typing out all of it before integrating it into business systems at their work or anything else before testing it. Why would AI be any different? Why the hell are AI skeptics acting like getting help from an LLM would involve not testing anything? Of course I test it! Why on earth wouldn't I? Just as I tested code made by freelancers I hired on commission before using the code I bought from them. Do AI skeptics really not test their own code? Are you all insane?
- disgruntledphd2 2y ago
  > While I only know R, which is obviously a simple language Take it from someone who started with R, R is 100% not a simple language. If you can write good R, you're probably a surprisingly good potential SE as R is kinda insane and inconsistent due to 50+ years of history (from S, to R etc).
  - jakderrida 2y ago
    Hmmm.. I'm trying to imagine interviewing for SE and telling them I got wealthy from a crypto market-making algorithm I coded in R during Covid and the interviewer responding with anything but laughter or with silence as they ponder legal ways to question my mental health. It's an excellent language, I think, for many reasons. One is that you can work with data within hours because even before learning what packages or classes are, you got native objects for data storage, wrangling, and analysis. Even import my Excel data and rapidly learn the native function cheat sheet so fast that I was excited to learn what packages are because I couldn't wait to see what I could do. That was my experience in like 2010, maybe, and after having C++ and Python go in and out my head during college multiple times. I view R as simple only because I actually felt more helpless to keep learning it than helpless to ever learn coding at all. Worth noting that I was a Stat/Probability tutor with a Finance degree and much Excel experience.
remarkEon 2y ago
>The code ChatGPT generates is often bad in ways that are hard to detect. If you are not an experienced software engineer, the defects could be impossible to detect, until you/ChatGPT has gone and exposed all your customers to bad actors, or crash at runtime, or do something terribly incorrect. I wonder about this a lot, because there's a future here where a decent amount of software engineering is offloaded to these AIs and we reach a point, in the near future, where no one really knows or understands what's going on. That seems bad. Put another way, suppose that your primary care doctor is really just using MedAI to diagnose and recommend treatment for whatever it is you went in to see him about. Over time, these sorts of shortcuts metastasize and the doctor ends up not really knowing anything about you, or the other patients, or what he's really doing as a doctor ... it's just MedAI (with whatever wrongness rate is tolerable for the insurance adjusters). Again, seems bad. There's a palpable loss of human knowledge here that's enabled by a "tool" that's allegedly going to make us all better off.
- disgruntledphd2 2y ago
  The closest analogy here is that we don't have as full-featured autopilots in airplanes as we could, because they reduce safety.
  - remarkEon 2y ago
    Right, good point. Maybe I'm making an argument that some features, or scope of features, should be highly regulated along the same lines.

TeMPOraL 2y ago

My experience actually agrees with you. It's just that the set of use cases that either: - Are hard (or boring) to do, but easy to evaluate - for me, e.g. writing code, OCR, ideation; or - Don't require a perfectly correct answer, but more of a starting point or map of the problem space; or - Are very subjective, or creative, with there being no single correct answer, is surprisingly large. It covers pretty much everything, but not everything for everyone at the same time.

sumedh 2y ago

> The code ChatGPT generates is often bad in ways that are hard to detect. Does it work though, yes it does. There are many human coders who write bad code and life goes.

deleted 2y ago

[deleted]

ignoramous 2y ago

> Okay I guess I've just had a different experience entirely. I've seen both the good and the bad. I really like the good parts. Most recently, Claude Sonnet 3.5 fixed a math error in my code (I prompted it to check for it from a well-written bug report, and it did it fix it ever so perfectly). These days, it is pretty much second nature for me to pull up a new file & prompt Copilot to complete writing the entire code from my comment trails. I don't think I've seen as much change in my coding behaviour since Borland Turbo C -> NetBeans.

nvarsj 2y ago

I agree. I've just seen it hallucinate too many things that on the surface seem very plausible but are complete fabrications. Basically my trust is near 0 for anything chatgpt, etc. spits out. My latest challenge is dealing with people that trust chatgp to be infallible, and just quote the garbage to make themselves look like they know what they are talking about.

gwervc 2y ago
> things that on the surface seem very plausible but are complete fabrications LLMs are language model, it's crazy people expect them to be correct in anything beyond surface level language.
- nvarsj 2y ago
  Yeah, I was probably being a bit too harsh in my original comment. I do find them useful, you just have to be wary of the output.

itsoktocry 2y ago

If your procees is asking it to "write me all this code", then you slap it in production, you're going to have a bad time. But there's intermediate ground. >I am not consulting ChatGPT over, say, a dietician or doctor Do you know any doctors, by chance? You have way more faith in experts than I do.

kortilla 2y ago
ChatGPT is just statistically associating what it’s observed online. I wouldn’t take dietary advice from the mean output of Reddit with more trust than an expert.
- interstice 2y ago
  Doctors can be associating what they’ve learned, often with heavy biases from hypochondriacs and not enough time per patient to really consider the options. I’ve had multiple friends get seriously ill before a doctor took their symptoms seriously, and this is a country with decent healthcare by all accounts. Human biases are bad too.
  - beachy 2y ago
    > Doctors can be associating what they’ve learned, often with heavy biases from hypochondriacs So true. And it's hard to question a doctor's advice, because of their aura of authority, whereas it's easy to do further validation of an LLMs diagnosis. I had to change doctor recently when moving towns. It was only when chancing on a good doctor that I realised how bad my old doctor was - a nice guy but cruising to retirement. And my experience with cardiologists has been the same. Happy to get medical advice from an LLM though I'd certainly want prescriptions and action plans vetted by a human.
    - throwaway2037 2y ago
      > It was only when chancing on a good doctor that I realised how bad my old doctor was How did you determine the new doctor is "good"?

CuriouslyC 2y ago

Easy answer. Ask ChatGPT to write testable code, and tests for the code, then just verify the tests. If the tests don't work, have ChatGPT use the test output to rewrite the code until it does. If you can't have ChatGPT write testable code because of your architecture, you have other problems. People with bad process and bad architecture saying AI is bad because it doesn't work well with their dumpster fire systems, 100% facepalm.

aleph_minus_one 2y ago
> If you can't have ChatGPT write testable code because of your architecture, you have other problems. There exist lots of reasons why code is hard to test automatically that have nothing to do with the architecture of the code, but with the domain for which the code is written and runs.

ern 2y ago

I recently needed to help a downstream team with a problem with an Android app. I never did mobile app dev before, but I was able to spin up a POC (having not coded in Java for 22 years) and solve the problem with the help of ChatGPT 4.0. Sure I probably would have been able to do it without ChatGPT, but it was so much easier to have something to bounce ideas off-of. A safety net, if you will. The hallucination risk was irrelevant: it did hallucinate a little early on. I told it it was a hallucinating, and we moved onto a different way of solving the problem. It was easy enough to verify it was working as expected.

gexla 2y ago
Seems to me this is the equivalent of fast retrieval and piecing together from a huge amount of examples in the data. This might take far more time if you were to do this yourself. That's a plus for the tools. In other words, a massively expensive (for the service provider) auto-complete. But try to do something much more simple but has much fewer examples (a typical case is something which has bad documentation) in the data, and it falls apart. I even tried to use Perplexity to create a dead simple CLI command, and it hallucinated an answer (looking at the docs, it misused the parameter, and may have picked up on someone who gave an incorrect answer in the data.)

dmix 2y ago

It's already gotten significantly better and faster in a few yrs. Maybe LLMs will hit a wall in the next 5yrs but even if it does it's still extremely useful and there are always other ways to optimize the current technology where this is already a major development for society.

jakderrida 2y ago

>The code ChatGPT generates is often bad in ways that are hard to detect. If you are not an experienced software engineer, the defects could be impossible to detect I keep hearing this, but it's incorrect. While I only know R, which is obviously a simple language, I would never type out all my code and go without testing to ensure it does what I intended before using it regularly. So I can't imagine someone that knows a more complex language just typing out all of it before integrating it into business systems at their work or anything else before testing it. Why would AI be any different? Why the hell are AI skeptics acting like getting help from an LLM would involve not testing anything? Of course I test it! Why on earth wouldn't I? Just as I tested code made by freelancers I hired on commission before using the code I bought from them. Do AI skeptics really not test their own code? Are you all insane?

disgruntledphd2 2y ago
> While I only know R, which is obviously a simple language Take it from someone who started with R, R is 100% not a simple language. If you can write good R, you're probably a surprisingly good potential SE as R is kinda insane and inconsistent due to 50+ years of history (from S, to R etc).
- jakderrida 2y ago
  Hmmm.. I'm trying to imagine interviewing for SE and telling them I got wealthy from a crypto market-making algorithm I coded in R during Covid and the interviewer responding with anything but laughter or with silence as they ponder legal ways to question my mental health. It's an excellent language, I think, for many reasons. One is that you can work with data within hours because even before learning what packages or classes are, you got native objects for data storage, wrangling, and analysis. Even import my Excel data and rapidly learn the native function cheat sheet so fast that I was excited to learn what packages are because I couldn't wait to see what I could do. That was my experience in like 2010, maybe, and after having C++ and Python go in and out my head during college multiple times. I view R as simple only because I actually felt more helpless to keep learning it than helpless to ever learn coding at all. Worth noting that I was a Stat/Probability tutor with a Finance degree and much Excel experience.

remarkEon 2y ago

>The code ChatGPT generates is often bad in ways that are hard to detect. If you are not an experienced software engineer, the defects could be impossible to detect, until you/ChatGPT has gone and exposed all your customers to bad actors, or crash at runtime, or do something terribly incorrect. I wonder about this a lot, because there's a future here where a decent amount of software engineering is offloaded to these AIs and we reach a point, in the near future, where no one really knows or understands what's going on. That seems bad. Put another way, suppose that your primary care doctor is really just using MedAI to diagnose and recommend treatment for whatever it is you went in to see him about. Over time, these sorts of shortcuts metastasize and the doctor ends up not really knowing anything about you, or the other patients, or what he's really doing as a doctor ... it's just MedAI (with whatever wrongness rate is tolerable for the insurance adjusters). Again, seems bad. There's a palpable loss of human knowledge here that's enabled by a "tool" that's allegedly going to make us all better off.

disgruntledphd2 2y ago
The closest analogy here is that we don't have as full-featured autopilots in airplanes as we could, because they reduce safety.
- remarkEon 2y ago
  Right, good point. Maybe I'm making an argument that some features, or scope of features, should be highly regulated along the same lines.