6 ms·

There is a skill to it. You can get lucky as a beginner but if you want consistent success you gotta learn the ropes (strengths, weaknesses, failure modes etc).

There is a skill to it. You can get lucky as a beginner but if you want consistent success you gotta learn the ropes (strengths, weaknesses, failure modes etc). A quick way of getting seriously improved results though: if you are literally using GPT-4 as you mention—that is an ancient model! Parent comment says GPT-4.1 (yes openai is unimaginably horrible at naming but that ".1" isn't a minor version increment). And even though 4.1 is far better, I would never use it for real work. Use the strongest models; if you want to stick with openai use o3 (it's now super cheapt too). Gemini 2.5 Pro is roughly equivalent to o3 for another option. IMO Claude models are stronger in agentic setting, but won't match o3 or gemini 2.5 pro for deep problem solving or nice, "thought out" code.

bradly 1y ago
Specific model I was using was o4-mini-high which the drop-down model selector describes as "Great at coding and visual reasoning".
- westoncb 1y ago
  Gotcha. Yeah, give o3 a try. If you don't want to get a sub, you can use it over the api for pennies. They do have you do this biometric registration thing that's kind of annoying if you want to use over api though. You can get the Google pro subscription (forget what they call it) that's ordinarily $20/mo for free right now (1 month free; can cancel whenever), which gives unlimited Gemini 2.5 Pro access.
  - slacktivism123 1y ago
    Yeah, this model didn't work it seems. You're holding it wrong. You need to utter the right series of incantations to get some semblance of truth. What, you used the model that was SOTA one week ago? Big mistake, that explains why. You need to use this SOTA model that came out one day ago instead. That model definitely wasn't trained to overfit the week-old benchmarks and dismiss the naysayers. Look, a pelican! What? You haven't verified your phone number and completed a video facial scan and passed a background check? You're NGMI.
    - Hackbraten 1y ago
      > Look, a pelican! Love this reference :)
  - bradly 1y ago
    Thank you for the tip on o3. I will switch to that and see how it goes. I do have a paid sub for ChatGPT, but from the dropdown model descriptions "Great at coding" sounded better than "Advanced reasoning". And 4 is like almost twice as much as 3.
    - westoncb 1y ago
      lol yep, fully get that. And I mean I'm sure o4 will be great but the '-mini' variant is weaker. Some of it will come down to taste and what kind of thing you're working on too but personal preferences aside, from the heavy LLM users I talk to o3 and gemini 2.5 pro at the moment seem to be top if you're dialoging with them directly (vs using through an agent system).
    - thom 1y ago
      I’d also recommend basically always having search enabled. That’s eliminated major hallucinations for me.
    - TeMPOraL 1y ago
      In my current experience: - o3 is the bestest and my go-to, but its strength comes from it combining reasoning with search - it's the one model you can count on finding things out for you instead of going off vibe and training data; - GPT 4.5 feels the smartest, but also has tight usage limits and doesn't do search like o3 does; I use it when I need something creative done, or switch to it mid-conversation to have it reason off an already primed context; - o4-mini / o4-mini-hard - data transformation, coding stuff that doesn't require looking things up - especially when o3 looked stuff up already, and now I just need ChatGPT to apply it into code/diagrams; - gpt-4o - only for image generation, and begrudgingly when I run out of quota on GPT 4.5 o3 has been my default starting model for months now; most of my queries generally benefit from having a model that does autonomous reasoning+search. Agentic coding stuff, that I push to Claude Code now.
      - agos 1y ago
        the fact that one needs to know stuff like this and that it changes every three months seriously limits the usefulness of LLMs for me
  - andrepd 1y ago
    > Gotcha. Yeah, give o3 a try. If you don't want to get a sub, you can use it over the api for pennies. They do have you do this biometric registration thing that's kind of annoying if you want to use over api though. I hope you appreciate just how crazy this sentence sounds, even in an age when this is normalised.
    - westoncb 1y ago
      Yep, it's surreal.
- TeMPOraL 1y ago
  I'm curious how you ended up in such a conversation in the first place. Hallucinations are one thing, but I can't remember the last time when the model was saying that it actually run something somewhere that wasn't a tool use call, or that it owns a laptop, or such - except when role-playing. I wonder if the advice on prompting models to role play isn't backfiring now, especially in conversational setting. Might even be a difference between "you are an AI assistant that's an expert programmer" vs. "you are an expert programmer" in the prompt, the latter pushing it towards "role-playing a human" region of the latent space. (But also yeah, o3. Search access is the key to cutting down on amount of guessing the answers, and o3 is using it judiciously. It's the only model I use for "chat" when the topic requires any kind of knowledge that's niche or current, because it's the only model I see can reliably figure out when and what to search for, and do it iteratively.)
  - bradly 1y ago
    What is really interesting is in the "thinking" section it said "I need to reassure the user..." so my intuition is that it thought it was right, but did not think I would think they were right, but if they just gave me the confidence, I would try the code and unblock myself. Maybe it thought this was the best % chance I would listen to it and so it is the correct response?
    - TeMPOraL 1y ago
      Maybe? Depends on what followed that thought process. I've noticed this couple times with o3, too - early on, I'd catch a glimpse of something like "The user is asking X... I should reassure them that Y is correct" or such, which raised an eyebrow because I already know Y was bullshit and WTF with the whole reassuring business... but then the model would continue actually exploring the question and the final answer showed no trace of Y, or any kind of measurement. I really wish OpenAI gave us the whole thought process verbatim, as I'm kind of curious where those "thoughts" come from and what happens to them.
      - bradly 1y ago
        Ehh... I did ask it if it would be able to figure this out or if I should try another model :|
- msgodel 1y ago
  All LLMs can fail this way. It's kind of weird to see people running into this kind of issue with modern large models with all the RL and getting confused. No one starting today seems to have good intuition for them. One person I knew insisted LLMs could do structural analysis for months until he saw some completely absurd output from one. This used to be super common with small GPTs from around 2022 and so everyone just intuitively knew to watch out for it.
deleted 1y ago
[deleted]
andrepd 1y ago
Literally astrology at this point. We don't understand the black box bs generating machine, but actually if you prod it this and that way according to some vague vibe, then it yields results that even if wrong are enough to fool you. And christ, every single time there's the same retort: "ah but of course your results are shit, you must not be using gpt-4.69-o7-turbo-pro which came out this morning". Come on...
- danielbln 1y ago
  You sit at the opposite of the spectrum, refusing with all your might that there might be something useful there at all. It's all just a BS generator that nothing, nothing at all useful can come out of, right? You might think you are a staunch critic and realist that no hype can touch and you see through all of it, when in fact your are wilfully ignorant.
  - brahma-dev 1y ago
    Here's some BS for you. That's an unfair mischaracterization of their position. Criticism doesn't equal rejection, and skepticism isn't the same as ignorance. Pointing out limitations, failures, or hype doesn't mean they are claiming there's nothing useful or that the entire technology is inherently worthless. Being critical is not about denying all value—it’s about demanding evidence, accuracy, and clarity amid inflated claims. In fact, responsible critique helps improve technology by identifying where it falls short, so it can evolve into something genuinely useful and reliable. What you're calling "willful ignorance" is, in reality, a refusal to blindly accept marketing narratives or inflated expectations. That’s not being closed-minded—that’s being discerning. If there is something truly valuable, it will stand up to scrutiny.
  - andrepd 1y ago
    > refusing with all your might that there might be something useful there at all How does this follow from what I wrote? I addressed two very concrete points.

bradly 1y ago

Specific model I was using was o4-mini-high which the drop-down model selector describes as "Great at coding and visual reasoning".

westoncb 1y ago
Gotcha. Yeah, give o3 a try. If you don't want to get a sub, you can use it over the api for pennies. They do have you do this biometric registration thing that's kind of annoying if you want to use over api though. You can get the Google pro subscription (forget what they call it) that's ordinarily $20/mo for free right now (1 month free; can cancel whenever), which gives unlimited Gemini 2.5 Pro access.
- slacktivism123 1y ago
  Yeah, this model didn't work it seems. You're holding it wrong. You need to utter the right series of incantations to get some semblance of truth. What, you used the model that was SOTA one week ago? Big mistake, that explains why. You need to use this SOTA model that came out one day ago instead. That model definitely wasn't trained to overfit the week-old benchmarks and dismiss the naysayers. Look, a pelican! What? You haven't verified your phone number and completed a video facial scan and passed a background check? You're NGMI.
  - Hackbraten 1y ago
    > Look, a pelican! Love this reference :)
- bradly 1y ago
  Thank you for the tip on o3. I will switch to that and see how it goes. I do have a paid sub for ChatGPT, but from the dropdown model descriptions "Great at coding" sounded better than "Advanced reasoning". And 4 is like almost twice as much as 3.
  - westoncb 1y ago
    lol yep, fully get that. And I mean I'm sure o4 will be great but the '-mini' variant is weaker. Some of it will come down to taste and what kind of thing you're working on too but personal preferences aside, from the heavy LLM users I talk to o3 and gemini 2.5 pro at the moment seem to be top if you're dialoging with them directly (vs using through an agent system).
  - thom 1y ago
    I’d also recommend basically always having search enabled. That’s eliminated major hallucinations for me.
  - TeMPOraL 1y ago
    In my current experience: - o3 is the bestest and my go-to, but its strength comes from it combining reasoning with search - it's the one model you can count on finding things out for you instead of going off vibe and training data; - GPT 4.5 feels the smartest, but also has tight usage limits and doesn't do search like o3 does; I use it when I need something creative done, or switch to it mid-conversation to have it reason off an already primed context; - o4-mini / o4-mini-hard - data transformation, coding stuff that doesn't require looking things up - especially when o3 looked stuff up already, and now I just need ChatGPT to apply it into code/diagrams; - gpt-4o - only for image generation, and begrudgingly when I run out of quota on GPT 4.5 o3 has been my default starting model for months now; most of my queries generally benefit from having a model that does autonomous reasoning+search. Agentic coding stuff, that I push to Claude Code now.
    - agos 1y ago
      the fact that one needs to know stuff like this and that it changes every three months seriously limits the usefulness of LLMs for me
- andrepd 1y ago
  > Gotcha. Yeah, give o3 a try. If you don't want to get a sub, you can use it over the api for pennies. They do have you do this biometric registration thing that's kind of annoying if you want to use over api though. I hope you appreciate just how crazy this sentence sounds, even in an age when this is normalised.
  - westoncb 1y ago
    Yep, it's surreal.
TeMPOraL 1y ago
I'm curious how you ended up in such a conversation in the first place. Hallucinations are one thing, but I can't remember the last time when the model was saying that it actually run something somewhere that wasn't a tool use call, or that it owns a laptop, or such - except when role-playing. I wonder if the advice on prompting models to role play isn't backfiring now, especially in conversational setting. Might even be a difference between "you are an AI assistant that's an expert programmer" vs. "you are an expert programmer" in the prompt, the latter pushing it towards "role-playing a human" region of the latent space. (But also yeah, o3. Search access is the key to cutting down on amount of guessing the answers, and o3 is using it judiciously. It's the only model I use for "chat" when the topic requires any kind of knowledge that's niche or current, because it's the only model I see can reliably figure out when and what to search for, and do it iteratively.)
- bradly 1y ago
  What is really interesting is in the "thinking" section it said "I need to reassure the user..." so my intuition is that it thought it was right, but did not think I would think they were right, but if they just gave me the confidence, I would try the code and unblock myself. Maybe it thought this was the best % chance I would listen to it and so it is the correct response?
  - TeMPOraL 1y ago
    Maybe? Depends on what followed that thought process. I've noticed this couple times with o3, too - early on, I'd catch a glimpse of something like "The user is asking X... I should reassure them that Y is correct" or such, which raised an eyebrow because I already know Y was bullshit and WTF with the whole reassuring business... but then the model would continue actually exploring the question and the final answer showed no trace of Y, or any kind of measurement. I really wish OpenAI gave us the whole thought process verbatim, as I'm kind of curious where those "thoughts" come from and what happens to them.
    - bradly 1y ago
      Ehh... I did ask it if it would be able to figure this out or if I should try another model :|
msgodel 1y ago
All LLMs can fail this way. It's kind of weird to see people running into this kind of issue with modern large models with all the RL and getting confused. No one starting today seems to have good intuition for them. One person I knew insisted LLMs could do structural analysis for months until he saw some completely absurd output from one. This used to be super common with small GPTs from around 2022 and so everyone just intuitively knew to watch out for it.

deleted 1y ago

[deleted]

andrepd 1y ago

Literally astrology at this point. We don't understand the black box bs generating machine, but actually if you prod it this and that way according to some vague vibe, then it yields results that even if wrong are enough to fool you. And christ, every single time there's the same retort: "ah but of course your results are shit, you must not be using gpt-4.69-o7-turbo-pro which came out this morning". Come on...

danielbln 1y ago
You sit at the opposite of the spectrum, refusing with all your might that there might be something useful there at all. It's all just a BS generator that nothing, nothing at all useful can come out of, right? You might think you are a staunch critic and realist that no hype can touch and you see through all of it, when in fact your are wilfully ignorant.
- brahma-dev 1y ago
  Here's some BS for you. That's an unfair mischaracterization of their position. Criticism doesn't equal rejection, and skepticism isn't the same as ignorance. Pointing out limitations, failures, or hype doesn't mean they are claiming there's nothing useful or that the entire technology is inherently worthless. Being critical is not about denying all value—it’s about demanding evidence, accuracy, and clarity amid inflated claims. In fact, responsible critique helps improve technology by identifying where it falls short, so it can evolve into something genuinely useful and reliable. What you're calling "willful ignorance" is, in reality, a refusal to blindly accept marketing narratives or inflated expectations. That’s not being closed-minded—that’s being discerning. If there is something truly valuable, it will stand up to scrutiny.
- andrepd 1y ago
  > refusing with all your might that there might be something useful there at all How does this follow from what I wrote? I addressed two very concrete points.