6 ms·
From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
I recently worked on running a thorough healthcare eval on GPT-5. The results show a (slight) regression in GPT-5 performance compared to GPT-4 era models.
I found this to be an interesting finding. Here are the detailed results: https://www.fertrevino.com/docs/gpt5_medhelm.pdf https://www.fertrevino.com/docs/gpt5_medhelm.pdf
- ancorevard 1y agoso since reasoning_effort is not discussed anywhere, I assume you used the default which is "medium"?
- energy123 1y agoAlso, were tool calls allowed? The point of reasoning models is to delete the facts so finite capacity goes towards the dense reasoning engine rather than recall, with the facts sitting elsewhere.
- deleted 1y ago[deleted]
- woeirua 1y agoInteresting topic, but I'm not opening a PDF from some random website. Post a summary of the paper or the key findings here first.
- BoredPositron 1y agoIt's hacker news. You can handle a PDF.
- jeffbee 1y agoI approve of this level of paranoia, but I would just like to know why PDFs are dangerous (reasonable) but HTML is not (inconsistent).
- HeatrayEnjoyer 1y agoPDFs can run almost anything and have an attack surface the size of Greece's coast.
- zamadatix 1y agoThat's not very different than web browsers, but usually security concerned people just disable scripting functionality and such in their viewer (browser, pdf reader, rtf viewer, etc) instead of focusing on the file extension it comes in. I think pdf.js even defaults to not running scripts in PDFs by default (would need to double check), if you want to view it in the browser's sandbox. Of course there's still always text rendering based security attacks and such but, again, there's nothing unique to that vs a webpage in a browser.
- hypoxia 1y agoDid you try it with high reasoning effort?
- ares623 1y agoSorry, not directed at you specifically. But every time I see questions like this I can’t help but rephrase in my head: “Did you try running it over and over until you got the results you wanted?”
- chairmansteve 1y agoOr... "Did you try a room full of chimpanzees with typewriters?"
- SequoiaHope 1y agoWhat you describe is a person selecting the best results, but if you can get better results one shot with that option enabled, it’s worth testing and reporting results.
- ares623 1y agoI get that. But then if that option doesn't help, what I've seen is that the next followup is inevitably "have you tried doing/prompting x instead of y"
- theshackleford 1y ago> I get that. But then if that option doesn't help, what I've seen is that the next followup is inevitably "have you tried doing/prompting x instead of y" Maybe I’m misunderstanding, but it sounds like you’re framing a completely normal proces (try, fail, adjust) as if it’s unreasonable? In reality, when something doesn’t work, it would seem to me that the obvious next step is to adapt and try again. This does not seem like a radical approach but instead seems to largely be how problem solving sort of works? For example, when I was a kid trying to push start my motorcycle, it wouldn’t fire no matter what I did. Someone suggested a simple tweak, try a different gear. I did, and instantly the bike roared to life. What I was doing wasn’t wrong, it just needed a slight adjustment to get the result I was after.
- xnx 1y agoHave you looked at comparing to Google's foundation models or specialty medical models like MedGemma (https://developers.google.com/health-ai-developer-foundations/medgemma https://developers.google.com/health-ai-developer-foundation...)?
- fertrevino 1y agoThat would be an interesting extension. MedGemma isn't part of the original benchmark either [1]. Since Gemini 2.0 Flash is on 6th place, expectations are for MedGemma to achieve higher than that :) [1]https://crfm.stanford.edu/helm/medhelm/latest/#/leaderboard https://crfm.stanford.edu/helm/medhelm/latest/#/leaderboard
- username135 1y agoI wonder what changed with the models that created regression?
- teaearlgraycold 1y agoNot sure but with each release it feels like they’re just wiping the dirt around and not actually cleaning.
- degamad 1y agoObligxkcd: https://xkcd.com/1838/ https://xkcd.com/1838/
- fertrevino 1y agoloved the cartoon :)
- oezi 1y agoThere is some speculation that GPT-5 uses a router to decide which expert model to deploy (e.g. to mini vs o/thinking models). So the router might decide that the query can be solved by a cheaper model and this model gives worse results.
- deleted 1y ago[deleted]
- aresant 1y agoFeels like a mixed bag vs regression? eg - GPT-5 beats GPT-4 on factual recall + reasoning (HeadQA, Medbullets, MedCalc). But then slips on structured queries (EHRSQL), fairness (RaceBias), evidence QA (PubMedQA). Hallucination resistance better but only modestly. Latency seems uneven (maybe more testing?) faster on long tasks, slower on short ones.
- woeirua 1y agoDefinitely seems like GPT5 is a very incremental improvement. Not what you’d expect if AGI were imminent.
- p1esk 1y agoWhat would you expect?
- TrainedMonkey 1y agoGPT-5 feels like cost engineering. The model is incrementally better, but they are optimizing for least amount of compute. I am guessing investors love that.
- narrator 1y agoI agree. I have found GPT-5 significantly worse on medical queries. It feels like it skips important details and is much worse than o3, IMHO. I have heard good things about GPT-5 Pro, but that's not cheap. I wonder if part of the degraded performance is where they think you're going into a dangerous area and they get more and more vague, for example like they demoed on launch day with the fireworks example. It gets very vague when talking about non-abusable prescription drugs for example. I wonder if that sort of nerfing gradient is affecting medical queries. After seeing some painfully bad results, I'm currently using Grok4 for medical queries with a lot of success.
- fertrevino 1y agoInteresting, it seems the anecdotal experience agrees with the benchmark results.
- deleted 1y ago[deleted]
- credit_guy 1y agoHere's my experience: for some coding tasks where GPT 4.1, Claude Sonnet 4, Gemini 2.5 Pro were just spinning for hours and hours and getting nowhere, GPT 5 just did the job without a fuss. So, I switched immediately to GPT 5, and never looked back. Or at least I never looked back until I found out that my company has some Copilot limits for premium models and I blew through the limit. So now I keep my context small, use GPT 5 mini when possible, and when it's not working I move to the full GPT 5. Strangely, it feels like GPT 5 mini can corrupt the full GPT 5, so sometimes I need to go back to Sonnet 4 to get unstuck. To each their own, but I consider GPT 5 a fairly bit move forward in the space of coding assistants.
- czk 1y agoits possible to use gpt-5-high on the plus plan with codex-cli, its a whole different beast! i dont think theres any other way for plus users to leverage gpt-5 with high reasoning. codex -m gpt-5 model_reasoning_effort="high"
- benlc 1y agoInterestingly I'm experiencing the opposite as you. Was mostly using Claude Sonnet 4 and GPT 4.1 through copilot for a few months and was overall fairly satisfied with it. First task I threw at GPT 5, it excelled in a fraction of the time Sonnet 4 normally takes, but after a few iterations, it all went downhill. GPT 5 almost systematically does things I didn't ask it to do. After failing to solve an issue for almost an hour, I switched back to Claude which fixed it in the first try. YMMV
- AndyNemmity 1y agoYeah, GPT 5 got into death loops faster than any other LLM, and I stopped using it for anything more than UI prototypes.
- agos 1y agoany thread on HN about AI (there's constantly at least one in homepage nowadays) goes like this: "in my experience [x model] one shots everything and [y model] stumbles and fumbles like a drunkard", for _any_ combination of X and Y. I get the idea of sharing what's working and what's not, but at this point it's clear that there are more factors to using these with success and it's hard to replicate other people's successful workflows.
- kumarvvr 1y agoI have an issue with the words "understanding", "reasoning", etc when talking about LLMs. Are they really understanding, or putting out a stream of probabilities?
- sema4hacker 1y agoThe latter. When "understand", "reason", "think", "feel", "believe", and any of a long list of similar words are in any title, it immediately makes me think the author already drank the kool aid.
- manveerc 1y agoIn the context of coding agents, they do simulate “reasoning” when you feed them the output and it is able to correct itself.
- qwertytyyuu 1y agoI agree with “feel” and “believe” but what words would you suggest instead of “understand” and “reason’?
- sema4hacker 1y agoNone. Don't anthropomorphize at all. Note that "understanding" has now been removed from the HN title but not the linked pdf.
- platypii 1y agoWhy not? We are trying to evaluate AI's capabilities. It's OBVIOUS that we should compare it to our only prior example of intelligence -- humans. Saying we shouldn't compare or anthropomorphize machine is a ridiculous hill to die on.
- sema4hacker 1y agoIf you are comparing the performance of a computer program with the performance of a human, then using terms implying they both "understand" wrongly implies they work in the same human-like way, and that ends up misleading lots of people, especially those who have no idea (understanding!) how these models work. Great for marketing, though.
- causality0 1y agoI've definitely seen some unexpected behavior from gpt5. For example, it will tell me my query is banned and then give me a full answer anyway.
- deleted 1y ago[deleted]
- CuriouslyC 1y agoGPT-5 is like an autistic savant
- mattwad 1y agoi thought cursor was getting really bad, then i found out i was on a gpt 5 trial. gonna stick with claude :)
- 0xDEAFBEAD 1y agoSo which of these benchmarks are most relevant for an ordinary user who wants to talk to AI about their health issues? I'm guessing HeadQA, Medbullets, MedHallu, and perhaps PubMedQA? (Seems to me that "unsupported speculation" could be a good thing for a patient who has yet to receive a diagnosis...) Maybe in practice it's better to look at RAG benchmarks, since a lot of AI tools will search online for information before giving you an answer anyways? (Memorization of info would matter less in that scenario)
- andai 1y agoDid this use reasoning or not? GPT-5 with Minimal reasoning does roughly the same as 4o on benchmarks.