32 ms·
O3 Turns Pro
- iLoveOncall 1y ago> My experience so far is that waiting a long time is annoying, sufficiently annoying that you often won’t want to wait. My solution for this has been to use non-reasoning models, and so far in 90% of the situations I have received the exact same results from both.
- jasonjmcghee 1y agoOn the complete other end of the spectrum, I found deep research (whether it's actually performing searches or not) to be a significant upgrade in quality. But you need to be cool with having to wait 15-30 minutes. It's certainly not for everything, but definitely worth trying. It tends to output significantly longer and more detailed output. So when you want that kind of thing- works well. Especially if you need up to date stuff or want to find related sources.
- pinoy420 1y ago[dead]
- joshstrange 1y agoDeep research is very cool, no doubt, but run it on a problem space you are familiar with and you will see the shortcomings. Anytime I do my own “deep” research I like to then throw the same problem at OpenAI and see how well it fares. Often it misses things or gets things subtly wrong. The results look impressive so it’s easy to fool people and I’m not saying the results are useless, I’ve absolutely gotten value out of it, but I don’t love using it for anything I actually care about.
- bcrosby95 1y agoI view the results more as a starting point than an end unto itself. For that I think it's pretty useful.
- matwood 1y agoSame, it will pull enough sources together that I end up with an idea of where to go next.
- joshstrange 1y agoAbsolutely, I agree it's useful as a starting point, sometimes it's all I need (if it's low-stakes and I just wanted a bit more data). I was just cautioning "trusting" it completely, since it's very easy to fall into that trap (I've done it).
- jacobsenscott 1y ago[flagged]
- crubier 1y agoWriting a Pull Request can take me 8 hours. Reviewing a Pull Request of the same size takes me 30min. Here you go.
- Y_Y 1y agoP ⊆ NP
- Nition 1y agoYeah this is the big benefit of current LLM AI even with the mistakes and hallucinations IMO. All the things that are hard to answer but easy to verify. Not just programming. e.g. You have a complex medical problem. Hard to ask Google. Ask AI, it gives you some possible answers, then you can search up those. Or you want to identify a plant. AI looks at your photo and tells you what it is, then search that name to verify. There were existing ways to do some of these things, but this covers all of them.
- add-sub-mul-div 1y agoThe majority of people just want to go home at 5 after putting in as little effort as possible. Their bosses just want to save money in the short term. The interests could not be more aligned and optimized.
- huxley 1y agoNot necessarily, you don’t need to know the answer, the fabulation might: * give an error * return the wrong result * not be internally consistent with the rest of the content * be logically impossible * be factually impossible * have basic errors It is entirely possible (and quite common) to know something is wrong without knowing what a right answer is.
- bananapub 1y ago
- vessenes 1y agoI'm using Pro. It's definitely a "hand it to the team and have them schedule a meeting to get back to me" speed tool. But, it "feels" better to me than o3, and significantly better than gemini/claude for that use case. I do trust it more on confabulations; my current trust hierarchy would be o3-pro -> o3 -> gemini -> claude opus -> (a bunch of stuff) -> 4o. That said, I'd like this quality with a relatively quick tool using model; I'm not sure what else I'd want to call it "AGI" at that point.
- bananapub 1y agowhat do you trust it to do? the only example uses I see written about on HN appear to basically be Substack users asking o3 marketing questions and then writing substack posts about it, and a smattering of vague posts about debugging.
- vessenes 1y agoLong form research reporting. Example: Pull together a list of the top 20 startups funded in Germany this year, valuation, founder and business model. Estimate which is most likely to want to take on private equity investment from a lower mid market US PE fund, as well as which would be most suitable taking into consideration their business model, founders and market; write an approach letter in english and in german aimed at getting a meeting. make sure that it's culturally appropriate for german startup founders. I have no idea what the output of this query would be by the way, but it's one I would trust to get right on * the list of startups * the letter and its cultural sensitivity * broad strokes of what the startup is doing Stuff I'd "trust but verify" would be * Names of the founders * Size of company and target market Stuff I'd double check / keep my own counsel on * Suitability and why (note that o3 pro is def. better at this than o3 which is already not bad; it has some genuinely novel and good ideas, but often misses things.)
- leptons 1y agoThis is all stuff I would expect an LLM to "hallucinate" about. Every bit of it.
- snissn 1y agoI’ve found throw the problem at 3 o3 pros and have another one evaluate and synthesize works really well
- ActionHank 1y agoSo like, a whole forest of trees per query is what we're saying here?
- LeafItAlone 1y agoIdeally just a few split atoms
- kridsdale1 1y agoNow You’re Playing With Agent Power!
- franze 1y agoI use Claude Code a lot. A lot lot. I make it do Atomic Git commits for me. When it gets stuck and instead of just saying so starts to refactor half of the codebase, I jump back to commit where the issue first appeared and get a summary of the involved files. Those in full text (not files) into o3 pro. And you can be sure it finds the issue or gives a direction where the issue does not appear. Would love o3-pro as am MCP so whenever Claude Code goes on a "lets refactor everything" coding spree it just asks o3 pro.
- jgalt212 1y ago> When it gets stuck and instead of just saying so starts to refactor half of the codebase That's pretty scary.
- franze 1y agoAtomic Commits. I put this into Claude.md and need to remind it every other hour. But yeah, you need to jump back every few hours or so.
- nevertoolate 1y agoCan you give an example what claude works on autonomously for hours? I only use the chat, maybe I’m just not prompting well, but I throw away almost everything claude writes and solve it in significantly less lines of code using the proper abstractions.
- ActionHank 1y agoYeah, so far, I've only seen cases where the work is extremely simple and using pervasively used libraries and solutions to create widely implemented solutions. Add something a little out there and things start to unravel.
- franze 1y agocurrently i am coding a node/react/ts firebase app that allows dynamic multiagent workflows to automate content workflows (a workflow.json defines call this model and the pass this part of the output of that model to that model and then combine it with this model to do that) my setup is claude code in yolo mode with playwright MCP + browser MCP (to do stuff in the logged i firebase web interface) plus search enabled. the prototype was developed via firebase studio until i reached a dead end there, then i used claude code to rip out firebase genkit and hooked in google-genai, openai, ... the whole codebase goes into google gemini studio (caus the million token window) to write tickets, more tickets and even more tickets. claude code then has the job to implemt these tickets (create a detailed tasklist for each ticket first) and then code it until done. end of each tasklist is a working playwright end to end test with verified output. and atomic commits. i hooked anydesk to my computer so i can check i at some point to tell to to continue or to read Claude.md again (the meta instructions which basically tells it to not to fallbacks, mock data or cheat in amy other way.) ever fourth ticket is refactoring for sinplicity and documentation. the tickets mist be updated before each commit and moved to the do done folder only when 100 tested ok. so yeah, when i wale up in the morning either magic happend and the tockets are all done. or it got stuck and refactores half the codebase. in that case it works for an hoor to go over all git commits to find out where it went wrong. what i need are multiple coding agent which challenge each other at crucial points.
- A_D_E_P_T 1y agoChat just isn't the best format for something that takes 15-20 minutes (on average) to come up with a response. Email would unironically be better. Send a very long and detailed prompt, like a business email, and get a response back whenever it's ready. Then you can refine the prompt in another email, etc. But I should note that o3-pro has been getting faster for me lately. At first every damn thing, however simple, took 15+ minutes. Today I got a few answers back within 5 minutes.
- throw14082020 1y agoDon't you chat with someone over various apps, but then get a notification minutes, hours, or days later? Email is just an instance of chat? So I disagree. Are you building an AI agent accessible over email?
- A_D_E_P_T 1y agoHere's how I view it: Chat is more of a "conversational" medium, where you dash off short and informal messages and expect a very quick if not immediate response. Email is more of a "correspondence" medium, where you write longer and sometimes much more formal messages, and expect a response within ~24 hours. The latter would be the better way to use ultra-compute-heavy/long-delay models like o3-Pro, because you don't expect the flow of a conversation, and it would nudge you to be much more detailed in your prompt. In fact, if it had a larger context window, an email-format o3-Pro would be perfect for lawyers and for certain types of data analysis. I'm not building anything, but if you'd like to, lmk.
- starik36 1y agoI've tried o3 Pro for my use cases (parsing emails in the legal profession) and didn't have better results than the non pro. In fact, o1-preview has given me more consistently correct results than any other model. But it's being sunset next month so I have to move to o3.
- AaronAPU 1y agoIMO 4o is much better at people-parsing. The reasoning models o1-pro / o3-pro are really good at writing code and solving algorithmic problems.
- resters 1y agowhat is people parsing?
- starik36 1y agoThe email from the lawyer might mention lots of names. Who are the plaintiffs, who are defendants, their attorneys, assistants, or insurance adjusters. The model parses out who is who and connects names to titles to email addresses.
- resters 1y agoInteresting, that's what I thought it meant, but didn't realize it was a term of art.
- AaronAPU 1y agoThings like inferring the meaning of “people parsing” when it isn’t explicitly defined but can be implied by context. Not strict rational A+B=C, nuance.
- starik36 1y agoI've tried it with various models. And 4o is really good given that it returns data at least 10 times faster. But if you ask it to fill out a Json document, o3 (or other reasoning models) is still better, more correct and predictable. Or at least, better enough to justify waiting a minute for the API call to return vs 3-5 seconds.
- swyx 1y ago> Arena has gotten quite silly if treated as a comprehensive measure (as in Gemini 2.5 Flash is rated above o3) > The problem with o3-pro is that it is slow. well maybe Arena is not that silly then. poorly argued/organized article.
- jes5199 1y agoI haven’t tried pro yet but just yesterday I asked O3 to review a file and I saw a message in the chain-of-thought like “it’s going to be hard to give a comprehensive answer within the time limit” so now I’m tempted
- rotcev 1y agoI use O3-pro not as a coding model, but as a strategic assistant. For me, the long delay between responses makes the model unsuitable for coding workflows, however, it is actually a feature when it comes to getting answers to hard questions impacting my (or my friend's/family's) day to day life.
- b0a04gl 1y agowhen o3 pricing dropped 80%, most wrote the entire model family off as a downgrade (including me). but usage patterns flipped people finally ran real tasks through it. it's one of the few that holds state across fragmented prompts without collapsing context. used it to audit a messy auth flow spread over 6 services. didn't shortcut, didn't hallucinate edge cases. slow, but deliberate. in kahneman terms, it runs system 2 by default. many still benchmark on token speed, missing what actually matters
- lysecret 1y agoThis feels very Ai generated.
- mettamage 1y agoSome people write in similar ways yea. I've also been accused of writing as an AI. But we're still human mate. Stop discriminating or actually solve the problem. I've had enough of this attitude.
- cshimmin 1y agoalmost as though the AIs were trained on a corpus of text written by... humans
- b0a04gl 1y agoyes im agi by the way
- kridsdale1 1y agohi agi we’ve been trying so hard to find you
- motoxpro 1y agoI would say the opposite. Unless the person has a lot of custom instructions going on. Getting sentences like "but usage patterns flipped people finally ran real tasks through it." seem like it would take some amount of work.
- metalrain 1y ago"'take your profits’ in quality versus quantity is up to you." As mainly AI invester not AI user, I think profitability is great importance. It has been race to top so far, soon we see race to the bottom.
- resters 1y agoRight! We are in a sense lucky to be getting access to actual state-of-the-art models. Soon the actual model may be kept internal and the customers will get "good enough for solid ROI" distilled versions that can be hosted profitably.
- boole1854 1y agoHere are my own anecdotes from using o3-pro recently. My primary use cases where I am willing to wait 10-20 minutes for an answer from the "big slow" model (o3-pro) is code reviews of large amounts of code. I have been comparing results on this task from the three models above. Oddly, I see many cases where each model will surface issues that the other two miss. In previous months when running this test (e.g., Claude 3.7 Sonnet vs o1-pro vs earlier Gemini), that wasn't the case. Back then, the best model (o1-pro) would almost always find all the issues that the other models found. But now it seems they each have their own blindspots (although they are also all better than the previous generation of models). With that said, I am seeing Claude Opus 4 (w/extended thinking) be distinctly worse at missing problems which o3-pro and Gemini find. It seems fairly consistent that Opus will be the worst out of the three (despite sometimes noticing things the others do not). Whether o3-pro or Gemini 2.5 Pro is better is less clear. o3-pro will report more issues, but it also has a tendency to confabulate problems. My workflow involves providing the model with a diff of all changes, plus the full contents of the files that were changed. o3-pro seems to have a tendency to imagine and report problems in the files that were not provided to it. It also has an odd new failure mode, which is very consistent: it gets confused by the fact that I provide both the diff and the full file contents. It "sees" parts of the same code twice and will usually report that there has accidentally been some code duplicated. Base o3 does this as well. None of the other models get confused in that way, and I also do not remember seeing that failure mode with o1-pro. Nevertheless, it seems o3-pro can sometimes find real issues that Gemini 2.5 Pro and Opus 4 cannot more often than vice versa. Back in the o1-pro days, it was fairly straightforward in my testing for this use case that o1-pro was simply better across the board. Now with o3-pro compared particularly with Gemini 2.5 Pro, it's no longer clear whether the bonus of occasionally finding a problem that Gemini misses is worth the trouble of (1) waiting way longer for an answer and (2) sifting through more false positives. My other common code-related use case is actually writing code. Here, Claude Code (with Opus 4) is amazing and has replaced all my other use of coding models, including Cursor. I now code almost exclusively by peer programming with Claude Code, allowing it to be the code writer while I oversee and review. The OpenAI competitor to Claude Code, called Codex CLI, feels distinctly undercooked. It has a recurring problem where it seems to "forget" that it is an agent that needs to go ahead and edit files, and it will instead start to offer me suggestions about how I can make the change. It also hallucinates running commands on a regular basis (e.g., I tell it to commit the changes we've done, and outputs that it has done so, but it has not.) So where will I spend my $200 monthly model budget? Answer: Claude, for nearly unlimited use of Claude Code. For highly complex tasks, I switch to Gemini 2.5 Pro, which is still free in AI Studio. If I can wait 10+ minutes, I may hand it to o3-pro. But once my ChatGPT Pro subscription expires this month, I may either stop using o3-pro altogether, or I may occasionally use it as a second opinion by paying on-demand through the API.
- highfrequency 1y agoWhat is the difference between o3 pro and deep research? From a glance, both seem to take 10-15mins to respond and use o3 as the base model.
- thebiggening 1y agoI feel really sorry for anyone using o3. It is really, really bad...
- thebiggening 1y agonot being ironic I would rather commit seppuku than use o3