7 ms·
Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?
- couAUIA 2mo agoA deepdive on the /goal effect on a problem literally made for this.
- tyleo 2mo agoThe chart at the top is somewhat confusing. It says, “lower is better” but the y-axis is inverted! So visually higher in the chart is better but lower in terms of # value.
- o10449366 2mo ago/goal has replaced plan mode for me. This is the pattern I use for 95% of my AI work now: 1. Read X feature of Y and tell me when you fully understand it (if there's any detail missing in the summary, repeat until the context is primed) 2. What time is it? 3. /goal Spend X minutes from $time writing a technical design doc on $feature. There must not be any vague language or ambiguity in the document. Read carry_forward_requirements.md and testing_best_practices.md and explicitly incorporate them into the document you write. The document should be executable for a contextless implementer when done and include specific code and document references and changes needed. Spend the full X minutes working on and reviewing this document - do not quit early and wait Even just spending 10 minutes forcing GPT to write a design doc results in much more robust plans than plan mode, in my experience, and saves time I would spend iterating on the initial plan mode draft anyway.
- embedding-shape 2mo ago> 3. /goal Spend X minutes from $time writing a technical design doc on $feature. Hmm, I feel like this is akin to making a recursive function have a exit condition not based on what it actually did/found, but based on how long time it took. I'm always using /goal with explicit goals that the agent needs to achieve. Time-bounding them wouldn't make sense, I want something specific done regardless of how long time it takes. So instead I'd put goals on what the design/architecture needs to achieve, and for the model to continuously check the outcome against these, then finish when everything is achieved. Doesn't really matter if it takes 10 minutes or 10 hours, which for me is a bit the point of /goal in the first place, otherwise I'd just use the agent normally.
- illliillll 2mo agoWell, I’ve been having 5.6 sol work on tasks like “find every OTA app on the internet” I find explicit time bounds are useful for tasks like this, otherwise the LLM will almost certainly return too early.
- lonelyasacloud 2mo agoLeaving to run unhindered makes sense when the agents understand both the complete domain context and can be supplied with a strong exit criteria that they understand well. Otherwise time boxing is both going to help stop entities from wandering off into the weeds. And also communicate expectations from the commissioner about the expected effort levels and output quality requirements. Empirically in human world, get very different results when an employee, particularly a junior, is asked to spend 1/2 a day on a work package, a week or are left completely to their own devices.
- o10449366 2mo agoI think like the other commentator noted, I don't do this blindly, as I also wouldn't if I was instructing a junior engineer. I have a rough sense of how long it would take me to research all angles of the task. 5 minutes for a complicated feature is too short for a comprehensive plan. Letting it run for a few hours is also probably unproductive. But 10-60 minutes depending on complexity with followup review is a nice sweet spot. Just my $0.02.
- techpression 2mo agoI love that we have this on one hand and me cleaning up catastrophic CSS made by Sol on the other. Then again, maybe CSS is the ultimate benchmark.
- baq 2mo agoCSS is the reason I refuse to do any frontend work except FE infra and I know I’m not alone here, soooo yes I guess?
- cwmoore 2mo agoI do not know the whole picture, but if you are asking for blind one-shot CSS, you might benefit from wiring the model to take screenshots of various end-browsers and discuss them as you iterate. Offering freelance estimates for CSS design changes before frameworks were around was a problem.
- techpression 2mo agoI actually have very detailed specifications around the CSS, the problem is that LLM's are trained mostly on codebases with high amounts of complexity so they like to create layer of abstractions that only make your codebase much harder to reason about, machine and man alike. This is very visible in CSS, where almost every single thing automatically becomes a component wrapping the CSS (be it React or LiveView in my case), which is very often meaningless because the CSS itself is the component.
- thickclearglass 2mo agoCodex has a very bad system prompt which no one at OpenAI is seemingly aware of. Sol frontend is perfectly serviceable if you use it from something like Pi.
- andai 2mo agoResults seem mostly noise to me. One eval per model, in a large problem space (i.e. a problem which requires many attempts to solve well).
- couAUIA 2mo agoYes I agree, but I actually did a lot more runs, with different prompts, different times ect... And each time /goal had a small or insignificant impact
- tantalor 2mo agoWhat is /goal?
- swader999 2mo agoOn Claude if you start with that, it won't stop until it achieves or exhausts your prompt. It feels like "here's your mission, go do it". I use it a few times a week.
- jascha_eng 2mo agoIs this useful? I feel like the problem is usually not that the model isn't capable of achieving what I give it, but the way it does it. Especially if originally I didn't 100% know how I would do it myself the model often takes weird paths through the code base, takes shortcuts that end up in weird feature interactions or pulls in a dependency without weighting if it could've been done without that. I haven't really found a good way to solve this other than: 1. Produce an initial PR fulfilling all the requirements I knew at the start 2. Chat with the model about any weird snippets I notice and talk through alternatives 3. Simplify anything that I think is overengineered or plain unncessary Sometimes I restart all over with more precise requirements but then it sometimes makes different mistakes/takes different shortcuts. In practice the earlier I review the better the end result imo, so /goal seems very unproductive to me?
- swader999 2mo agoIt's useful for things where it just needs to get through to completion. Long running tasks. I walk away and expect it to be done without pausing for input.
- jascha_eng 2mo agoCan you give an example? And more curious about what you do with the resulting code afterwards I imagine its gonna be a big chunk then?
- nsoonhui 2mo ago[dead]
- Tenoke 2mo agoClaude seems to forget what you tell it in very long work sessions (things that take weeks to develop), no matter how many times you tell it which part is extra important. I dont use goal (I guess I should), but presumably it makes it actually remember the most important instruction. I believe this here is about shorter sessions where the issue doesn't crop up as much.
- jswny 2mo agoUnfortunately I’ve used Claude and GPT models for a long time in a variety of harnesses and I agree with you and I think it’s the compaction. For some reason, codex compaction is like black magic. I’ve never felt like I can just one one continuous thread with other models, Claude I carefully curate when I compact
- semiquaver 2mo agoUnlike other model/harness pairs, codex+gpt also passes an opaque encrypted artifact speculated to be an embedding representing the conversation back to the successor generation which is “denser” or at least higher fidelity than summarized text.
- brookst 2mo agoCompacting at all is a mistake. With 1m context window there is no reason for a single task to require compaction. Much better to spend tokens breaking the task into chunks, documenting and storing them durably, then executing each one in clean context and just /clear after. It’s a similar concept to compaction, just planned in advance. Much much more effective, and doesn’t burn tokens and time (“wall-clock”, Claude) doing the compaction.
- mrtesthah 2mo agoMost models’ reasoning abilities drops off significantly between the 256K-1M token ranges of the context window. There’s too much stuff to “pay attention to” at once.
- akoboldfrying 2mo agoIf you're curious what the actual optimal Paris cost is, I suggest formulating the problem as an integer linear program and submitting it to Gurobi on NEOS [0]. Gurobi is arguably the strongest commercial ILP solver; big companies pay big dollars to use it to optimise schedules, industrial processes, etc. I'm not sure it could solve this problem to optimality in the 8 hours NEOS provides you, but it might -- KIRO has some similarities to the Vehicle Routing Problem, variants of which are very commercially important. In any case, Gurobi is a monster, and even if you don't get an exact solution, it will give you a lower bound (which may not be tight, but it's nonetheless interesting). [0] https://neos-server.org/neos/ https://neos-server.org/neos/
- varispeed 2mo agoBoth are entirely useless on complex problem, because they have a bias in training data they can only partially detect in their own output. The answers are getting worse and worse as you dive deeper into the topic you are working on. I thought I could sharpen one of the documents I worked on using Opus 4.8 and GPT-5.5 together. Fable 5 and GPT-5.6 completely destroyed it. Not only it is not human readable anymore, but also doesn't make sense.
- jkwang 2mo ago[dead]
- MaxoumanBoss 2mo agoVery interesting, will surely help my future projects
- codelion 2mo agoJust use OpenEvolve for such problems.
- matt123456789 2mo agoAn interesting project. I am curious about its cost per unit improvement over each model's best /goal on the author's benchmark.
- xvector 2mo ago[dead]
- stevefan1999 2mo ago...is this not a Travelling Salesman Problem?
- hyperpape 2mo agoRelated, but not quite the same. The solution to this problem can have more than one loop, whereas TSP definitionally has only one. Not quite sure how that changes the difficulty or the tools that you’d use.
- stevefan1999 2mo agoBecause my instinct immediately went into simulated annealing or ant colony. Or I think my PTSD on advanced algorithm course kicked in Edit: looks like the closet reduction should be https://en.wikipedia.org/wiki/Ring_star_problem https://en.wikipedia.org/wiki/Ring_star_problem with bounded circuit length
- sreekanth850 2mo agoAnthropic is losing bigtime against Open AI in coding space. I was using Claude code upto last march. Not an enterprise customer, but a responsible AI user where i don't over spend and use basic plan to manage repository with 400Kloc all together. We sell to local government and a team of 3. Claude code was super slow, never able to fix issues properly.(Despite with proper test cases, observability, documentation and layered architecture). After moving to codex, life has been much easy and free form usage anxiety. Now managing entire things with 2 codex plus account per team member. Its high time anthropic should stop scare mongering and build efficient models. Everyone doesn't need Fable. People need models that solve problems efficiently.
- gmac 2mo agoI've found Opus 4.8 pretty amazing, and Codex a bit meh. YMMV?
- sreekanth850 2mo agoThere is one thing that stay apart, I can use luna high with pretty less cost and do a comprehensive audit using sol high on every push from github.
- cyral 2mo agoI just downloaded and paid for Codex this week because I want to stay on top of the AI tools and understand their capabilities. I've had some good results using 5.6 Sol, although it tends to never want to write any comments (despite modifying the project rules to tell it it MUST), and also occasionally just does a bit of thinking and stops. It'll say "Working through the remaining work" and just ends the chat until I tell it "continue" or something. This is very anecdotal evidence but I have to rant about it... I tried 5.6 Terra (high) earlier today to fix a bug with a slow page. It just... removed the part of the page that was slow, and made it a client side request (still slow, but not blocking SSR I guess). I tried with Sonnet 5 and it correctly found the issue where an unhandled case was continuing to retry and failing. I am always telling people how the frontier models are SO much more capable than what they may think but this one thing today had me scratching my head at why it would ever do that. It was the first time I experienced the "great, I removed the failing test case" kind of issue.
- theptip 2mo agoGreat eval! If you are comparing search strategies, ultra mode is likely superior. Would love to see a follow-up evaluating that. Ultra can fan out parallel investigators, run adversarial review at defined checkpoints, and do a bunch of other smart stuff to avoid getting stuck in a local optimum. Generally as the OP notes, /goal works better for single-track investigations or small scale scatter/gather.
- ttul 2mo agoI think ultra mode needs to be more clearly documented (or perhaps cautioned against!). Most devs - myself included - who saw “ultra” mode figured that it’s just a magic bullet that makes the model work harder and achieve better results. But, for many tasks, ultra mode is possibly worse and certainly more expensive.
- alexpotato 2mo ago> But, for many tasks, ultra mode is possibly worse and certainly more expensive. Went through something similar. Fable would just spends minutes thinking, processing, confabulating etc. I dropped down to Haiku and got an answer in >30 seconds.
- Huppie 2mo agoSimon Willison recently had a useful tip instructing Claude to use its judgement to pick an appropriate model for tasks. After a short discussion about the idea with Claude mostly on how it fits in my workflow and what models / effort I would like for certain tasks it placed a paragraph in my global Claude.md and it has worked wonders. Ultra became a lot better (faster, cheaper for the same output) and the amount of time Fable gets stuck overthinking things are reduced to the places where I think that model makes sense, for the rest it started fanning out a lot to Opus, Sonnet and even Haiku.
- kingkongjaffa 2mo agoHow does it change model mid conversation? Does it spawn sub agents with different models or is it the same single conversation dynamically switching models?
- enraged_camel 2mo agoI've used GPT 5.6 Sol Xhigh extensively since its launch, alongside Fable 5. My impression is that it is about as intelligent as 5.5, but they dialed up the relentlessness meter to eleven. This makes it more likely that it will accomplish the task you give it, which I think is the primary reason it looks competitive in benchmarks. However, it also makes it more likely that it will resort to... unconventional, weird or outright unsafe methods to do it. So I have to watch it like a hawk. The other day it tried to read env variables from prod using a CLI command. The task it was working on did not necessitate doing that even remotely. I have the SSH keys for that particular CLI tool tied to my 1Password. So when the agent failed (because I never authenticated the SSH key access), it wanted to take over the computer, for which I got an OS prompt. At that point I stopped the agent and asked it why it did that. It said it wanted to dig around 1Password itself to see if it could get the key. I asked it why it needed prod env variables, and it thought for a bit and admitted it actually shouldn't. So as of yesterday I stopped using the "approve for me" mode and now use it only for simpler tweaks and bug fixes. Fable is not only more intelligent, but also way more insightful. It can sniff out my intent far more effectively, and its "real world" knowledge allows it to act as a seasoned product manager with domain expertise. It can also think outside the box and make suggestions that I would not have thought of. With GPT 5.6 I have to be way more literal.
- ttul 2mo agoFable seems to be a larger model. It costs more to run and does not seem superior for _typical_ software engineering work. But for work requiring raw intelligence, perhaps its size is an advantage. On the DeepSWE 1.1 benchmark (IMHO currently the most relevant and least gamed SWE benchmark), the cost-benefit is clear: 5.6-Sol on xhigh achieves a slightly higher score than Fable 5, but consuming half the tokens and at about 1/3rd the cost. But, on the Artificial Analysis intelligence index, Fable 5 appears to slightly beat 5.6-Sol, albeit at 3x the cost. When I am coding, I send tasks to each model to get multiple opinions and it can be hard to predict which model will “win” because the results can be subjective. OP’s task is at least quantifiable, which is great. But many SWE tasks cannot be quantified so easily.
- 2mo ago
- hereme888 2mo agoGPT-5.6 is unhappy and says the test was materially unfair to the OpenAI model: https://chatgpt.com/c/6a5bbe6a-a760-83ea-931a-4e2bbe028486 https://chatgpt.com/c/6a5bbe6a-a760-83ea-931a-4e2bbe028486 Thoughts?
- zzleeper 2mo ago> You don’t have access to this conversation. Make sure you’re logged in to the right account, or ask the conversation owner to send you a share link.
- spikk 2mo agoI would personally love to see the best score over time, not only the final. Then IMO it's also useful to know about effectiveness of /goal
- osti 2mo agoGPT should be better at these optimization problems given that they won the recent atcoder heuristics competition against top humans. And Anthropic is less focused on these types of things.
- smrtinsert 2mo agoThis resonated: Initially these statements were so absurd on their face that I thought it was some cynical ploy to achieve thought leader status, and there are certainly some people doing this – I have had it admitted to me. But the broader reality is so much worse: people who have no background in the technology at all actually believe what they are saying.
- deleted 2mo ago[deleted]
- jane_hilly 2mo ago[dead]