7 ms·

Building a Personal AI Factory

266 points by derek 1y ago

steveklabnik 1y ago
I'd love to see more specifics here, that is, how Claude and o3 talk to each other, an example session, etc.
- breckenedge 1y ago
  I presume via Goose via MCP in Claude Code: > I also have a local mcp which runs Goose and o3.
  - steveklabnik 1y ago
    Ah, I skimmed the docs for Goose but I couldn't figure out exactly what it is that it does, which is a common issue for docs. For example: https://block.github.io/goose/docs/category/tutorials/ https://block.github.io/goose/docs/category/tutorials/ I just want to see an example workflow before I set this up in CI or build a custom extension to it!
    - breckenedge 1y ago
      Classic Steve Klabnik comment.
      - IncreasePosts 1y ago
        An uncommon Aaron Breckenridge comment
      - steveklabnik 1y ago
        It's true that I deeply care about docs! Turns out they're good for both humans and LLMs :)
- web3aj 1y ago
  [flagged]
- schmookeeg 1y ago
  I use Zen MCP and OpenRouter. Every once in awhile, my instance of claude code will "phone a friend" and use Gemini for a code review. Often unprompted, sometimes me asking for "analysis" or "ultrathink" about a thorny feature when I doubt the proposed implementation will work out or cause footguns. It's wild to see in action when it's unprompted. For planning, I usually do a trip out to Gemini to check our work, offer ideas, research, and ratings of completeness. The iterations seem to be helpful, at least to me. Everyone in these sorta threads asks for "proofs" and I don't really know what to offer. It's like 4 cents for a second opinion on what claude's planning has cooked up, and the detailed response has been interesting. I loaded 10 bucks onto OpenRouter last month and I think I've pulled it down by like 50 cents. Meanwhile I'm on Claude Max @ $200/mo and GPT Plus for another $20. The OpenRouter stuff seems like less than couch change. $0.02 :D
  - conradev 1y ago
    proof -> show the code if you can! Then engineers can judge for themselves
    - schmookeeg 1y ago
      Yeahhhhhh I've been to enough code reviews / PR reviews to know this will result in 100 opinions about what color the drapes should be and what a catastrophe we've vibe coded for ourselves. If I shoot something to GH I'll highlight it for others, but nothing yet. I can appreciate this makes me look like I'm shilling. It makes usable code for my projects. It often gets into the weeds and makes weird tesseracts of nonsense that I need to discover, tear down, and re-prompt it to not do that again. It's cheap or free to try. It saves me time, particularly in languages I am not used to daily driving. Funnily enough, I get madder when I have it write ts/py/sql code since I'm most conversant in those, but for fringe stuff that I find tedious like AWS config and tests -- it mostly just works. Will it rot my brain? Maybe? If this thing turns me from an engineer to a PM, well, I'll have nobody to blame but myself as I irritate other engineers and demand they fibonacci-size underdefined jira tix. :D I think there's going to be a lot of momentum in this direction in the coming year. I'm fortunate that my clients embrace this stuff and we all look for the same hallucinations in the codebase and shut them down and laugh together, but I worry that I'm not exactly justifying my rate by being an LLM babysitter.
IncreasePosts 1y ago
Okay, what is he actually building with this? I have a problem where half the times I see people talking about their AI workflow, I can't tell if they are talking about some kind of dream workflow that they have, or something they're actually using productively
- ClawsOnPaws 1y ago
  I keep coming to the same conclusion, which basically is: if I had an LLM write it for me, I just don't care about it. There are 2 projects out of the maybe 50 or so that are LLM generated, and even for those two I cared enough to make changes myself without an LLM. The rest just sit there because one day I thought huh wouldn't it be neat if, and then realized actually I cared more about having that thought than having the result of that thought. Then you end up fighting with different models and implementation details and then it messes up something and you go back and forth about how you actually want it to work, and somehow this is so much more draining and exhausting than just getting the work done manually with some slight completion help perhaps, maybe a little bit of boilerplate fill-in. And yes, this is after writing extensive design docs, then having some reasoning LLM figure out the tasks that need to be completed, then having some models talk back and forth about what needs to happen and while it's happening, and then I spent a whole lot of money on what exactly? Questionably working software that kinda sorta does what I wanted it to do? If I have a clear idea, or an existing codebase, if I end up guiding it along, agents and stuff are pretty cool I guess. But vibe coding? Maybe I'm in the minority here but as soon as it's a non trivial app, not just a random small script or bespoke app kind of deal, it's not fun, I often don't get the results I actually wanted out of it even if I tried to be as specific as I wanted with my prompting and design docs and example data and all that, it's expensive, code is still messy as heck, and at the end I feel like I just spent a whole lot of time actually literally arguing with my computer. Why would I want to do that?
  - jwpapi 1y ago
    I’ve written a full stack monorepo with over 1,000 files alone now. I’ve started with AI doing a lot of the work, but the percentage goes down and down. For me a good codebase is not about how much you’ve written, but about how it’s architectured. I want to have an app that has the best possible user and dev experience meaning its easy to maintain and easy to extend. This is achieved by making code easy to understand, for yourself, for others. In my case it’s more like developing a mindset building a framework than to push feature after feature. I would think it’s like that for most companies. You can get an unpolished version of most apps easily, but polishing takes 3-5x the time. Lets not talk about development robustness, backend security etc etc. Like AI has just way too many slippages for me in these cases. However I would still consider myself a heavy AI user, but I mainly use it to discuss plans,(what google used to be) or to check it if I’ve forgotten anything. For most features in my app I’m faster typing it out exactly the way I want it. (with a bit of auto-complete) The whole brain-coordination works better. I guess long talk, but you’re not alone trust your instinct. You don’t seem narrow minded.
photon_garden 1y ago
It’s hard to evaluate setups like this without knowing how the resulting code is being used. Standalone vibe coded apps for personal use? Pretty easy to believe. Writing high quality code in a complex production system? Much harder to believe.
- kasey_junk 1y ago
  I don’t really understand this article or the workflow it’s describing as it’s kind of vague. But I use multiple agents talking to each other, async agents, git work trees etc on complex production systems as my day to day workflow. I wouldn’t say I go so far as to never change the outputs but I certainly view it as signal when I don’t get the outputs I want that I need to work on my workflow.
- 9cb14c1ec0 1y ago
  Exactly. I use claude code as a major speedup in coding, but I stay in the loop on every code change to make sure it is creating an optimal system. The few times that I've just let it run have resulted in bugs that customers had to deal with.
  - Aeolun 1y ago
    I think you can probably get a pretty decent thing going if you have models review output they haven’t written themselves (not still in context anyway)
  - stpedgwdgfhgdd 1y ago
    I noticed that i hardly look anymore at the generated Go code… I do give a lot of attention to the tests. I let CC write some failing tests, implement. Let it run against some real scenarios. Find bugs, let it write more tests, fix. And iterate. Writing this I realise, i should more clearly separate the functional tests from the implementation oriented unit tests.
skybrian 1y ago
> It’s essentially free to fire off a dozen attempts at a task - so I do. What sort of subscription plan is that?
- steveklabnik 1y ago
  Claude Code's $200 Max subscription can take a lot of usage. I haven't done a dozen things at once, but I have worked on two side projects simultaneously with it before. ccusage shows me getting over 10x the value of paying via API tokens this month so far...
  - simonw 1y ago
    I had to look that up: https://github.com/ryoppippi/ccusage https://github.com/ryoppippi/ccusage npx ccusage@latest Outputs a table of your token usage over the last few days, which it reads from the jsonl files that Claude Code leaves tucked away in the ~/.claude/ directory.
    - steveklabnik 1y ago
      Don’t sleep on the other options either, the live updates are cool, see where you’re at in the five hour session.
  - Aeolun 1y ago
    Given you can nearly run two full code instances with Opus, and Opus is claimed to be 5x more expensive than Sonnet, you can maybe do 10 sonnet instances at the same time?
solomonb 1y ago
> When something goes wrong, I don’t hand-patch the generated code. I don’t argue with claude. Instead, I adjust the plan, the prompts, or the agent mix so the next run is correct by construction. I don't think "correct by construction" means what OP thinks it means.
- btbuildem 1y ago
  Also, aren't they just rolling the dice here? Can you turn down the temperature via Claude Code?
- deleted 1y ago
  [deleted]
vFunct 1y ago
The issue I'm facing with multiple agents working on separate work trees is that each independent agent tends to have completely different ideas on absolutely every detail, leading to inconsistent user experience. For example, an agent working on the dashboard for the Documents portion of my project has a completely different idea from the agent working on the dashboard for the Design portion of my project. The design consistency is not there, not just visually, but architecturally. Database schema and API ideas are inconsistent, for example. Even on the same input things are wildly different. It seems that if it can be different, it will be different. You start to update instruction files to get things consistent, but then these end up being thousands of lines on a large project just to get the foundations right, eating into the context window. I think ultimately we might need smaller language models trained on certain rules & schemas only, instead of on the universe of ideas that a prompt could result in. Small language models are likely the correct path.
- Swizec 1y ago
  > each independent agent tends to have completely different ideas on absolutely every detail, leading to inconsistent user experience > The design consistency is not there, not just visually, but architecturally. Seniors always gonna have to senior. Doesn't matter if the coders are AI or humans. You have to make sure you provide enough structures for the agents to move in roughly the same direction while allowing enough flexibility that you're not better off just writing the code.
- pjm331 1y ago
  I’ve had success with building the first version of a thing mostly by hand and then telling Claude code to look at it as an example of how to do things when building the next N of them
  - swader999 1y ago
    The things that work on a regular dev team translate well to the agentic mode.
marviel 1y ago
Thanks for the writeup! I talked about a similar, but slightly simpler workflow in my post on "Vibe Specs". https://lukebechtel.com/blog/vibe-speccing https://lukebechtel.com/blog/vibe-speccing I use these rules in all my codebases now. They essentially cause the AI to do two things differently: (1) ask me questions first (2) Create a `spec.md` doc, before writing any code. Seems not too dissimilar from yours, but I limit it to a single LLM
- rolha-capoeira 1y ago
  I guess a lot of us are trying this (naturally) as solo devs, where we can take an engineering-first mindset and build a machine or factory that spits out gizmos. I haven't gotten to the finish line, mostly because for me, the holy grail is code confidence via e2e tests that the agent generated (separately, not alongside the implementation).
  - marviel 1y ago
    Totally. Yeah I think your approach is a solid take!
- deleted 1y ago
  [deleted]
- _1tem 1y ago
  Claude Code now handles this natively with “plan mode”. Bit slow and annoying to do it manually with .md files in my opinion.
  - marviel 1y ago
    Yeah just learned about this! The md files are actually pretty great for shareability, versioning, and picking up where you left off.
simonw 1y ago
My hunch is that this article is going to be almost completely impenetrable to people who haven't yet had the "aha" moment with Claude Code. That's the moment when you let "claude --dangerously-skip-permissions" go to work on a difficult problem and watch it crunch away by itself for a couple of minutes running a bewildering array of tools until the problem is fixed. I had it compile, run and debug a Mandelbrot fractal generator in 486 assembly today, executing in Docker on my Mac, just to see how well it could do. It did great! https://gist.github.com/simonw/ba1e9fa26fc8af08934d7bc0805b9b80 https://gist.github.com/simonw/ba1e9fa26fc8af08934d7bc0805b9...
- gerdesj 1y ago
  Crack on - this is YC! Why are you not already a unicorn?
  - lucubratory 1y ago
    An LLM wrapper does not have serious revenue potential. Being able to do very impressive things with Claude Code has a pretty strict ceiling on valuation because at any point Anthropic could destroy your business by removing access, incorporating whatever you're doing into their core feature set, etc.
    - petesergeant 1y ago
      Having worked with some serious pieces of enterprise software, I don't think this is right. Anthropic is not going to perfect multi-vendor integrations, spin up a support team, and solution architect your problems for you. Enterprise software gets into the walls, and can be very hard to displace once deployed. If you build an LLM-wrapper resume parser, once you've got it into your client's workflows, they're going to find it hard to unembed it to replace it with raw Anthropic.
      - ffsm8 1y ago
        But if you did become a unicorn, It would suddenly become very easy to replace for anthropic, because they're the ones actually providing the sauce and can just replicate your efforts. So your window of opportunity is to be too small for anthropic to notice and get interested. That can't be called unicorn That was the point he was making, at least that's how I understood it
dkdcio 1y ago
I went down this (and even built a bit of internal web tooling) —- it’s like playing multiple games of online poker for me (instead of the factoria analogy here) it’s really promising, but I found focusing on a single task and doing it well is still more efficient for now. excited for where this goes
apwell23 1y ago
ppl are getting slowly disillusioned with vibe coding. yes AI assisted workflow might be here to stay but it won't be the magical put programmers out of job thing. And this the best product market fit for LLMs. I imagine it will be even worse in other domains.
- azan_ 1y ago
  Are they though? I’m seeing more and more people that used gpt4 and got substandard results get blown away with Claude code and opus once they gave it a chance. Also remember that progress has not stopped (whether it has slowed down is also controversial), so I wouldn’t make strong assumptions that ai won’t replace many devs. I hope it won’t, I really like intellectual work associated with it.
- petesergeant 1y ago
  > ppl are getting slowly disillusioned with vibe coding. This is the absolute polar opposite from my experience. I'm in a large non-tech community with a coders channel, and every day we get a few more Claude Code converts. I would say that vibe-coding is moving into the main-stream with experienced, professional developers who were deeply skeptical a few months ago. It's no longer fancy auto-complete: I have myself seen the magic of wishing a (low importance) front-end app into existence from scratch in an hour or so that would have taken me an order of magnitude more time beforehand.
  - apwell23 1y ago
    oh yea thats true. I was talking more about ppl who have been vibe coding for a while. https://www.reddit.com/r/ClaudeAI/comments/1loj3a0/this_pretty_much_sums_it_up/ https://www.reddit.com/r/ClaudeAI/comments/1loj3a0/this_pret...
  - deleted 1y ago
    [deleted]
  - stavros 1y ago
    I don't doubt that LLMs are extremely useful for making simple things quickly. I haven't been able to get them to write hard code on their own, though. I was trying to make a sound card with a Pi Pico the other day, and had crackling and popping in the audio. I kept telling Opus to fix that, it kept being absolutely convinced it knows what the problem is every time, and went through multiple iterations of being absolutely sure it will solve the problem this time (with every time bringing a different reason for why the pops are there), and spent $35. In the end, it had written 500 lines, the problem was still there, and the code didn't work any differently. It worries me that I don't know what those 500 lines were for. In my experience, LLMs are amazing for writing 10-20 lines at a time, while you review and fix any errors. If I let them go to town on my code, I've found that's an expensive way to get broken code.
geekymartian 1y ago
ADHD coding, brute forcing product generation until you get it right? Just freaking write the code that you can expand and modify in the future instead of increasing your carbon footprint.
- cube00 1y ago
  The end goal is to remove the developer from this equation. Business owner asks for a new CRUD app and there it is in production. Of course it's full of full of bugs, slow as syrup, saves to a public unauthed database but that's none of my business *gulps scalding hot tea*
  - 6510 1y ago
    You have users fill out bug reports then throw some buckets of money at it. You could even add a magic button for when things don't work that reruns the same prompt and possibly get better results. A slot machine animation while waiting would be cool.
    - danielbln 1y ago
      It's like a salt mine in here. Go ahead and hand weave your copper cable code while the world moves on and accelerates. Will there be slop along the way? Oh hell yes. The Model T car was notorious for blowing out tires left and right, to the point that a carriage might have been less hassle at times. Yet here we are.
      - hagbarth 1y ago
        I am not certain this is acceleration in anything other than lines of code.
      - 6510 1y ago
        The model T didn't manifest out of thin air after some incantation nor did it have a mysterious purposes somewhere between some and quite a lot. I assure you, compared to shit coin 90210 these are quite the interesting times. It could be much bigger than the model T or much bigger than asbestos.
gerdesj 1y ago
"Here’s the secret sauce: iterate the inputs": No it isn't. There are no short cuts to ... anything. You expend a lot of input for a lot of output and I'm not too sure you understand why. "Example: an agent once wrote code ..." - not exactly world beating. If you believe this will take over the world, then go full on startup. YC is your oyster. I've run my own firm for 25 years. Nothing exciting and certainly not YC excitable. You wont with this.
- tranchebald 1y ago
  You come across as a massive hater. Maybe it’s a cultural thing. Do you actually have employees?
  - gerdesj 1y ago
    Not a hater at all. I only expound my view from my experience. Accusing me of hate is a bit extreme. I have 20 employees. Cultural? Really?
c4pt0r 1y ago
Maybe a bit off-topic, but the minimalist style of the blog looks really cool.
petesergeant 1y ago
> I keep several claude code windows open, each on its own git-worktree. Can someone convince me they're doing their due-diligence on this code if they're using this approach? I am smart and I am experienced, and I have trouble keeping on top of the changes and subtle bugs being created by one Claude Code.
webprofusion 1y ago
The basic idea is that you can continuously document what your system should do (high level and detailed features), how it should prove it has done that, optionally how you want it to do it (architecture and code style etc). The multi-model AI part is just the (current) tool to help avoid bias and make fine tuned selections for certain parts of the task. Eventually large complex systems will be built and re-built from a set of requirements and software will finally match the stated requirements. The only "legacy code" will be legacy requirements specifications. Fix your requirements, not the generated code.
- qiine 1y ago
  sorry but again... https://i.pinimg.com/736x/03/af/06/03af0602a8caa51507717edd63bb75ba.jpg https://i.pinimg.com/736x/03/af/06/03af0602a8caa51507717edd6...
  - pydry 1y ago
    It is weirdly easy to create a language that expresses specifications that is more complex and difficult to understand than the code which implements it. E.g. Z notation.
  - webprofusion 1y ago
    This is deeply incorrect. Code is not and never will be a requirements specification.
guicen 1y ago
This "AI factory for everyone" model may be able to break resource inequality and allow people from more places to participate in truly valuable entrepreneurship.
namuol 1y ago
No real mention of results that aren’t self-referential. I guess vibe-coding is on its way to becoming the next 3D printing: Expensive hobby best suited for endless tinkering. What’s today’s vibe coding equivalent of a “benchy”? Todo apps?
- deleted 1y ago
  [deleted]
- SchemaLoad 1y ago
  3D printing actually is useful though. Basically everyone designing products or any kind of engineering is using it. The only reason it never took off for the average consumer is that every pre designed piece of plastic junk you could ever want to download and print is already available from Amazon. In a pre online shopping world 3D printing would be far more useful for the average person. Going forward it looks like it's only really useful for people who can design their own files for actually custom stuff you can't buy.
  - namuol 1y ago
    Yeah I’m not saying either aren’t useful, just that they can both be a trap for tinkerers.
am17an 1y ago
I actually don't understand how you can offload the instruction pointer of the program to another program, permanently. How are you accountable for anything then? You can't debug, you can't program, just a tourist in your own home. Own your code, even if AI wrote it.
nico 1y ago
> If you know Factorio you know it’s all about building a factory that can produce itself This is a very interesting concept Could this be extended to the point of an LLM producing/improving itself? If not, what are the current limitations to get to that point?
- NitpickLawyer 1y ago
  > Could this be extended to the point of an LLM producing/improving itself? Check out aider writing aider stats here: https://aider.chat/HISTORY.html https://aider.chat/HISTORY.html
  - nico 1y ago
    Super interesting, thank you for the link Aider writing its own code is definitely cool and within the same concept I’d love to see an LLM or some sort of coding model that modifies/trains the model itself
dgunay 1y ago
I am experimenting with a similar workflow and thought I'd share my experience. I might be a little too hung up on the details compared to a lot of these agent cluster testimonials I've read, but unlike the author I'll be open and say that the codebase I work on is several hundred thousand lines of Go and currently does serve a high 5 to low 6 figure number of real, B2C users. Performance requirements are forgiving but correctness and reliability are very important. Finance. Currently I use a very basic setup of scripts that clone a repo, configure an agent, and then run it against a prompt in a tmux session. I rely mainly on codex-cli since I am only given an OpenAI key to work with. The codex instances ping me in my system notifications when it's my turn, and I can easily quake-mode my terminal into view and then attach to the session (with a bit of help from fzf). I haven't gotten into MCP yet but it's on my radar. I can sort of see the vision. For those small but distracting tasks, they are very helpful and I (mostly) passively produce a lot more small PRs to clean up papercuts around our codebase now. The "cattle not pets" mentality remains relevant - I just fire off a quick prompt when I feel the urge to get sidetracked on something minor. I haven't gotten as much out of them for more involved tasks. Maybe I haven't really got enough of a context flywheel going yet, but I do typically have to intervene most of the time. Even on a working change, I always read the generated code first and make any edits for taste before submitting it for code review since I still view the output as my complete responsibility. I still mostly micromanage the change control process too (branching, making commits, and pushing). I've dabbled in tools that can automate this but haven't gotten around to it. I 100% resonate with the "fix the inputs, not the outputs" mindset as well. It's incredibly powerful without AI and our industry has been slowly but surely adopting it in more places (static typing, devops, IAC, etc). With nondeterministic processes like LLMs though it feels a lot harder to achieve, more like practice and not science.
- barrenko 1y ago
  There's been a lot of talk recently (with "recently" being measured in days for the agents field) about context management, but I'm having the hardest time managing my own context when using these methods.
nilirl 1y ago
"Fix inputs" => The assumption is there exists some perfect input that will give you exactly what you want. It probably works well for small inputs and tasks well-represented in the training data (like writing code for well-represented domains). But how does this work for old code, large codebases, and emergencies? - Do you still "learn" the system like you used to before? - How do you think of refactoring if you don't get a feel for the experience of working through the code base? Overall: I like it. I think this adds speed for code that doesn't need to be reinvented. But new domains, new tools, new ways to model things, the parts that are fun to a developer, are still our monsters to slay.
- _1tem 1y ago
  > But how does this work for old code, large codebases, and emergencies? Have you actually tried Claude Code? It works pretty well on my old code, medium size SaaS codebase. I’ve had it build entire features end to end in (backend, front end, data migrations, tests) in one or two prompts.
barrenko 1y ago
> Is 'Azure OpenAI subscription' cheaper than ChatGPT via OpenAI?
mmarian 1y ago
And here I am struggling to get Claude to create a nice-looking search bar a la booking.com , with some adjustments for my personal use case; it does ok, but never gets to the end result and once I refreshed my Tailwind knowledge it felt much slower than hand coding. I feel like I'm living in a different world.
- hamstergene 1y ago
  I think coding assistants aren't great at UI/UX yet because they can't see, their understanding of left/right/lighter/darker is guessed from textual descriptions that accompanied CSS tutorials but they are never actually imagining the looks of what they are working with. I had Cursor repeatedly fix and mess up a CSS grid, over and over again, until I switched to HTML table so that browser would handle layout. Once switched from visuals ("leftmost") to semantics ("first cell in a row") the agent immediately started getting tasks done right. I guess keep them on backend/library tasks for now. I am sure the companies are already working on getting a snapshot of a browser page and feeding it back into multimodal model so it can comprehend what "looking" means.
  - mmarian 1y ago
    Thx for sharing your experience, good to know I'm not the only one struggling ^_^ The advice makes sense as well.
- derencius 1y ago
  I use Claude and Cursor in parallel. cursor is doing great on the ui, I took quick screenshots to instruct the changes I wanted and it got it right.
  - mmarian 1y ago
    Cheers! It's hard to keep track of what's good for what.
caporaltito 1y ago
Show us the code, mate.
hamish-b 1y ago
This sounds great, and is similar to the workflow I get from a high level stand point with https://ampcode.com/ https://ampcode.com/ - albeit without the model wrangling. To the author & anyone reading - publicly release your agent harnesses, even if its shit or vibe coded! I am constantly iterating on my meta and seeking to improve.
nurettin 1y ago
This sounds nice and great and all, but I wonder what the output is like and if there is a measurable difference between doing the factory and trying to two shot the whole thing with claude 4 sonnet.
GTP 1y ago
> That loop is the factory: the code itself is disposable; the instructions and agents are the real asset. Why do I hear the words "technical debt"? More to the point, the risk I see with this approach is that the author would end hp throwing away working and well tested code to implement some minor change. This has an high risk of introducing many easily avoidable bugs.
mfalcon 1y ago
"Outputs are disposable; plans and prompts compound." I agree with this and it aligns with the general opinion about what is the true value the SWE's bring to the table.
codemonkey-zeta 1y ago
> Because most of my day-to-day is in clojure I tend to use sonnet 4 to get the parens right. In case the author is lurking, you may want to apply the same fix they do in clojure-mcp: https://github.com/bhauman/clojure-mcp/blob/8150b855282babcd858a5aa3b6c141a587e51d2a/src/clojure_mcp/tools/eval/core.clj#L37 https://github.com/bhauman/clojure-mcp/blob/8150b855282babcd... The insight that team had was that LLMs get confused with parens, but they are excellent at indentation, so if you run parinfer over the LLMs output it will be correct in 99% of cases.
PhilippGille 1y ago
> Next claude code execute the plan, either with sonnet 3.7 or sonnet 4 depending on the complexity of the task. Because most of my day-to-day is in clojure I tend to use sonnet 4 to get the parens right. This made me chuckle. Perfect example of why heavily LLM-driven devs and processes might want to pick a popular programming language which the LLM had a ton of training data for. Or a strong point for specialized LLMs (e.g. here it could be a smaller/cheaper/faster Clojure-specialized model).
puersum 1y ago
I believe we need to find more effective ways to integrate AI into our workflows. Anyone who is actively trying to adopt AI has likely encountered similar challenges, yet a definitive solution has yet to emerge. In my view, a key principle at this stage is to assign AI minimal responsibility and highly specific tasks. For example, I'm currently experimenting with an agent workflow for stock research. I've set up two AI roles: a 'Bullish Guy' and a 'Bearish Guy' and have them debate the pros and cons of a specific stock. The premise is that through this adversarial process, the AIs are forced to research opposing viewpoints, leading to a more comprehensive understanding and a superior final analysis. The idea was inspired by the kinds of arguments you see on social media.
- MatveySecured 1y ago
  hey! Your agent workflow for stock research sounds very interesting. Can you share a link please?
dearilos 1y ago
I'm thinking about this a little differently. You have to catch most issues at code review. You can have an agent spit out code, but on PR open you should have another agent verify rules based on rules you define as a team. It's what I'm building at wispbit.
neurostimulant 1y ago
Imagine a future where a program source code is a bunch of .md files, the build script is telling some ai agents to execute the plan in the .md files, and creating a new version is done by re-doing the same steps but with newer and smarter ai agent. I'll probably became a farmer by then.

266 points by derek 1y ago

steveklabnik 1y ago

I'd love to see more specifics here, that is, how Claude and o3 talk to each other, an example session, etc.

breckenedge 1y ago
I presume via Goose via MCP in Claude Code: > I also have a local mcp which runs Goose and o3.
- steveklabnik 1y ago
  Ah, I skimmed the docs for Goose but I couldn't figure out exactly what it is that it does, which is a common issue for docs. For example: https://block.github.io/goose/docs/category/tutorials/ https://block.github.io/goose/docs/category/tutorials/ I just want to see an example workflow before I set this up in CI or build a custom extension to it!
  - breckenedge 1y ago
    Classic Steve Klabnik comment.
    - IncreasePosts 1y ago
      An uncommon Aaron Breckenridge comment
    - steveklabnik 1y ago
      It's true that I deeply care about docs! Turns out they're good for both humans and LLMs :)
web3aj 1y ago
[flagged]
schmookeeg 1y ago
I use Zen MCP and OpenRouter. Every once in awhile, my instance of claude code will "phone a friend" and use Gemini for a code review. Often unprompted, sometimes me asking for "analysis" or "ultrathink" about a thorny feature when I doubt the proposed implementation will work out or cause footguns. It's wild to see in action when it's unprompted. For planning, I usually do a trip out to Gemini to check our work, offer ideas, research, and ratings of completeness. The iterations seem to be helpful, at least to me. Everyone in these sorta threads asks for "proofs" and I don't really know what to offer. It's like 4 cents for a second opinion on what claude's planning has cooked up, and the detailed response has been interesting. I loaded 10 bucks onto OpenRouter last month and I think I've pulled it down by like 50 cents. Meanwhile I'm on Claude Max @ $200/mo and GPT Plus for another $20. The OpenRouter stuff seems like less than couch change. $0.02 :D
- conradev 1y ago
  proof -> show the code if you can! Then engineers can judge for themselves
  - schmookeeg 1y ago
    Yeahhhhhh I've been to enough code reviews / PR reviews to know this will result in 100 opinions about what color the drapes should be and what a catastrophe we've vibe coded for ourselves. If I shoot something to GH I'll highlight it for others, but nothing yet. I can appreciate this makes me look like I'm shilling. It makes usable code for my projects. It often gets into the weeds and makes weird tesseracts of nonsense that I need to discover, tear down, and re-prompt it to not do that again. It's cheap or free to try. It saves me time, particularly in languages I am not used to daily driving. Funnily enough, I get madder when I have it write ts/py/sql code since I'm most conversant in those, but for fringe stuff that I find tedious like AWS config and tests -- it mostly just works. Will it rot my brain? Maybe? If this thing turns me from an engineer to a PM, well, I'll have nobody to blame but myself as I irritate other engineers and demand they fibonacci-size underdefined jira tix. :D I think there's going to be a lot of momentum in this direction in the coming year. I'm fortunate that my clients embrace this stuff and we all look for the same hallucinations in the codebase and shut them down and laugh together, but I worry that I'm not exactly justifying my rate by being an LLM babysitter.

IncreasePosts 1y ago

Okay, what is he actually building with this? I have a problem where half the times I see people talking about their AI workflow, I can't tell if they are talking about some kind of dream workflow that they have, or something they're actually using productively

ClawsOnPaws 1y ago
I keep coming to the same conclusion, which basically is: if I had an LLM write it for me, I just don't care about it. There are 2 projects out of the maybe 50 or so that are LLM generated, and even for those two I cared enough to make changes myself without an LLM. The rest just sit there because one day I thought huh wouldn't it be neat if, and then realized actually I cared more about having that thought than having the result of that thought. Then you end up fighting with different models and implementation details and then it messes up something and you go back and forth about how you actually want it to work, and somehow this is so much more draining and exhausting than just getting the work done manually with some slight completion help perhaps, maybe a little bit of boilerplate fill-in. And yes, this is after writing extensive design docs, then having some reasoning LLM figure out the tasks that need to be completed, then having some models talk back and forth about what needs to happen and while it's happening, and then I spent a whole lot of money on what exactly? Questionably working software that kinda sorta does what I wanted it to do? If I have a clear idea, or an existing codebase, if I end up guiding it along, agents and stuff are pretty cool I guess. But vibe coding? Maybe I'm in the minority here but as soon as it's a non trivial app, not just a random small script or bespoke app kind of deal, it's not fun, I often don't get the results I actually wanted out of it even if I tried to be as specific as I wanted with my prompting and design docs and example data and all that, it's expensive, code is still messy as heck, and at the end I feel like I just spent a whole lot of time actually literally arguing with my computer. Why would I want to do that?
- jwpapi 1y ago
  I’ve written a full stack monorepo with over 1,000 files alone now. I’ve started with AI doing a lot of the work, but the percentage goes down and down. For me a good codebase is not about how much you’ve written, but about how it’s architectured. I want to have an app that has the best possible user and dev experience meaning its easy to maintain and easy to extend. This is achieved by making code easy to understand, for yourself, for others. In my case it’s more like developing a mindset building a framework than to push feature after feature. I would think it’s like that for most companies. You can get an unpolished version of most apps easily, but polishing takes 3-5x the time. Lets not talk about development robustness, backend security etc etc. Like AI has just way too many slippages for me in these cases. However I would still consider myself a heavy AI user, but I mainly use it to discuss plans,(what google used to be) or to check it if I’ve forgotten anything. For most features in my app I’m faster typing it out exactly the way I want it. (with a bit of auto-complete) The whole brain-coordination works better. I guess long talk, but you’re not alone trust your instinct. You don’t seem narrow minded.

photon_garden 1y ago

It’s hard to evaluate setups like this without knowing how the resulting code is being used. Standalone vibe coded apps for personal use? Pretty easy to believe. Writing high quality code in a complex production system? Much harder to believe.

kasey_junk 1y ago
I don’t really understand this article or the workflow it’s describing as it’s kind of vague. But I use multiple agents talking to each other, async agents, git work trees etc on complex production systems as my day to day workflow. I wouldn’t say I go so far as to never change the outputs but I certainly view it as signal when I don’t get the outputs I want that I need to work on my workflow.
9cb14c1ec0 1y ago
Exactly. I use claude code as a major speedup in coding, but I stay in the loop on every code change to make sure it is creating an optimal system. The few times that I've just let it run have resulted in bugs that customers had to deal with.
- Aeolun 1y ago
  I think you can probably get a pretty decent thing going if you have models review output they haven’t written themselves (not still in context anyway)
- stpedgwdgfhgdd 1y ago
  I noticed that i hardly look anymore at the generated Go code… I do give a lot of attention to the tests. I let CC write some failing tests, implement. Let it run against some real scenarios. Find bugs, let it write more tests, fix. And iterate. Writing this I realise, i should more clearly separate the functional tests from the implementation oriented unit tests.

skybrian 1y ago

> It’s essentially free to fire off a dozen attempts at a task - so I do. What sort of subscription plan is that?

steveklabnik 1y ago
Claude Code's $200 Max subscription can take a lot of usage. I haven't done a dozen things at once, but I have worked on two side projects simultaneously with it before. ccusage shows me getting over 10x the value of paying via API tokens this month so far...
- simonw 1y ago
  I had to look that up: https://github.com/ryoppippi/ccusage https://github.com/ryoppippi/ccusage npx ccusage@latest Outputs a table of your token usage over the last few days, which it reads from the jsonl files that Claude Code leaves tucked away in the ~/.claude/ directory.
  - steveklabnik 1y ago
    Don’t sleep on the other options either, the live updates are cool, see where you’re at in the five hour session.
- Aeolun 1y ago
  Given you can nearly run two full code instances with Opus, and Opus is claimed to be 5x more expensive than Sonnet, you can maybe do 10 sonnet instances at the same time?

solomonb 1y ago

> When something goes wrong, I don’t hand-patch the generated code. I don’t argue with claude. Instead, I adjust the plan, the prompts, or the agent mix so the next run is correct by construction. I don't think "correct by construction" means what OP thinks it means.

btbuildem 1y ago
Also, aren't they just rolling the dice here? Can you turn down the temperature via Claude Code?
deleted 1y ago
[deleted]

vFunct 1y ago

The issue I'm facing with multiple agents working on separate work trees is that each independent agent tends to have completely different ideas on absolutely every detail, leading to inconsistent user experience. For example, an agent working on the dashboard for the Documents portion of my project has a completely different idea from the agent working on the dashboard for the Design portion of my project. The design consistency is not there, not just visually, but architecturally. Database schema and API ideas are inconsistent, for example. Even on the same input things are wildly different. It seems that if it can be different, it will be different. You start to update instruction files to get things consistent, but then these end up being thousands of lines on a large project just to get the foundations right, eating into the context window. I think ultimately we might need smaller language models trained on certain rules & schemas only, instead of on the universe of ideas that a prompt could result in. Small language models are likely the correct path.

Swizec 1y ago
> each independent agent tends to have completely different ideas on absolutely every detail, leading to inconsistent user experience > The design consistency is not there, not just visually, but architecturally. Seniors always gonna have to senior. Doesn't matter if the coders are AI or humans. You have to make sure you provide enough structures for the agents to move in roughly the same direction while allowing enough flexibility that you're not better off just writing the code.
pjm331 1y ago
I’ve had success with building the first version of a thing mostly by hand and then telling Claude code to look at it as an example of how to do things when building the next N of them
- swader999 1y ago
  The things that work on a regular dev team translate well to the agentic mode.

marviel 1y ago

Thanks for the writeup! I talked about a similar, but slightly simpler workflow in my post on "Vibe Specs". https://lukebechtel.com/blog/vibe-speccing https://lukebechtel.com/blog/vibe-speccing I use these rules in all my codebases now. They essentially cause the AI to do two things differently: (1) ask me questions first (2) Create a `spec.md` doc, before writing any code. Seems not too dissimilar from yours, but I limit it to a single LLM

rolha-capoeira 1y ago
I guess a lot of us are trying this (naturally) as solo devs, where we can take an engineering-first mindset and build a machine or factory that spits out gizmos. I haven't gotten to the finish line, mostly because for me, the holy grail is code confidence via e2e tests that the agent generated (separately, not alongside the implementation).
- marviel 1y ago
  Totally. Yeah I think your approach is a solid take!
deleted 1y ago
[deleted]
_1tem 1y ago
Claude Code now handles this natively with “plan mode”. Bit slow and annoying to do it manually with .md files in my opinion.
- marviel 1y ago
  Yeah just learned about this! The md files are actually pretty great for shareability, versioning, and picking up where you left off.

simonw 1y ago

My hunch is that this article is going to be almost completely impenetrable to people who haven't yet had the "aha" moment with Claude Code. That's the moment when you let "claude --dangerously-skip-permissions" go to work on a difficult problem and watch it crunch away by itself for a couple of minutes running a bewildering array of tools until the problem is fixed. I had it compile, run and debug a Mandelbrot fractal generator in 486 assembly today, executing in Docker on my Mac, just to see how well it could do. It did great! https://gist.github.com/simonw/ba1e9fa26fc8af08934d7bc0805b9b80 https://gist.github.com/simonw/ba1e9fa26fc8af08934d7bc0805b9...

gerdesj 1y ago
Crack on - this is YC! Why are you not already a unicorn?
- lucubratory 1y ago
  An LLM wrapper does not have serious revenue potential. Being able to do very impressive things with Claude Code has a pretty strict ceiling on valuation because at any point Anthropic could destroy your business by removing access, incorporating whatever you're doing into their core feature set, etc.
  - petesergeant 1y ago
    Having worked with some serious pieces of enterprise software, I don't think this is right. Anthropic is not going to perfect multi-vendor integrations, spin up a support team, and solution architect your problems for you. Enterprise software gets into the walls, and can be very hard to displace once deployed. If you build an LLM-wrapper resume parser, once you've got it into your client's workflows, they're going to find it hard to unembed it to replace it with raw Anthropic.
    - ffsm8 1y ago
      But if you did become a unicorn, It would suddenly become very easy to replace for anthropic, because they're the ones actually providing the sauce and can just replicate your efforts. So your window of opportunity is to be too small for anthropic to notice and get interested. That can't be called unicorn That was the point he was making, at least that's how I understood it

dkdcio 1y ago

I went down this (and even built a bit of internal web tooling) —- it’s like playing multiple games of online poker for me (instead of the factoria analogy here) it’s really promising, but I found focusing on a single task and doing it well is still more efficient for now. excited for where this goes

apwell23 1y ago

ppl are getting slowly disillusioned with vibe coding. yes AI assisted workflow might be here to stay but it won't be the magical put programmers out of job thing. And this the best product market fit for LLMs. I imagine it will be even worse in other domains.

azan_ 1y ago
Are they though? I’m seeing more and more people that used gpt4 and got substandard results get blown away with Claude code and opus once they gave it a chance. Also remember that progress has not stopped (whether it has slowed down is also controversial), so I wouldn’t make strong assumptions that ai won’t replace many devs. I hope it won’t, I really like intellectual work associated with it.
petesergeant 1y ago
> ppl are getting slowly disillusioned with vibe coding. This is the absolute polar opposite from my experience. I'm in a large non-tech community with a coders channel, and every day we get a few more Claude Code converts. I would say that vibe-coding is moving into the main-stream with experienced, professional developers who were deeply skeptical a few months ago. It's no longer fancy auto-complete: I have myself seen the magic of wishing a (low importance) front-end app into existence from scratch in an hour or so that would have taken me an order of magnitude more time beforehand.
- apwell23 1y ago
  oh yea thats true. I was talking more about ppl who have been vibe coding for a while. https://www.reddit.com/r/ClaudeAI/comments/1loj3a0/this_pretty_much_sums_it_up/ https://www.reddit.com/r/ClaudeAI/comments/1loj3a0/this_pret...
- deleted 1y ago
  [deleted]
- stavros 1y ago
  I don't doubt that LLMs are extremely useful for making simple things quickly. I haven't been able to get them to write hard code on their own, though. I was trying to make a sound card with a Pi Pico the other day, and had crackling and popping in the audio. I kept telling Opus to fix that, it kept being absolutely convinced it knows what the problem is every time, and went through multiple iterations of being absolutely sure it will solve the problem this time (with every time bringing a different reason for why the pops are there), and spent $35. In the end, it had written 500 lines, the problem was still there, and the code didn't work any differently. It worries me that I don't know what those 500 lines were for. In my experience, LLMs are amazing for writing 10-20 lines at a time, while you review and fix any errors. If I let them go to town on my code, I've found that's an expensive way to get broken code.

geekymartian 1y ago

ADHD coding, brute forcing product generation until you get it right? Just freaking write the code that you can expand and modify in the future instead of increasing your carbon footprint.

cube00 1y ago
The end goal is to remove the developer from this equation. Business owner asks for a new CRUD app and there it is in production. Of course it's full of full of bugs, slow as syrup, saves to a public unauthed database but that's none of my business *gulps scalding hot tea*
- 6510 1y ago
  You have users fill out bug reports then throw some buckets of money at it. You could even add a magic button for when things don't work that reruns the same prompt and possibly get better results. A slot machine animation while waiting would be cool.
  - danielbln 1y ago
    It's like a salt mine in here. Go ahead and hand weave your copper cable code while the world moves on and accelerates. Will there be slop along the way? Oh hell yes. The Model T car was notorious for blowing out tires left and right, to the point that a carriage might have been less hassle at times. Yet here we are.
    - hagbarth 1y ago
      I am not certain this is acceleration in anything other than lines of code.
    - 6510 1y ago
      The model T didn't manifest out of thin air after some incantation nor did it have a mysterious purposes somewhere between some and quite a lot. I assure you, compared to shit coin 90210 these are quite the interesting times. It could be much bigger than the model T or much bigger than asbestos.

gerdesj 1y ago

"Here’s the secret sauce: iterate the inputs": No it isn't. There are no short cuts to ... anything. You expend a lot of input for a lot of output and I'm not too sure you understand why. "Example: an agent once wrote code ..." - not exactly world beating. If you believe this will take over the world, then go full on startup. YC is your oyster. I've run my own firm for 25 years. Nothing exciting and certainly not YC excitable. You wont with this.

tranchebald 1y ago
You come across as a massive hater. Maybe it’s a cultural thing. Do you actually have employees?
- gerdesj 1y ago
  Not a hater at all. I only expound my view from my experience. Accusing me of hate is a bit extreme. I have 20 employees. Cultural? Really?

c4pt0r 1y ago

Maybe a bit off-topic, but the minimalist style of the blog looks really cool.

petesergeant 1y ago

> I keep several claude code windows open, each on its own git-worktree. Can someone convince me they're doing their due-diligence on this code if they're using this approach? I am smart and I am experienced, and I have trouble keeping on top of the changes and subtle bugs being created by one Claude Code.

webprofusion 1y ago

The basic idea is that you can continuously document what your system should do (high level and detailed features), how it should prove it has done that, optionally how you want it to do it (architecture and code style etc). The multi-model AI part is just the (current) tool to help avoid bias and make fine tuned selections for certain parts of the task. Eventually large complex systems will be built and re-built from a set of requirements and software will finally match the stated requirements. The only "legacy code" will be legacy requirements specifications. Fix your requirements, not the generated code.

qiine 1y ago
sorry but again... https://i.pinimg.com/736x/03/af/06/03af0602a8caa51507717edd63bb75ba.jpg https://i.pinimg.com/736x/03/af/06/03af0602a8caa51507717edd6...
- pydry 1y ago
  It is weirdly easy to create a language that expresses specifications that is more complex and difficult to understand than the code which implements it. E.g. Z notation.
- webprofusion 1y ago
  This is deeply incorrect. Code is not and never will be a requirements specification.

guicen 1y ago

This "AI factory for everyone" model may be able to break resource inequality and allow people from more places to participate in truly valuable entrepreneurship.

namuol 1y ago

No real mention of results that aren’t self-referential. I guess vibe-coding is on its way to becoming the next 3D printing: Expensive hobby best suited for endless tinkering. What’s today’s vibe coding equivalent of a “benchy”? Todo apps?

deleted 1y ago
[deleted]
SchemaLoad 1y ago
3D printing actually is useful though. Basically everyone designing products or any kind of engineering is using it. The only reason it never took off for the average consumer is that every pre designed piece of plastic junk you could ever want to download and print is already available from Amazon. In a pre online shopping world 3D printing would be far more useful for the average person. Going forward it looks like it's only really useful for people who can design their own files for actually custom stuff you can't buy.
- namuol 1y ago
  Yeah I’m not saying either aren’t useful, just that they can both be a trap for tinkerers.

am17an 1y ago

I actually don't understand how you can offload the instruction pointer of the program to another program, permanently. How are you accountable for anything then? You can't debug, you can't program, just a tourist in your own home. Own your code, even if AI wrote it.

nico 1y ago

> If you know Factorio you know it’s all about building a factory that can produce itself This is a very interesting concept Could this be extended to the point of an LLM producing/improving itself? If not, what are the current limitations to get to that point?

NitpickLawyer 1y ago
> Could this be extended to the point of an LLM producing/improving itself? Check out aider writing aider stats here: https://aider.chat/HISTORY.html https://aider.chat/HISTORY.html
- nico 1y ago
  Super interesting, thank you for the link Aider writing its own code is definitely cool and within the same concept I’d love to see an LLM or some sort of coding model that modifies/trains the model itself

dgunay 1y ago

I am experimenting with a similar workflow and thought I'd share my experience. I might be a little too hung up on the details compared to a lot of these agent cluster testimonials I've read, but unlike the author I'll be open and say that the codebase I work on is several hundred thousand lines of Go and currently does serve a high 5 to low 6 figure number of real, B2C users. Performance requirements are forgiving but correctness and reliability are very important. Finance. Currently I use a very basic setup of scripts that clone a repo, configure an agent, and then run it against a prompt in a tmux session. I rely mainly on codex-cli since I am only given an OpenAI key to work with. The codex instances ping me in my system notifications when it's my turn, and I can easily quake-mode my terminal into view and then attach to the session (with a bit of help from fzf). I haven't gotten into MCP yet but it's on my radar. I can sort of see the vision. For those small but distracting tasks, they are very helpful and I (mostly) passively produce a lot more small PRs to clean up papercuts around our codebase now. The "cattle not pets" mentality remains relevant - I just fire off a quick prompt when I feel the urge to get sidetracked on something minor. I haven't gotten as much out of them for more involved tasks. Maybe I haven't really got enough of a context flywheel going yet, but I do typically have to intervene most of the time. Even on a working change, I always read the generated code first and make any edits for taste before submitting it for code review since I still view the output as my complete responsibility. I still mostly micromanage the change control process too (branching, making commits, and pushing). I've dabbled in tools that can automate this but haven't gotten around to it. I 100% resonate with the "fix the inputs, not the outputs" mindset as well. It's incredibly powerful without AI and our industry has been slowly but surely adopting it in more places (static typing, devops, IAC, etc). With nondeterministic processes like LLMs though it feels a lot harder to achieve, more like practice and not science.

barrenko 1y ago
There's been a lot of talk recently (with "recently" being measured in days for the agents field) about context management, but I'm having the hardest time managing my own context when using these methods.

nilirl 1y ago

"Fix inputs" => The assumption is there exists some perfect input that will give you exactly what you want. It probably works well for small inputs and tasks well-represented in the training data (like writing code for well-represented domains). But how does this work for old code, large codebases, and emergencies? - Do you still "learn" the system like you used to before? - How do you think of refactoring if you don't get a feel for the experience of working through the code base? Overall: I like it. I think this adds speed for code that doesn't need to be reinvented. But new domains, new tools, new ways to model things, the parts that are fun to a developer, are still our monsters to slay.

_1tem 1y ago
> But how does this work for old code, large codebases, and emergencies? Have you actually tried Claude Code? It works pretty well on my old code, medium size SaaS codebase. I’ve had it build entire features end to end in (backend, front end, data migrations, tests) in one or two prompts.

barrenko 1y ago

> Is 'Azure OpenAI subscription' cheaper than ChatGPT via OpenAI?

mmarian 1y ago

And here I am struggling to get Claude to create a nice-looking search bar a la booking.com , with some adjustments for my personal use case; it does ok, but never gets to the end result and once I refreshed my Tailwind knowledge it felt much slower than hand coding. I feel like I'm living in a different world.

hamstergene 1y ago
I think coding assistants aren't great at UI/UX yet because they can't see, their understanding of left/right/lighter/darker is guessed from textual descriptions that accompanied CSS tutorials but they are never actually imagining the looks of what they are working with. I had Cursor repeatedly fix and mess up a CSS grid, over and over again, until I switched to HTML table so that browser would handle layout. Once switched from visuals ("leftmost") to semantics ("first cell in a row") the agent immediately started getting tasks done right. I guess keep them on backend/library tasks for now. I am sure the companies are already working on getting a snapshot of a browser page and feeding it back into multimodal model so it can comprehend what "looking" means.
- mmarian 1y ago
  Thx for sharing your experience, good to know I'm not the only one struggling ^_^ The advice makes sense as well.
derencius 1y ago
I use Claude and Cursor in parallel. cursor is doing great on the ui, I took quick screenshots to instruct the changes I wanted and it got it right.
- mmarian 1y ago
  Cheers! It's hard to keep track of what's good for what.

caporaltito 1y ago

Show us the code, mate.

hamish-b 1y ago

This sounds great, and is similar to the workflow I get from a high level stand point with https://ampcode.com/ https://ampcode.com/ - albeit without the model wrangling. To the author & anyone reading - publicly release your agent harnesses, even if its shit or vibe coded! I am constantly iterating on my meta and seeking to improve.

nurettin 1y ago

This sounds nice and great and all, but I wonder what the output is like and if there is a measurable difference between doing the factory and trying to two shot the whole thing with claude 4 sonnet.

GTP 1y ago

> That loop is the factory: the code itself is disposable; the instructions and agents are the real asset. Why do I hear the words "technical debt"? More to the point, the risk I see with this approach is that the author would end hp throwing away working and well tested code to implement some minor change. This has an high risk of introducing many easily avoidable bugs.

mfalcon 1y ago

"Outputs are disposable; plans and prompts compound." I agree with this and it aligns with the general opinion about what is the true value the SWE's bring to the table.

codemonkey-zeta 1y ago

> Because most of my day-to-day is in clojure I tend to use sonnet 4 to get the parens right. In case the author is lurking, you may want to apply the same fix they do in clojure-mcp: https://github.com/bhauman/clojure-mcp/blob/8150b855282babcd858a5aa3b6c141a587e51d2a/src/clojure_mcp/tools/eval/core.clj#L37 https://github.com/bhauman/clojure-mcp/blob/8150b855282babcd... The insight that team had was that LLMs get confused with parens, but they are excellent at indentation, so if you run parinfer over the LLMs output it will be correct in 99% of cases.

PhilippGille 1y ago

> Next claude code execute the plan, either with sonnet 3.7 or sonnet 4 depending on the complexity of the task. Because most of my day-to-day is in clojure I tend to use sonnet 4 to get the parens right. This made me chuckle. Perfect example of why heavily LLM-driven devs and processes might want to pick a popular programming language which the LLM had a ton of training data for. Or a strong point for specialized LLMs (e.g. here it could be a smaller/cheaper/faster Clojure-specialized model).

puersum 1y ago

I believe we need to find more effective ways to integrate AI into our workflows. Anyone who is actively trying to adopt AI has likely encountered similar challenges, yet a definitive solution has yet to emerge. In my view, a key principle at this stage is to assign AI minimal responsibility and highly specific tasks. For example, I'm currently experimenting with an agent workflow for stock research. I've set up two AI roles: a 'Bullish Guy' and a 'Bearish Guy' and have them debate the pros and cons of a specific stock. The premise is that through this adversarial process, the AIs are forced to research opposing viewpoints, leading to a more comprehensive understanding and a superior final analysis. The idea was inspired by the kinds of arguments you see on social media.

MatveySecured 1y ago
hey! Your agent workflow for stock research sounds very interesting. Can you share a link please?

dearilos 1y ago

I'm thinking about this a little differently. You have to catch most issues at code review. You can have an agent spit out code, but on PR open you should have another agent verify rules based on rules you define as a team. It's what I'm building at wispbit.

neurostimulant 1y ago

Imagine a future where a program source code is a bunch of .md files, the build script is telling some ai agents to execute the plan in the .md files, and creating a new version is done by re-doing the same steps but with newer and smarter ai agent. I'll probably became a farmer by then.