6 ms·

Yes, you need some kind of other source of truth. I think the best way to get that is to do clean room development with a different agent, but ultimately if you

Yes, you need some kind of other source of truth. I think the best way to get that is to do clean room development with a different agent, but ultimately if you give them the wrong idea they'll do the wrong thing. The other thing I do, not as much as I should, but it's very powerful, is to generate spikes and deliberately throw them away to understand how to prompt better. Like I generated a swift version of the react native app I'm working on, and Alloy provers for the state transitions. None of it is production quality but getting great results that way is useful to scope future work.

hedgehog 9d ago
If you've managed people, these are all familiar problems. I found you need much more than a functional specification, you also need motivation, background, related work, ideas tried, etc., because those help disambiguate the right path in the inevitable situation where your original task description is unclear or conflicts with itself.
- jaggederest 9d ago
  Yes I'm using the entire consultancy stack - define values, etc, and work your way down the "where do these not match reality on the ground and need change", but for little robot people instead of (arguably less messy) humans.
andai 8d ago
> I think the best way to get that is to do clean room development with a different agent What is this about? Could you give an example?
- jaggederest 8d ago
  So, if I'm building, say an iOS app, I have a) the code itself (swift/react native), b) the internal test suite (unit tests), and c) the e2e test suite with something like XCUITest that drives the physical development phone, plus d) the backend (which has similar tests but is basically a mirror, so I won't elaborate) So when you're creating a (and hopefully b), you use one model with one context, and then you might use some other model to implement c. I kind of round robin the models and present them with seperate context - so for example I don't build a, b, and c together, I build a plus some b, then later go for a pass over b and c together, then maybe I used c to drive improvements in a, and I vary between openai and anthropic models as I do. Heres an example set of minimal prompts: a-focus: implement feature <x> based on #ticket in github, be sure to reference the engineering standards documents and the swift and react native skills as needed b-focus: improve test coverage in the repo for <subsystem z> to ensure that <feature x> is covered completely, and fix any outstanding gaps or omissions in that feature as you go (standard references above) c-focus (possibly in a separate repo): You are creating a black box xcuitest to drive a physical phone for testing <feature x>, here is the user specification and known issues, create failing tests for each known issue and an overall robust suite to ensure any user facing or ui issues are caught

hedgehog 9d ago

If you've managed people, these are all familiar problems. I found you need much more than a functional specification, you also need motivation, background, related work, ideas tried, etc., because those help disambiguate the right path in the inevitable situation where your original task description is unclear or conflicts with itself.

jaggederest 9d ago
Yes I'm using the entire consultancy stack - define values, etc, and work your way down the "where do these not match reality on the ground and need change", but for little robot people instead of (arguably less messy) humans.

andai 8d ago

> I think the best way to get that is to do clean room development with a different agent What is this about? Could you give an example?

jaggederest 8d ago
So, if I'm building, say an iOS app, I have a) the code itself (swift/react native), b) the internal test suite (unit tests), and c) the e2e test suite with something like XCUITest that drives the physical development phone, plus d) the backend (which has similar tests but is basically a mirror, so I won't elaborate) So when you're creating a (and hopefully b), you use one model with one context, and then you might use some other model to implement c. I kind of round robin the models and present them with seperate context - so for example I don't build a, b, and c together, I build a plus some b, then later go for a pass over b and c together, then maybe I used c to drive improvements in a, and I vary between openai and anthropic models as I do. Heres an example set of minimal prompts: a-focus: implement feature <x> based on #ticket in github, be sure to reference the engineering standards documents and the swift and react native skills as needed b-focus: improve test coverage in the repo for <subsystem z> to ensure that <feature x> is covered completely, and fix any outstanding gaps or omissions in that feature as you go (standard references above) c-focus (possibly in a separate repo): You are creating a black box xcuitest to drive a physical phone for testing <feature x>, here is the user specification and known issues, create failing tests for each known issue and an overall robust suite to ensure any user facing or ui issues are caught