bisonbear

7 ms·

1.
I compared Opus 4.8 vs. Opus 5 on 25 of my tasks to see what the difference was(stet.sh)
19 points by bisonbear 21d ago|0 comments
2.
by bisonbear 1mo ago
Reposting my comment from the other thread https://news.ycombinator.com/item?id=49288847 They compare it to 5.6 Terra, however https://cognition.com/frontiercode puts Terra at about 1/2 the price Als
3.
by bisonbear 1mo ago
They compare it to 5.6 Terra, however https://cognition.com/frontiercode puts Terra at about 1/2 the price Also have to compare to the recent Grok 4.6 release, which appears to straight up be better AND cheaper Hard to
4.
by bisonbear 1mo ago
Experimentation makes a lot of sense as agents need to be pleasant to work with, and power users are a willing and able cohort to do that differentiation. Agent capability is only part of the equation, as evidenced by the widespread frustra
5.
by bisonbear 1mo ago
This approach seems fundamentally predicated on being able to evaluate coding agents on your own code by having domain specific evals. With that knowledge, you can trust the routing logic is actually improving/maintaining perf while re
6.
by bisonbear 1mo ago
This is the right direction (Anthropic removing 80% of the system prompt https://claude.com/blog/the-new-rules-of-context-engineering... ) However what I wonder how we make these decisions on what to remove. Certainly
7.
by bisonbear 1mo ago
Thanks for the comment - I’ve actually been working on something to generate evals from private repos, as I 100% agree that public benchmarks are either contaminated (as OpenAI found with SWE-bench Verified), not high quality (as OpenAI fou
8.
by bisonbear 1mo ago
I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost
9.
I compared 5 popular token saving methods in Codex and found that none delivered(stet.sh)
2 points by bisonbear 2mo ago|0 comments
10.
by bisonbear 2mo ago
Also working on a product to build tasks from your own work for testing coding agents. Main thing I would offer is to look carefully at the agent trajectories - they love to figure out ways to cheat. Additionally, consider what "winnin
11.
by bisonbear 2mo ago
To me, this comes down to verifiability. How do we measure the quality of what an agent is doing on our codebase rather than simply measuring task accomplishment? > Verifying quality is orders of magnitude harder than "did the tests
12.
I ran Sonnet 5 vs. Opus 4.8 head to head on 24 tasks to see what's different(stet.sh)
1 points by bisonbear 2mo ago|0 comments
13.
by bisonbear 2mo ago
It depends on what you're measuring. I agree that model resourcefulness is useful, but if you're trying to simulate real user sessions, then Claude looking at upstream Git and fetching the answer directly is somewhat worthless. In
14.
by bisonbear 2mo ago
as a tip - models will always find a way to cheat, you will probably need to impose some restrictions on what they do / are able to access in the sandbox environment see https://cursor.com/blog/reward-hacking-codin
15.
by bisonbear 2mo ago
I've actually been working on a solution for this problem! https://www.stet.sh/ At a high level, it - Mines tasks from your merged PRs/commits - Replays them in Docker containers with different harness settings (c
16.
I evaluated GLM 5.2 against the frontier on tasks from real repos(stet.sh)
2 points by bisonbear 3mo ago|2 comments
17.
by bisonbear 3mo ago
beat saber is the only game I play on it and it's incredible
18.
by bisonbear 3mo ago
The most salient point here is the societal acceptance of consuming slop - somehow we've gotten to a point where the majority of people are ok with mediocre art. I feel that this is a trend that AI has only amplified. The commodificati
19.
by bisonbear 3mo ago
Agree - all of this is based on vibes (I also use TDD based on vibes FWIW). The only way to settle "does TDD / caveman / [insert random skill here] help" is to replay real PRs from your repo and measure quality
20.
I benchmarked Opus 4.8 vs. GPT 5.5 on 2 open source repos(stet.sh)
3 points by bisonbear 4mo ago|0 comments
21.
by bisonbear 4mo ago
> Seems like the progressive disclosure approach is the best for context efficiency; I wound up with a somewhat tight generic AGENTS.md, and the .cursor/rules individual files with glob matching for file names. Cursor honored those
22.
by bisonbear 4mo ago
> we lack common tools to assess and compare This has been bothering me for a while - the entire dev community is running on vibes when talking about AI. We're operating in an old paradigm, thinking that smart and logical additions
23.
by bisonbear 4mo ago
Yes, agree that low n makes overclaiming a real risk with this sort of optimization loop. Low n results can be useful directionally but can't claim superiority without expanding the dataset. If I were running this for a shared repo wit
24.
I used autoresearch to improve my AGENTS.md, measured against real tasks(stet.sh)
8 points by bisonbear 4mo ago|7 comments
25.
A brief investigation into the GPT-5.5 regression claims(stet.sh)
1 points by bisonbear 4mo ago|0 comments
26.
by bisonbear 4mo ago
Yeah, I've found that to be more effective. Going with the example "Always clarify intent before acting" > "Never act without getting intent first", seemingly because telling the agent NOT to do something sometim
27.
by bisonbear 4mo ago
My advice, from doing this myself and reading best practices, would be: - Keep it concise, use progressive disclosure / nested AGENTS.md for information expansion - Give agent the high level repo structure if necessary - Have a "w
28.
by bisonbear 4mo ago
AGENTS.md is extremely important - it's probably the highest leverage thing you can give your agent. It's injected into every turn, and the agents are trained to follow instructions. If anything, I think people are under-investing
29.
by bisonbear 4mo ago
I've been building a tool to do this - build a dataset based on tasks from your repo, then A/B test the agent with whatever change you're making to determine the impact prior to actually shipping it. If you want to check it o
30.
by bisonbear 4mo ago
Not the OP, but I've been thinking about this problem a lot - as devs we're overly reliant on vibes for evaluating coding agents. This is already a problem, and especially so if you're working in an engineering organization w

More ›