warwickmcintosh

7 ms·

1.
by warwickmcintosh 5mo ago
ROCm has improved but the reality is you're still fighting the driver stack more than the models. If you're actually doing local inference on AMD you're spending your time on CUDA compatibility layers, not the AI part. Two li
2.
by warwickmcintosh 5mo ago
Regex and NER both have fun edge cases for redaction. Medical record numbers that look like dates, addresses embedded in prose, account numbers with varying formats. Detection method matters more than people think.
3.
by warwickmcintosh 5mo ago
laziness makes you understand the problem before writing anything. an LLM will happily generate 500 lines for something that needed 20 because it never has to maintain any of it.
4.
by warwickmcintosh 6mo ago
The sanitised optimism problem mentioned upthread is the real gap here. Event stream logging tells you what tools were called and in what order, but it doesn't tell you whether the agent's self-reported outcome matches reality.
5.
by warwickmcintosh 6mo ago
LLM as judge drifts in weird ways if you don't have ground truth to calibrate against. Good that you've got that built in. Would love to see eval stability tracking over time though, same prompt different day sometimes gives diffe
6.
by warwickmcintosh 6mo ago
Biggest change for us has been alert volume. AI-generated code ships faster, which means more deploys per day, which means more things to monitor...
7.
by warwickmcintosh 6mo ago
The "every new model is THE one" cycle is getting a bit old but the Capybara tier thing is actually worth paying attention to.
8.
by warwickmcintosh 6mo ago
The approval layer problem is interesting because the naive approach (approve every action) doesn't scale, but the fully autonomous approach terrifies anyone running these in production...
9.
by warwickmcintosh 6mo ago
I've noticed the time-of-day variance too. My working theory is it's related to load, not model changes. Same prompt at 6am Sydney time (when US is asleep) consistently gets better results than the same prompt at noon. The "i
10.
by warwickmcintosh 6mo ago
Retry logic modeling is a great inclusion most cost estimates miss this completely. In my experience retries account for 15-25% of total cost in production agent systems, especially with tool calling where validation failures trigger re-pro
11.
by warwickmcintosh 6mo ago
The file interaction heatmap is a smart addition. When debugging long agent runs, knowing which files got touched 30+ times vs once tells you immediately where the agent got stuck in a retry loop. One thing I'd love to see: a timeline
12.
by warwickmcintosh 6mo ago
The 71% reduction is interesting but I'd want to see where those tokens are actually going in a typical agent session. In my experience running multi-step coding agents, the git output itself is rarely the bottleneck...