behat

6 ms·

1.
A look at coding agent benchmarks, and what may be interesting next(tuneloop.io)
3 points by behat 14d ago|0 comments
2.
What would it take to match model intelligence to the task?(tuneloop.io)
1 points by behat 16d ago|0 comments
3.
Benchmarking – Frontier models go out of their way to cheat(tuneloop.io)
2 points by behat 20d ago|0 comments
4.
by behat 23d ago
Daughter loves getting these in the mail, and this paragraph is absolutely right! >> That year, Parton explained that as well as improving child literacy, “it’s the fact they get recognised. They get this little book with their little
5.
by behat 1mo ago
Appreciate the detail in this and the previous post on creating internal benchmarks! Have you all attempted finetuning smaller OSS models on your repos for coding?
6.
by behat 1mo ago
well, at least they are not leaving to be MTS at Anthropic.
7.
Show HN: Tuneloop – a local CLI for analyzing coding agent session transcripts(github.com)
5 points by behat 2mo ago|0 comments
8.
by behat 4mo ago
Thanks for writing this, and look forward to the one on procedural memory. Seems like teams are encoding procedural knowledge in skills repositories, and I wonder if there’s additional utility from an auto created procedural memory layer
9.
by behat 4mo ago
>> Superlog scans your codebase and infrastructure to add new alerts, metrics and dashboards, preventing tricky failure modes and observability decay. This is interesting, and my prior belief here has been that this automates a one ti
10.
by behat 5mo ago
This is a very interesting read on failure modes of AI agents in prod. Curious about this section on the system prompt change: >> After multiple weeks of internal testing and no regressions in the set of evaluations we ran, we felt c
11.
by behat 5mo ago
>> how the runbooks can self heal if results from some steps in the middle are not expected. Yeah this is a very interesting angle. Our primary mechanism here is via agent created auto-memories today. The agent keeps track of the most
12.
by behat 5mo ago
Do you mean how we connect to internal data? Today, you can connect any API endpoint to Relvy, so if you have internal business data / dashboards that you look at while debugging, Relvy can do the same if there's an API for it. Mo
13.
by behat 5mo ago
Nice to see you here, Will! I’d generally recommend using open telemetry for instrumentation so that you keep the option of switching between telemetry vendors. Re: runbooks, yeah even larger teams don’t have good ones to begin with. Relvy
14.
by behat 5mo ago
> They want extension to their agent. If a project tells me I have to use their interface or agentic setup, it's 95% not going to happen Yes, there’s definitely friction there. It may be that the right form factor is that you trigge
15.
by behat 5mo ago
Yes! That boundary between what can be automated and what still needs human judgement has shifted so much this last year. Things like 'go check this dashboard' can now be automated. ROI on runbooks (or good documentation in genera
16.
by behat 5mo ago
heh, I was just about to post the following on your previous comment re: reproducible benchmark results. Thanks for posting the blog. With the docker images that we offer, in theory, people can re-run the benchmark themselves with our agent
17.
by behat 5mo ago
For the debugging workflow you described, we would be a standalone replacement for cursor or other agents. We don't yet write code so can't replace your cursor agents entirely. Re: diffentiation - yes, faster, more accurate and mo
18.
by behat 5mo ago
Thank you :)
19.
by behat 5mo ago
Thanks. Yeah, Cursor / Claude code + MCP is powerful. We differentiate on two fronts, mainly: 1) Greater accuracy with our specialized tools: Most MCP tools allow agents to query data, or run *ql queries - this overwhelms context windo
20.
Launch HN: Relvy (YC F24) – On-call runbooks, automated(relvy.ai)
48 points by behat 5mo ago|25 comments
21.
Ramp: How we made Ramp sheets self-maintaining(twitter.com)
3 points by behat 6mo ago|0 comments
22.
LLM Costs of AI investigating production alerts(relvy.ai)
6 points by behat 6mo ago|1 comments
23.
OpenRCA benchmark – Improving Claude's root cause analysis accuracy by 12 pp(relvy.ai)
12 points by behat 6mo ago|0 comments
24.
by behat 7mo ago
An on-call runbook execution engine - being able to take plain text runbook steps like - look at logs for $service, check for dependency failures - look at so-and-so dashboard. and execute them when an alert fires. We are at https:/&
25.
Can AI debug problem scenarios in the OpenTelemetry demo application?(relvy.ai)
2 points by behat 1y ago|0 comments
26.
by behat 3y ago
Heh. Built a macOS app that does something like this a while ago - https://github.com/bharathpbhat/EssentialApp Back then, I used on device OCR and then sent the text to gpt. I’ve been wanting to re-do this with local
27.
by behat 3y ago
Nice! Built something similar earlier to get fixes from chatgpt for error messages on screen. No voice input because I don't like speaking. My approach then was Apple Computer Vision Kit for OCR + chatgpt. This reminds me to test out O
28.
How GitHub Copilot is getting better at understanding your code(github.blog)
24 points by behat 3y ago|0 comments
29.
by behat 3y ago
Thank you for sharing! The HF docs seem easy to follow. My application is text generation itself, so may have different results.
30.
by behat 3y ago
Thank you for sharing your experience. The linked blog post is great!

More ›