21 ms·

People always say this about the evals, but I find it hard to have a practical implementation of such a thing where you won’t end up spending 100x the amount of

People always say this about the evals, but I find it hard to have a practical implementation of such a thing where you won’t end up spending 100x the amount of time on the evals than building the skill itself. Like, ok, I have a debugging skill, now how do I make evals except for the most trivial things?

TobTobXX 10d ago
A skill should only document behaviour the LLM didn't/couldn't exhibit on its own. So you take your failed case (eg. working with gdb or whatever), write a skill and then test for that failed case.
- hakunin 10d ago
  There are also skills that help LLM do the thing it can do without the skill, but faster (by cutting out unnecessary discovery). I guess for such skills the fail case is "being slow"? I imagine many fail cases can burn a lot of tokens/usage/time because failing LLMs can be very persistent. Maybe some upper bound (turn count, timeout) would help too.
RicDan 10d ago
You don't. If you're using skills to force the AI to fullfill some must criterias, it's not going to work. Must criterias need deterministic checks -> be it hooks or what not. This is also my biggest gripe with AI. I.e. for specifications, no matter what hype machine I tried, it never fulfilled my criterias, which are: easily verifiable, concise, small specs. Hence I built https://github.com/RicardoMonteiroSimoes/Yamlet https://github.com/RicardoMonteiroSimoes/Yamlet initially for claude code, but then decided to use extend it for pi.dev. I now have a dedicated docker image for pi.dev, that only contains Yamlet plugin, and whenever I work on spec I spin it up. The end result is a .yaml file that easily works in git + git diff, so that I can then proceed with the technical specs-
jurgenburgen 10d ago
Why do you have a debugging skill? Just tell it to read the docs. Skills are for packaging instructions for how to interact with your organizations homebrew process and tools. By definition skills shouldn’t be useful outside of your org because they’re just docs and third party tools already have them for humans.
- stingraycharles 10d ago
  That’s one, very narrow use case of skills.
- well_ackshually 9d ago
  Anthropic must love you. Re-blow hundred of thousands of tokens to relearn how to use your profiler and build system at every debugging attempt.
  - jurgenburgen 9d ago
    Can you elaborate why a debugging skill would save those tokens?
    - hadlock 9d ago
      Rather than teach the agent where grafana is, how to use a sentry trace id to find a otel traceparent, where my ALB is, I'm what clusters do what, which namespace prod is in, what our stack looks like, that thing that looks broken actually isn't, etc etc etc. I just paste in the sentry trace id, a guess of what might be wrong ("i think we overtuned gunicorn again" or "developer bob pushed short sha 123456 and nothing is working") and say "use your investigate skill" then get up and go get a coffee and usually by the time i get back i have an investigate doc filled out from a common template in my notes repo. The agent almost never spends any time spinning it's wheels finding out what the various environments do, where they're located, what our metrics and logging looks like, how to access it etc etc. To be fair it's the only skill i have/use but I got real tired of explaining the same 12 things over and over. Having it document every incident means I have a dense library of every problem we've run into over the last six months which helps identify recurring problems for RCA
mark_l_watson 9d ago
I am starting to wonder if I am doing something wrong: I ignore evals and instead I just try new models or new harnesses (or tweak my own harnesses) by solving problems I want to solve in any case; I just use new tools and form my own subjective opinions of them.
- alexhans 9d ago
  When I say evals I mean the evals you write that verify that your use cases are upheld. Think of it like a regression test for different behaviours/user stories. The idea would be that if you already know what you want from an autonomous system, you don't need to verify manually every time and instead just run these tests to see if there's any regression of any kind. Generally I recommend structure output and evals that are just a plain assertion, if possible. Cheaper, faster, deterministic assertions. Does that make more sense?
deleted 9d ago
[deleted]

TobTobXX 10d ago

A skill should only document behaviour the LLM didn't/couldn't exhibit on its own. So you take your failed case (eg. working with gdb or whatever), write a skill and then test for that failed case.

hakunin 10d ago
There are also skills that help LLM do the thing it can do without the skill, but faster (by cutting out unnecessary discovery). I guess for such skills the fail case is "being slow"? I imagine many fail cases can burn a lot of tokens/usage/time because failing LLMs can be very persistent. Maybe some upper bound (turn count, timeout) would help too.

RicDan 10d ago

You don't. If you're using skills to force the AI to fullfill some must criterias, it's not going to work. Must criterias need deterministic checks -> be it hooks or what not. This is also my biggest gripe with AI. I.e. for specifications, no matter what hype machine I tried, it never fulfilled my criterias, which are: easily verifiable, concise, small specs. Hence I built https://github.com/RicardoMonteiroSimoes/Yamlet https://github.com/RicardoMonteiroSimoes/Yamlet initially for claude code, but then decided to use extend it for pi.dev. I now have a dedicated docker image for pi.dev, that only contains Yamlet plugin, and whenever I work on spec I spin it up. The end result is a .yaml file that easily works in git + git diff, so that I can then proceed with the technical specs-

jurgenburgen 10d ago

Why do you have a debugging skill? Just tell it to read the docs. Skills are for packaging instructions for how to interact with your organizations homebrew process and tools. By definition skills shouldn’t be useful outside of your org because they’re just docs and third party tools already have them for humans.

stingraycharles 10d ago
That’s one, very narrow use case of skills.
well_ackshually 9d ago
Anthropic must love you. Re-blow hundred of thousands of tokens to relearn how to use your profiler and build system at every debugging attempt.
- jurgenburgen 9d ago
  Can you elaborate why a debugging skill would save those tokens?
  - hadlock 9d ago
    Rather than teach the agent where grafana is, how to use a sentry trace id to find a otel traceparent, where my ALB is, I'm what clusters do what, which namespace prod is in, what our stack looks like, that thing that looks broken actually isn't, etc etc etc. I just paste in the sentry trace id, a guess of what might be wrong ("i think we overtuned gunicorn again" or "developer bob pushed short sha 123456 and nothing is working") and say "use your investigate skill" then get up and go get a coffee and usually by the time i get back i have an investigate doc filled out from a common template in my notes repo. The agent almost never spends any time spinning it's wheels finding out what the various environments do, where they're located, what our metrics and logging looks like, how to access it etc etc. To be fair it's the only skill i have/use but I got real tired of explaining the same 12 things over and over. Having it document every incident means I have a dense library of every problem we've run into over the last six months which helps identify recurring problems for RCA

mark_l_watson 9d ago

I am starting to wonder if I am doing something wrong: I ignore evals and instead I just try new models or new harnesses (or tweak my own harnesses) by solving problems I want to solve in any case; I just use new tools and form my own subjective opinions of them.

alexhans 9d ago
When I say evals I mean the evals you write that verify that your use cases are upheld. Think of it like a regression test for different behaviours/user stories. The idea would be that if you already know what you want from an autonomous system, you don't need to verify manually every time and instead just run these tests to see if there's any regression of any kind. Generally I recommend structure output and evals that are just a plain assertion, if possible. Cheaper, faster, deterministic assertions. Does that make more sense?

deleted 9d ago

[deleted]