8 ms·

My company ran a test and found that they reduce token output on flagship model by something like 2-4x, and that number has been increasing with newer models. I

My company ran a test and found that they reduce token output on flagship model by something like 2-4x, and that number has been increasing with newer models. I suspect the increased subagent usage is driving this trend, because this means we're relying on models to do their own prompt engineering. Yes, they are just text, and can therefore be replaced with good prompting. However, this also means they confer a real benefit: a good set of skills creates a transferable baseline, raising the skill floor and offering a more consistent experience across the organization.

colechristensen 9d ago
I am somewhat confused by takes like this. Of course skills are just prompts, this is the whole point. A skill is just a stored prompt you want to put more information into than you're likely to type out every time you intend to do that thing. Documentation of a business process.
- estearum 9d ago
  What are you confused by? You're saying the same thing they said. They added the additional claim that writing the skills down (apparently) prevents the models from having to self-prompt on the fly and therefore reduces token consumption.
  - mlmonkey 9d ago
    I think skills also count towards the token counts. In the end, everything becomes a 1-D array of input text.
    - estearum 9d ago
      Yes, skills that are actually used count toward token consumption. The question is whether the number of tokens required to achieve a certain behavior/intelligence/quality is equal between you manually providing those tokens via skills versus the model "deriving" the "skills" it needs on-the-fly in order to produce the outcome you want. The claim above is that the former requires far fewer tokens. Also skills only consume tokens when they are used, and part of the value is that the model will dynamically find and disclose only what's needed (assuming the skill is "well-designed").
    - sundarurfriend 9d ago
      Their claim is not about the prompt or skill tokens, it's about output tokens - skills can help the model bypass some thinking tokens or avoid reasoning deadends, and that way reduce output token usage. That's what they seem to have found empirically from their testing. (If it's truly 2x-4x, the time savings in waiting for the output is a pretty nice benefit too.)
- woud420 9d ago
  It's not just a stored prompt, you can attach re-useable scripts to them to offer more determinism. ex: a script that validates that a PR follows exactly the template you want, with a max of N lines per entry. The more determinism you have, the more consistent you can be and the more leverage you can build. (yes I understand that skill calls are non deterministic).
  - bakies 9d ago
    That's just a stored prompt that references a script :)
    - woud420 7d ago
      Yes, that's my point :)
  - verdverm 9d ago
    We do that, but keep the scripts in the code and just tell them in the markdown where the scripts are, same with "references" (docs/) for us. It never made sense to me to put those in a skill dir, many are useful across skills and for humans (many written for humans before agents were a thing) One of the more interesting benefits to skills is that many harnesses now run the inline command(s) in backticks, shortcutting the model needing to make a tool call. This is helpful for deterministically building up context content for the skill before the agent ever sees it. We take this further in some instances and have workflows that (1) does deterministic context gathering (2) invokes an agent (3) processes a file the agent is told to produce. This has made our PR review agent much better and removed it's access to all credential files. We have a step that gathers the diff + existing pull request comments into a .review dir, let the agent process that and create a comments.jsonl, then run a script in a new step to apply the comments against the API
    - woud420 7d ago
      +1 on all of this, it is similar to the pattern I see emerging as well. For example I have open-sourced (let's be honest, it's just my personal tooling of taste) a repo on gh to try and formalize these things. I'd rather not share out of mostly imposter syndrome but essentially it's a scaffolding tool ala "create-react-app" for the stack I generally use. Part of that is to create standardized "docs/{architecture, decisions, contracts, operations}" folders that my harness recognizes in my repos to have a better sense of where was it. Glad others are converging towards similar ideas!

colechristensen 9d ago

I am somewhat confused by takes like this. Of course skills are just prompts, this is the whole point. A skill is just a stored prompt you want to put more information into than you're likely to type out every time you intend to do that thing. Documentation of a business process.

estearum 9d ago
What are you confused by? You're saying the same thing they said. They added the additional claim that writing the skills down (apparently) prevents the models from having to self-prompt on the fly and therefore reduces token consumption.
- mlmonkey 9d ago
  I think skills also count towards the token counts. In the end, everything becomes a 1-D array of input text.
  - estearum 9d ago
    Yes, skills that are actually used count toward token consumption. The question is whether the number of tokens required to achieve a certain behavior/intelligence/quality is equal between you manually providing those tokens via skills versus the model "deriving" the "skills" it needs on-the-fly in order to produce the outcome you want. The claim above is that the former requires far fewer tokens. Also skills only consume tokens when they are used, and part of the value is that the model will dynamically find and disclose only what's needed (assuming the skill is "well-designed").
  - sundarurfriend 9d ago
    Their claim is not about the prompt or skill tokens, it's about output tokens - skills can help the model bypass some thinking tokens or avoid reasoning deadends, and that way reduce output token usage. That's what they seem to have found empirically from their testing. (If it's truly 2x-4x, the time savings in waiting for the output is a pretty nice benefit too.)
woud420 9d ago
It's not just a stored prompt, you can attach re-useable scripts to them to offer more determinism. ex: a script that validates that a PR follows exactly the template you want, with a max of N lines per entry. The more determinism you have, the more consistent you can be and the more leverage you can build. (yes I understand that skill calls are non deterministic).
- bakies 9d ago
  That's just a stored prompt that references a script :)
  - woud420 7d ago
    Yes, that's my point :)
- verdverm 9d ago
  We do that, but keep the scripts in the code and just tell them in the markdown where the scripts are, same with "references" (docs/) for us. It never made sense to me to put those in a skill dir, many are useful across skills and for humans (many written for humans before agents were a thing) One of the more interesting benefits to skills is that many harnesses now run the inline command(s) in backticks, shortcutting the model needing to make a tool call. This is helpful for deterministically building up context content for the skill before the agent ever sees it. We take this further in some instances and have workflows that (1) does deterministic context gathering (2) invokes an agent (3) processes a file the agent is told to produce. This has made our PR review agent much better and removed it's access to all credential files. We have a step that gathers the diff + existing pull request comments into a .review dir, let the agent process that and create a comments.jsonl, then run a script in a new step to apply the comments against the API
  - woud420 7d ago
    +1 on all of this, it is similar to the pattern I see emerging as well. For example I have open-sourced (let's be honest, it's just my personal tooling of taste) a repo on gh to try and formalize these things. I'd rather not share out of mostly imposter syndrome but essentially it's a scaffolding tool ala "create-react-app" for the stack I generally use. Part of that is to create standardized "docs/{architecture, decisions, contracts, operations}" folders that my harness recognizes in my repos to have a better sense of where was it. Glad others are converging towards similar ideas!