7 ms·
AGENTS.md outperforms skills in our agent evals
- hahahahhaah 8mo agoNext.js sure makes a good benchmark for AI capability (and for clarity... this is not a compliment).
- farhanhubble 8mo ago> Before writing code, first explore the project structure, then invoke the nextjs-doc skill for documentation. Does the model even understand what this line even means?
- xnx 8mo agoAgents.md, skills, MCP, tools, etc. There's going to be a lot of areas explored that may yield no/negative benefit over the fundamentals of a clear prompt.
- thighbaugh 8mo ago> [Specifically Crafted Instructions For Working With A Specific Repository] outperforms [More General Instructions] in our agent evals ------> Captain Obvious Strikes Again! <---------- See the rest the comments for examples pedantic discussions about terms that are ultimately somewhat arbitrary and if anything suggest the singularity will be runaway technobabble not technological progress.
- aaroninsf 8mo agoYou will see another 14% bump in performance if you include in the first 16 lines of the README.md in the project, "Coding agents and LLM, see AGENTS.md"
- rohitghumare 8mo agoThat's why https://agenstskills.com https://agenstskills.com validate every skills
- farhanhubble 8mo agoSo the root cause was the model's indisposition to calling the skills. That seems contrary to what we see with function calling. Models call functions quite reliably most of the time. This is more likely because of the instructions not being clear about what skills are, as this snippet, albeit in isolation, seems to suggest: > Before writing code, first explore the project structure, then invoke the nextjs-doc skill for documentation.
- alex_metacraft 7mo agoI think this experiment has a fundamental flaw in its comparison setup. What they're comparing is: (A) a skill with a short description in the frontmatter, which the agent may or may not decide to invoke, vs. (B) a massive compressed index of documentation paths dumped directly into AGENTS.md, which is always in context. This isn't really "AGENTS.md vs skills." It's "always-in-context with high token count vs. lazy-loaded with a decision point." Of course the always-in-context version wins — you're giving the model way more information upfront. The agent literally can't miss it. That's not a surprising finding, it's almost tautological. The more interesting question they don't address: what did their skill descriptions actually look like? In my experience, the quality of the frontmatter description is the single biggest factor in whether a skill gets invoked. A vague "Documentation lookup skill" will get ignored. A specific "Use this when the user asks about API endpoints, authentication, rate limits, or SDK usage for the Vercel platform" will get picked up reliably. If you wrote equally detailed compressed pointers in AGENTS.md and equally detailed descriptions in skill frontmatter, the gap would likely be much smaller. The real takeaway isn't "skills are worse" — it's "if you don't invest effort in writing good skill descriptions, the agent won't know when to use them."
- ares623 8mo ago2 months later: "Anthropic introduces 'Claude Instincts'"
- EnPissant 8mo agoThis is confusing. TFA says they added an index to Agents.md that told the agent where to find all documentation and that was a big improvement. The part I don't understand is that this is exactly how I thought skills work. The short descriptions are given to the model up-front and then it can request the full documentation as it wants. With skills this is called "Progressive disclosure". Maybe they used more effective short descriptions in the AGENTS.md than they did in their skills?
- NitpickLawyer 8mo agoThe reported tables also don't match the screenshots. And their baselines and tests are too close to tell (judging by the screenshots not tables). 29/33 baseline, 31/33 skills, 32/33 skills + use skill prompt, 33/33 agent.md
- alex_metacraft 7mo agoGood catch on the numbers. 29/33 vs 33/33 is the kind of gap that could easily be noise with that sample size. You'd need hundreds of runs to draw any meaningful conclusion about a 4-point difference, especially given how non-deterministic these models are. This is a recurring problem with LLM benchmarking — small sample sizes presented with high confidence. The underlying finding (always-in-context > lazy-loaded) is probably directionally correct, but the specific numbers don't really support the strength of the claims in the article.
- sally_glance 8mo agoI also thought this is how skills work, but in practice I experienced similar issues. The agents I'm using (Gemini CLI, Opencode, Claude) all seem to have trouble activating skills on their own unless explicitly prompted. Yeah, probably this will be fixed over the next couple of generations but right now dumping the documentation index right into the agent prompt or AGENTS.md works much better for me. Maybe it's similar to structured output or tool calls which also only started working well after providers specifically trained their models for them.
- tottenhm 8mo ago> In 56% of eval cases, the skill was never invoked. The agent had access to the documentation but didn't use it. The agent passes the Turing test...
- cainxinth 8mo agoEven AI doesn’t RTFM
- pylotlight 8mo agoIt learnt from the best
- deadbabe 8mo agoIf humans would just RTFM they wouldn’t need AI.
- Zambyte 8mo agoIf AI would just RTFM it wouldn't need humans.
- slekker 8mo agoBut who would create AI?
- xhcuvuvyc 8mo agoTFM
- measurablefunc 8mo agoAI that don't read the manual.
- Rapzid 8mo agoLegend has it, to this day, TFM has not been read.
- pietz 8mo agoIsn't it obvious that an agent will do better if he internalizes the knowledge on something instead of having the option to request it? Skills are new. Models haven't been trained on them yet. Give it 2 months.
- WA 8mo agoNot so obvious, because the model still needs to look up the required doc. The article glances over this detail a little bit unfortunately. The model needs to decide when to use a skill, but doesn’t it also need to decide when to look up documentation instead of relying on pretraining data?
- sothatsit 8mo agoI believe the skills would contain the documentation. It would have been nice for them to give more information on the granularity of the skills they created though.
- velcrovan 8mo agoRemoving the skill does remove a level of indirection. It's a difference of "choose whether or not to make use of a skill that would THEN attempt to find what you need in the docs" vs. "here's a list of everything in the docs that you might need."
- deleted 8mo ago[deleted]
- rao-v 8mo agoIn a month or three we’ll have the sensible approach, which is smaller cheaper fast models optimized for looking at a query and identifying which skills / context to provide in full to the main model. It’s really silly to waste big model tokens on throat clearing steps
- Calavar 8mo agoI thought most of the major AI programming tools were already doing this. Isn't this what subagents are in Claude code?
- MillionOClock 8mo agoI don't know about Claude Code but in GitHub Copilot as far as I can tell the subagents are just always the same model as the main one you are using. They also need to be started manually by the main agent in many cases, whereas maybe the parent comment was referring about calling them more deterministically?
- jimmydoe 8mo agoCopilot is garbage, even MSFT employees I know all use cc. The only thing useful is you can route cc to use models in copilot sub which corp had a deal from their m365
- MillionOClock 8mo agoOn of the advantages of GitHub Copilot for me is that in terms of billing I find it very generous, depending on how you use it.
- rao-v 8mo agoSub-agents are typically one of the major models but with a specific and limited context + prompt. I’m talking about a small fast model focused on purely curating the skills / MCPs / files to provide to the main model before it kicks off. Basically use a small model up front to efficiently trigger the big model. Sub agents are at best small models deployed by the bigger model (still largely manually triggered in most workflows today)
- jryan49 8mo agoSomething that I always wonder with each blog post comparing different types of prompt engineering is did they run it once, or multiple times? LLMs are not consistent for the same task. I imagine they realize this of course, but I never get enough details of the testing methodology.
- only-one1701 8mo agoThis drives me absolutely crazy. Non-falsifiable and non-deterministic results. All of this stuff is (at best) anecdotes and vibes being presented as science and engineering.
- bluGill 8mo agoThat is my experience. Sometimes the LLM gives good results, sometimes it does something stupid. You tell it what to do, and like a stubborn 5 year old it ignores you - even after it tries it and fails it will do what you tell it for a while and then go back to the thing that doesn't work.
- CuriouslyC 8mo agoI always make a habit of doing a lot of duplicate runs when I benchmark for this reason. Joke's on me, in the time I spent doing 1 benchmark with real confidence intervals and getting no traction on my post, I could have done 10 shitty benchmarks or 1 shitty benchmark and 9x more blogspam. Perverse incentives rule us all.
- sothatsit 8mo agoThis seems like an issue that will be fixed in newer model releases that are better trained to use skills.
- thom 8mo agoYou need the model to interpret documentation as policy you care about (in which case it will pay attention) rather than as something it can look up if it doesn’t know something (which it will never admit). It helps to really internalise the personality of LLMs as wildly overconfident but utterly obsequious.
- deleted 8mo ago[deleted]
- smcleod 8mo agoSounds like they've been using skills incorrectly if they're finding their agents don't invoke the skills. I have Claude Code agents calling my skills frequently, almost every session. You need to make sure your skill descriptions are well defined and describe when to use them and that your tasks / goals clearly set out requirements that align with the available skills.
- velcrovan 8mo agoI think if you read it, their agents did invoke the skills and they did find ways to increase the agents' use of skills quite a bit. But the new approach works 100% of the time as opposed to 79% of the time, which is a big deal. Skills might be working OK for you at that 79% level and for your particular codebase/tool set, that doesn't negate anything they've written here.
- deleted 8mo ago[deleted]
- deleted 8mo ago[deleted]
- joebates 8mo agoIt's still not always reliable. I have a skill in a project named "determine-feature-directory" with a short description explaining that it is meant to determine the feature directory of a current branch. The initial prompt I provide will tell it to determine the feature directory and do other work. Claude will even state "I need to determine the feature directory..." Then, about 5-10% of the time, it will not use the skill. It does use the skill most of the time, but the low failure rate is frustrating because it makes it tough to tell whether or not a prompt change actually improved anything. Of course I could be doing something wrong, but it does work most of the time. I miss deterministic bugs. Recently, I stopped Claude after it skipped using a skill and just said "Aren't you forgetting something?". It then remembered to use the skill. I found that amusing.
- JamesSwift 8mo ago
- jgbuddy 8mo agoAm I missing something here? Obviously directly including context in something like a system prompt will put it in context 100% of the time. You could just as easily take all of an agent's skills, feed it to the agent (in a system prompt, or similar) and it will follow the instructions more reliably. However, at a certain point you have to use skills, because including it in the context every time is wasteful, or not possible. this is the same reason anthropic is doing advanced tool use ref: https://www.anthropic.com/engineering/advanced-tool-use https://www.anthropic.com/engineering/advanced-tool-use, because there's not enough context to straight up include everything. It's all a context / price trade off, obviously if you have the context budget just include what you can directly (in this case, compressing into a AGENTS.md)
- orlandohohmeier 8mo agoI’ve been using symlinked agent files for about a year as a hacky workaround before skils became a thing load additional “context” for different tasks, and it might actually address the issue you’re talking about. Honestly, it’s worked so well for me that I haven’t really felt the need to change it.
- mbm 8mo agoWhat sort of files do you generally symlink in?
- observationist 8mo agoThis is one of the reasons the RLM methodology works so well. You have access to as much information as you want in the overall environment, but only the things relevant to the task at hand get put into context for the current task, and it shows up there 100% of the time, as opposed to lossy "memory" compaction and summarization techniques, or probabilistic agent skills implementations. Having an agent manage its own context ends up being extraordinarily useful, on par with the leap from non-reasoning to reasoning chats. There are still issues with memory and integration, and other LLM weaknesses, but agents are probably going to get extremely useful this year.
- thorum 8mo agoThe article presents AGENTS.md as something distinct from Skills, but it is actually a simplified instance of the same concept. Their AGENTS.md approach tells the AI where to find instructions for performing a task. That’s a Skill. I expect the benefit is from better Skill design, specifically, minimizing the number of steps and decisions between the AI’s starting state and the correct information. Fewer transitions -> fewer chances for error to compound.
- verdverm 8mo agoYea, I am now separating them based on 1. Those I force into the system prompt using rules based systems and "context" 2. Those I let the agent lookup or discover I also limit what gets into message parts, moving some of the larger token consumers to the system prompt so they only show once, most notable read/write_file
- CjHuber 8mo agoThat feels like a stupid article. well of course if you have one single thing you want to optimize putting it into AGENTS.md is better. but the advantage of skills is exactly that you don't cram them all into the AGENTS file. Let's say you had 3 different elaborate things you want the agent to do. good luck putting them all in your AGENTS.md and later hoping that the agent remembers any of it. After all the key advantage of the SKILLs is that they get loaded to the end of the context when needed
- deleted 8mo ago[deleted]
- sheepscreek 8mo agoIt seems their tests rely on Claude alone. It’s not safe to assume that Codex or Gemini will behave the same way as Claude. I use all three and each has its own idiosyncrasies.
- verdverm 8mo agoI've done very similar things with my custom agent that uses Gemini and have gotten very similar results. Working on the evals to back that claim up
- newzino 8mo ago[flagged]
- jstummbillig 8mo agoWhy could you not have a combination of both?
- verdverm 8mo agoYou can and should, it works better than either alone
- mococa 8mo agoAh nice… vercel is vibecoded
- heliumtera 8mo agoweb people opted into react, dude. that says a lot. they used prisma to handle their database interactions. they preached tRPC and screamed TYPE SAFETY!!! you really think these guys will ever again touch the keyboard to program? they despise programming.
- dca88 8mo agoThis. I read this article and it pains me to see the amount of manpower put into doing anything but actually getting work done.
- BenoitEssiambre 8mo agoWouldn't this have been more readable with a \n newline instead of a pipe operator as a seperator? This wouldn't have made the prompt longer.
- ChrisArchitect 8mo agoTitle is: AGENTS.md outperforms skills in our agent evals
- heliumtera 8mo agoyou are telling me that a markdown saying: *You are the Super Duper Database Master Administrator of the Galaxy* does not improve the model ability reason about databases?
- verdverm 8mo agoThis largely mirrors my experience building my custom agent 1. Start from the Claude Code extracted instructions, they have many things like this in there. Their knowledge share in docs and blog on this aspect are bar none 2. Use AGENTS.md as a table of contents and sparknotes, put them everywhere, load them automatically 3. Have topical markdown files / skills 4. Make great tools, this is still opaque in my mind to explain, lots of overlap with MCP and skills, conceptually they are the same to me 5. Iterate, experiment, do weird things, and have fun! I changed read/write_file to put contents in the state and presented in the system prompt, same for the agents.md, now working on evals to show how much better this is, because anecdotally, it kicks ass
- aktau 8mo ago> I changed read/write_file to put contents in the state and presented in the system prompt, same for the agents.md, now working on evals to show how much better this is, because anecdotally, it kicks ass. Can you detail this a bit more? Do you put the actual contents of the file in the system prompt? Forever?
- meatcar 8mo agoWhat if instead of needing to run a codemod to cache per-lib docs locally, documentation could be distributed alongside a given lib, as a dev dependency, version locked, and accessible locally as plaintext. All docs can be linked in node_modules/.docs (like binaries are in .bin). It would be a sort of collection of manuals. What a wonderful world that would be.
- tobyjsullivan 8mo agoSounds a bit like man pages. I think you’re onto something.
- chr15m 8mo agoI'm not sure if this is widely known but you can do a lot better even than AGENTS.md. Create a folder called .context and symlink anything in there that is relevant to the project. For example READMEs and important docs from dependencies you're using. Then configure your tool to always read .context into context, just like it does for AGENTS.md. This ensures the LLM has all the information it needs right in context from the get go. Much better performance, cheaper, and less mistakes.
- d3m0t3p 8mo agoYea but the goal it not to bloat the context space. Here you "waste" context by providing non usefull information. What they did instead is put an index of the documentation into the context, then the LLM can fetch the documentation. This is the same idea that skills but it apparently works better without the agentic part of the skills. Furthermore instead of having a nice index pointing to the doc, They compressed it.
- bmitc 8mo agoWhat does it mean to waste context?
- bagels 8mo agoThe context window is finite. You can easily fill it with documentation and have no room left for the code and question you want to work on. It also means more tokens sent with every request, increasing cost if you're paying by the token.
- therealpygon 8mo agoContext quite literally degrades performance of attention with size in non-needle-in-haystack lookups in almost every model to varying degrees. Thus to answer the question, the “waste” is making the model dumber unnecessarily in an attempt to make it smarter.
- PKop 8mo agoThink of context switching when you yourself are programming. You can only hold some finite amount of concepts in your head at one time. If you have distractions, or try to focus on too many things at once, your ability to reason about your immediate problem degrades. Think also of legacy search engines: often, a more limited and focused search query vs a query that has too many terms, more precisely maps to your intended goal. LLM's have always been at any time limited in the amount of tokens it can process at one time. This is increasing, but one problem is chat threads continually increase in size as you send messages back and forth because within any session or thread you are sending the full conversation to the LLM every message (aside from particular optimizations that compact or prune this). This also increases costs which are charged per token. Efficiency of cost and performance/precision/accuracy dictates using the context window judiciously.
- thevinter 8mo agoI'm a bit confused by their claims. Or maybe I'm misunderstanding how Skills should work. But from what I know (and the small experience I had with them), skills are meant to be specifications for niche and well defined areas of work (i.e. building the project, running custom pipelines etc.) If your goal is to always give a permanent knowledge base to your agent that's exactly what AGENTS.md is for...
- deleted 8mo ago[deleted]
- meeech 8mo agoquestion: anyone recognize that eval UI or is it something they made in-house?
- songodongo 8mo ago> When it needs specific information, it reads the relevant file from the .next-docs/ directory. I guess you need to make sure your file paths are self-explanatory and fairly unique, otherwise the agent might bring extra documentation into the context trying to find which file had what it needed?
- w10-1 8mo agoThe key finding is that "compression" of doc pointers works. It's barely readable to humans, but directly and efficiently relevant to LLM's (direct reference -> referent, without language verbiage). This suggests some (compressed) index format that is always loaded into context will replace heuristics around agents.md/claude.md/skills.md. So I would bet this year we get some normalization of both the indexes and the referenced documentation (esp. matching terms). Possibly also a side issue: API's could repurpose their test suites as validation to compare LLM performance of code tasks. LLM's create huge adoption waves. Libraries/API's will have to learn to surf them or be limited to usage by humans.
- ai-christianson 8mo agoThey say compressed... but isn't this just "minified"?
- ethmarks 8mo agoMinification is still a form of compression, it just leaves the file more readable than more powerful compression methods (such as ZIP archives).
- throwaway314155 8mo agoI'd say minification/summarization is more like a lossy, semantic compression. This is only relevant to LLM's and doesn't really fit more classical notions of compression. Minification would definitely be a clearer term, even if compression _technically_ makes sense.
- jcheng 8mo agoWould’ve been perfectly readable and no larger if they had used newline instead of pipe.
- postalcoder 8mo agoThat's not the only useful takeaway. I found this to be true: > "Explore project first, then invoke skill" [produces better results than] "You MUST invoke the skill". I recently tried to get Antigravity to consistently adhere to my AGENTS.md (Antigravity uses GEMINI.md). The agent consistently ignored instructions in GEMINI.md like: - "You must follow the rules in [..]/AGENTS.md" - "Always refer to your instructions in [..]/AGENTS.md" Yet, this works every time: "Check for the presence of AGENTS.md files in the project workspace." This behavior is mysterious. It's like how, in earlier days, "let's think, step by step" invoked chain-of-thought behavior but analogous prompts did not.
- keeganpoppen 8mo agoi dont know why, but this just feels like the most shallow “i compare llms based on the specs” kind of analysis you can get… it has extreme “we couldn’t get the llm to intuit what we wanted to do, so we assumed that it was a problem with the llm and we overengineered a way to make better prompts completely by accident” energy…
- smrtinsert 8mo agoAre people running into mismatched code vs project a lot? I've worked on python and java codebases with claude code and have yet to run into a version mismatch issue. I think maybe once it got confused on the api available in python, but it fixed it by itself. From other blog posts similar to this it would seem to be a widespread problem, but I have yet to see it as a big problem as part of my day job or personal projects.
- AndyNemmity 8mo agoMy experience agrees with this. Which is why I use a skill that is a command, that routes requests to agents and skills.
- jascha_eng 8mo agoThis does not normalize for tokens used if their skill description was as large as the docs index and contained all the reasons the LLM might want to use the skill, it likely performs much better than just one sentence as well.
- deleted 8mo ago[deleted]
- gpm 8mo agoCompressing information in AGENTS.md makes a ton of sense, but why are they measuring their context in bytes and not tokens!?
- denolfe 8mo agoPreSession Hook from obra/superpowers injects this along with more logic for getting rid of rationalizing out of using skills: > If you think there is even a 1% chance a skill might apply to what you are doing, you ABSOLUTELY MUST invoke the skill. IF A SKILL APPLIES TO YOUR TASK, YOU DO NOT HAVE A CHOICE. YOU MUST USE IT. While this may result in overzealous activation of skills, I've found that if I have a skill related, I _want_ to use it. It has worked well for me.
- stingraycharles 8mo agoI always say “invoke your <x> skill to do X. then invoke your <y> skill to do Y. “ works pretty well
- tanishqkanc 8mo agothis is only gonna be an issue until the next gen models where the labs will aggressively post train the models to proactively call skills
- minimal_action 8mo agoIt's very interesting but presenting success rates without any measure of the error, or at least inline details about the number of iterations is unprofessional. Especially for small differences or when you found the "same" performance.
- onnimonni 8mo agoWould someone know if their eval tests are open source and where I could find them? Seems useful for iterating on Claude Code behaviour.
- JamesSwift 8mo agoI also was looking for specific info on the evals, because I wanted to see if they were separately confirming that shoving the skills into the main context didnt degrade the non-skills evals. Thats the other side of skills other than ability to the thing, they dont pollute the main context window with unnecessary information.
- holocen 8mo agoPrompted and built a bit of an extension of skills.sh with https://passivecontext.dev https://passivecontext.dev it basically just takes the skill and creates that "compressed" index. Still have to install the skill and all that, but might give others a bit of a short cut to experiment with.
- armcat 8mo agoFirstly this is great work from Vercel - I am especially impressed with the evals setup (evals are the most undervalued component in any project IMO). Secondly the result is not surprising and I’ve seen consistently the increase in performance when you always include an index (or in my case, Table of Contents as a json structure) in your system prompt. Applying this outside of coding agents (like classic document retrieval) also works very well!
- underlines 8mo agoOh got, this scales bad and bloats your context window! Just create an MCP server that does embedding retrieval or agentic retrieval with a sub agent on your framework docs. Finally add an instruction to AGENT.md to look up stuff using that MCP.
- someguyiguess 8mo agoThe problem is that Agents.md is only read on initial load. Once context grows too large the agent will not reload the md file and loses / forgets the info from Agents.md.
- bushbaba 8mo agoWhy you try and avoid re using the same session beyond the initial task or two
- taberiand 8mo agoOther comments suggest that the Agents.md is read into the system prompt and never leaves the context. But it's better to avoid excessive context regardless
- remify 8mo agoThat's the thing that bothers me here. They loaded the doc of course it will work but as your project grows you won't be able to put all your documentation in there (at least with current context handling). Skills are still very much relevant on big and diverse projects.
- devonkelley 8mo ago[flagged]
- deaux 8mo agoThis comment instantly set off my LLM alarm bells. Went into the profile, and guess what: next comment (not a one-liner) [0] on a completely different topic was posted 35 seconds later. And includes the classic "aren't just A. They're B.". Why are you doing this? Karma? 8 years old account and first post 3 days ago is a Show HN shilling your "AI agent" SaaS with a boatload of fake comments? [1] Pinging tomhow [0] https://news.ycombinator.com/item?id=46820417 https://news.ycombinator.com/item?id=46820417 [1] https://news.ycombinator.com/item?id=46782579 https://news.ycombinator.com/item?id=46782579
- SubiculumCode 8mo agoWow.
- devonkelley 8mo ago[dead]
- devonkelley 8mo agoDude I am not AI. Real human. Just started on HN.
- devonkelley 8mo ago[dead]
- deaux 8mo agoJust happen to post 2 comments within 30s on completely different posts, having all of the hallmarks of LLM output? With your other post being full of green accounts? With no account activity for 8 years? You're clearly posting comments straight from an LLM. It's not realistic to read the other post to a significant degree, think about it, and then type all of this: > The prompt injection concerns are valid, but I think there's a more fundamental issue: agents are non-deterministic systems that fail in ways that are hard to predict or debug. Security is one failure mode. But "agent did something subtly wrong that didn't trigger any errors" is another. And unlike a hacked system where you notice something's off, a flaky agent just... occasionally does the wrong thing. Sometimes it works. Sometimes it doesn't. Figuring out which case you're in requires building the same observability infrastructure you'd use for any unreliable distributed system. > The people running these connected to their email or filesystem aren't just accepting prompt injection risk. They're accepting that their system will randomly succeed or fail at tasks depending on model performance that day, and they may not notice the failures until later. Within 35 seconds of posting this one. And it just happens to have all LLM hallmarks there are. We both know it, you're on HN, people here aren't fools.
- bandrami 8mo agoBlackbox oracles make bad workflows, and tend to produce a whole lot of cargo culting. It's this kind of opacity (why does the markdown outperform agents? there's no real way to find out, even with a fully open or house model because the nature of the beast is that the execution path in a model can't be predicted) that makes me shy away from saying LLMs are "just another tool". If I can't see inside it -- and if even the vendor can't really see inside of it -- there's something fundamentally different.
- wakeless 8mo agoI did a similar set of evals myself utilising the baseline capabilities that Phoenix (elixir) ships with and then skillified them. Regularly the skills were not being loaded and thus not utilised. The outputs themselves were fine. This suggested that at some stage through the improvements of the models that baseline AGENTS.md had become redundant.
- msp26 8mo agoThis doesn't surprise me. I have a SKILL.md for marimo notebooks with instructions in the frontmatter to always read it before working with marimo files. But half the time Claude Code still doesn't invoke it even with me mentioning marimo in the first conversation turn. I've resorted to typing "read marimo skill" manually and that works fine. Technically you can use skills with slash commands but that automatically sends off the message too which just wastes time. But the actual concept of instructions to load in certain scenarios is very good and has been worth the time to write up the skill.
- shinhyeok 8mo agoBut aren't you guys released skills.sh?
- whinvik 8mo agoWhen we were trying to build our own agents we put quite a bit of effort on evals which was useful. But switching over to using coding agents we never did the same. Feels like building an eval set will be an important part of what engg orgs do going forward.
- motoboi 8mo agoModels are not AGI. They are text generators forced to generate text in a way useful to trigger a harness that will produce effects, like editing files or calling tools. So the model won’t “understand” that you have a skill and use it. The generation of the text that would trigger the skill usage is made via Reinforcement Learning with human generated examples and usage traces. So why don’t the model use skills all the time? Because it’s a new thing, there is not enough training samples displaying that behavior. They also cannot enforce that via RL because skills use human language, which is ambiguous and not formal. Force it to use skills always via RL policy and you’ll make the model dumber. So, right now, we are generating usage traces that will be used to train the future models to get a better grasp of when to use skills not. Just give it time. AGENTS.md, on the other hand, is context. Models have been trained to follow context since the dawn of the thing.
- bzGoRust 8mo agoI completed agree with your point
- DanOpcode 8mo agoWhat's RL?
- jacobkg 8mo agoReinforcement Learning https://en.wikipedia.org/wiki/Reinforcement_learning https://en.wikipedia.org/wiki/Reinforcement_learning
- wahnfrieden 8mo agoReinforcement learning
- anal_reactor 8mo ago> Models are not AGI https://en.wikipedia.org/wiki/GNU/Linux_naming_controversy https://en.wikipedia.org/wiki/GNU/Linux_naming_controversy
- vidarh 8mo ago
- carterschonwald 8mo agostatic linking va dynamic but we dont know the actual config and setup. and also the choice of totally changes the problem
- tdiff 8mo agoIs not that model-dependant? Skimmed through, but did not find which model the tests were run with.
- rcarmo 8mo agoEverything outperforms skills if the system prompt doesn’t prioritize them. No news here.
- psychoslave 8mo agoOver the last week I went with a bigger dig on using agent mode et work, and my experiment align with this observation. The first thing that surprising to me is how much the default tuning are leaned toward laudative stances, the user is always absolutely right, what was done is solving everything expected. But actually no, not a single actual check was done, a tone of code was produced but the goal is not at all achieved and of course many regressions now lure in the code base, when it's not straight breaking everything (which is at least less insidious). The thing that is surprising to me, is that it can easily drop thousands of lines of tests, and then it can be forced to loop over these tests until it succeed. In my experiments it still drop far too much noise code, but at least the burden of checking if it looks like it makes any sense is drastically reduced.
- hu3 8mo agoThat's my observation too. And I have been trying to improve the framework and abstractions/types to reduce the lines of code required for LLMs to create features in my web app. Did the LLM really needed to spit 1k lines for this feature? Could I create abstractions to make it feasible in under 300 lines? Of course there's cost and diminishing returns to abstractions so there are tradeoffs.
- underdeserver 8mo agoI don't think you can really learn from this experiment unless you specify which models you used, if you tried it against at least 3 frontier models, if you ran each eval multiple times, and what prompts you tried. These things are non-deterministic across multiple axes.
- user3939382 8mo agoI’m working on an AGI model that will make the discussion of skills look silly. Skills strikes in the right direction in some sense but it’s an extremely weak 1% echo of what’s actually needed to solve this problem.
- fatheranton 8mo ago[dead]
- deleted 8mo ago[deleted]
- epolanski 8mo agoI'm working on stuff in a similar space. I need to evaluate how do different project scaffolding impacts the results of Claude Code/Opencode (either with Anthropic models or third party) for agentic purpose. But I am unsure on how should I be testing and it's not very clear how did Vercel proceeded here.
- guluarte 8mo agoIn my experience, agents only follow the first two or three lines of AGENTS.md + message. As the context grows, they start following random rules and ignoring others.
- j45 8mo agoDon't want to dither the topic, but could skills not just be sub agents in this contextualization? There is a lot of language floating around what effectively groups of text files put together in different configurations, or selected reliably.
- micimize 8mo agoMeasuring in terms of KB is not quite as useful as it seems here IMO - this should be measured in terms of context tokens used. I ran their tool with an otherwise empty CLAUDE.md, and ran `claude /context`, which showed 3.1k tokens used by this approach (1.6% of the opus context window, bit more than the default system prompt. 8.3% is system tools). Otherwise it's an interesting finding. The nudge seems like the real winner here, but potential further lines of inquiry that would be really illuminating: 1. How do these approaches scale with model size? 2. How are they impacted by multiple such clauses/blocks? Ie maybe 10 `IMPORTANT` rules dilute their efficacy 3. Can we get best of both worlds with specialist agents / how effective are hierarchical routing approaches really? (idk if it'd make sense for vercel specifically to focus on this though)
- sghiassy 8mo agoN00b Question - how do you measure performance for AI agents like the way they did in this article? Are there frameworks to support this type of work?
- robertheadley 8mo agoI will have to look into this this weekend. Antigravity is my current favorite agentic IDE and I have been having problems getting it to explicitly follow my agent.md settings. If I remind it, it will be go, "oh yes, ok, sure." then do it, but the whole point is that I want to optimize my time with the agent.
- embedding-shape 8mo agoI feel like all agents currently do better if you explicitly end with "Remember to follow AGENTS.md", even if that's automatically injected into the context. Seems the same across all I'm using.
- killerstorm 8mo agoinb4 people re-discover RAG, re-branding it as a parallel speculative data lookup