6 ms·

Hey! Disclaimer: I am a Co-Founder of Armature (YC P26) which sells growth services to dev tools. This study is part of our broader work on how to influence co

by screm 13d ago

Hey! Disclaimer: I am a Co-Founder of Armature (YC P26) which sells growth services to dev tools. This study is part of our broader work on how to influence coding agents choices and get products picked. To understand how agents pick tools we measured close to 17k sessions on an environment where agents run exactly like in the real world, on various repositories, talking to different personas (vibe-coder, junior or senior engineers) in different sizes of companies. All the results are now public and we'd love to know what findings surprise you the most, here are a few we found interesting: - Claude Code rarely searches the web while Codex almost always does it and Cursor sits in the middle. - Coding agents disagree more frequently than they agree. - Some players (LangChain, Supabase, Netlify, Paypal, Adyen) are almost always mentioned in their categories but never chosen. - Modifying repository context can change the pick entirely. If you feel like digging, all the traces are there and we probably missed interesting learnings so let us know what you find!

vivifkjo 13d ago
Is there a way to force the usage of a tool for certain tasks? Example: alawys use cli "foobar" to retrieve weather starus. By tool I mean mcp server, cli, etc.
- screm 13d ago
  Not sure I got your question right but if you are wondering for your own coding agent then I guess the answer would be a skill? Here what I meant by "how to influence coding agents choices and get products picked" is from a vendor PoV, making sure any developer x codebase in the world asking for a tool in your category gets your tool recommended and implemented by the coding agent.
- watusername 13d ago
  Obviously you can ask the agent to use a specific tool, but the point of this article is about what they choose when the human on the other end has no opinion/taste/clue.
  - screm 13d ago
    Exactly!
josephg 13d ago
The data was cool. Then I tried to tap on one of the other tabs. “This content is easier to read while full screen!” - Ok I’m game. “Hey this is what makes armature special!” - I don’t care, I’m here to look at data, not onboard onto some random platform. It took ages to find the tiny “skip tour” button, hiding in black on black text. Then it gave me another popup, which I dismissed without reading. Then the full screen modal was visible but it was horizontally misaligned - the left edge was cut off and the right of my phone screen was all white. I closed the tab with great prejudice. (Safari on iOS if you wanna try reproducing it) I’m sure you - or Claude - built something you’re proud of. But I left your website frustrated.
- screm 13d ago
  Hey, thanks for the feedback, the leaderboards aren't displaying well on mobile indeed, we are currently shipping a fix that should help with that. Thanks anyway!
  - kouteiheika 13d ago
    FWIW I had the same reaction to the popups. Immediately closed the tab.
    - screm 13d ago
      Yep makes sense I’m relaxing them
      - screm 13d ago
        It should be better now, including in mobile, thanks both for the feedback!
hbarka 13d ago
> Claude Code rarely searches the web while Codex almost always does I’m trying to understand why they are opposite. I think it is true, I find myself giving a secondary prompt to Claude to “research this” and only then will it fetch. Codex is bang on fetching already.
- arcanemachiner 13d ago
  Must be system prompts and tool instructions guiding the agents differently.
- edoceo 13d ago
  Gemini CLI (at least mine) does web research all the time. I've noticed it hitting my own pages (I have to ask very specific things). I don't have any global or project rules to encourage that behavior.
  - antonvs 13d ago
    That’s a deliberate effort on Google’s part. Integrating AI and search is obviously pretty critical to their business.
- LunaSea 13d ago
  OpenAI is close to Microsoft so I assume that they have preferential and cheap access to the Bing search index. Claude is independent. Gemini should have Google search.
  - screm 13d ago
    [dead]
  - artyomsv 13d ago
    [dead]
- 42piratas 13d ago
  More likely the harness than the model or a search deal. In Claude Code, WebFetch prompts for permission per domain and WebSearch is a separately gated tool, so the cheapest path for the agent is almost always the files already in the repo. Codex has no equivalent friction in front of a fetch. Worth checking against your own permission settings before reading it as a model preference: allowlist a domain and the same agent will reach for it a lot more.
ai_critic 13d ago
Hi. I appreciate that you need to make rent, but if your business is basically "we do growth hacking and SEO tricks on models and get them to use products that aren't actually best for the job", you are scum. You are perpetuating shitty practices that have hurt developers for years now. Part of the reason people use AI is because of how useless search is due to the previous generation doing the same kind of thing you propose. Please do something else with your life.
- N_Lens 13d ago
  First time?
- screm 13d ago
  "we do growth hacking and SEO tricks on models and get them to use products that aren't actually best for the job" -> Well this could be seen the other way around. Today, without proper promotion of services, only incumbents / leaders that are in the models priors (from their training data) are getting chosen. This is ultimately favoring the big generalist players and not the newer or more tailored solutions that benefit from less exposure. I truly think there is something to be done to improve developers' experience too!

by screm 13d ago

vivifkjo 13d ago

Is there a way to force the usage of a tool for certain tasks? Example: alawys use cli "foobar" to retrieve weather starus. By tool I mean mcp server, cli, etc.

screm 13d ago
Not sure I got your question right but if you are wondering for your own coding agent then I guess the answer would be a skill? Here what I meant by "how to influence coding agents choices and get products picked" is from a vendor PoV, making sure any developer x codebase in the world asking for a tool in your category gets your tool recommended and implemented by the coding agent.
watusername 13d ago
Obviously you can ask the agent to use a specific tool, but the point of this article is about what they choose when the human on the other end has no opinion/taste/clue.
- screm 13d ago
  Exactly!

josephg 13d ago

The data was cool. Then I tried to tap on one of the other tabs. “This content is easier to read while full screen!” - Ok I’m game. “Hey this is what makes armature special!” - I don’t care, I’m here to look at data, not onboard onto some random platform. It took ages to find the tiny “skip tour” button, hiding in black on black text. Then it gave me another popup, which I dismissed without reading. Then the full screen modal was visible but it was horizontally misaligned - the left edge was cut off and the right of my phone screen was all white. I closed the tab with great prejudice. (Safari on iOS if you wanna try reproducing it) I’m sure you - or Claude - built something you’re proud of. But I left your website frustrated.

screm 13d ago
Hey, thanks for the feedback, the leaderboards aren't displaying well on mobile indeed, we are currently shipping a fix that should help with that. Thanks anyway!
- kouteiheika 13d ago
  FWIW I had the same reaction to the popups. Immediately closed the tab.
  - screm 13d ago
    Yep makes sense I’m relaxing them
    - screm 13d ago
      It should be better now, including in mobile, thanks both for the feedback!

hbarka 13d ago

> Claude Code rarely searches the web while Codex almost always does I’m trying to understand why they are opposite. I think it is true, I find myself giving a secondary prompt to Claude to “research this” and only then will it fetch. Codex is bang on fetching already.

arcanemachiner 13d ago
Must be system prompts and tool instructions guiding the agents differently.
edoceo 13d ago
Gemini CLI (at least mine) does web research all the time. I've noticed it hitting my own pages (I have to ask very specific things). I don't have any global or project rules to encourage that behavior.
- antonvs 13d ago
  That’s a deliberate effort on Google’s part. Integrating AI and search is obviously pretty critical to their business.
LunaSea 13d ago
OpenAI is close to Microsoft so I assume that they have preferential and cheap access to the Bing search index. Claude is independent. Gemini should have Google search.
- screm 13d ago
  [dead]
- artyomsv 13d ago
  [dead]
42piratas 13d ago
More likely the harness than the model or a search deal. In Claude Code, WebFetch prompts for permission per domain and WebSearch is a separately gated tool, so the cheapest path for the agent is almost always the files already in the repo. Codex has no equivalent friction in front of a fetch. Worth checking against your own permission settings before reading it as a model preference: allowlist a domain and the same agent will reach for it a lot more.

ai_critic 13d ago

Hi. I appreciate that you need to make rent, but if your business is basically "we do growth hacking and SEO tricks on models and get them to use products that aren't actually best for the job", you are scum. You are perpetuating shitty practices that have hurt developers for years now. Part of the reason people use AI is because of how useless search is due to the previous generation doing the same kind of thing you propose. Please do something else with your life.

N_Lens 13d ago
First time?
screm 13d ago
"we do growth hacking and SEO tricks on models and get them to use products that aren't actually best for the job" -> Well this could be seen the other way around. Today, without proper promotion of services, only incumbents / leaders that are in the models priors (from their training data) are getting chosen. This is ultimately favoring the big generalist players and not the newer or more tailored solutions that benefit from less exposure. I truly think there is something to be done to improve developers' experience too!