6 ms·

Can someone explain what the hell is going on here? Do websites want to prevent automated tooling, as indicated by everyone putting everything behind Cloudfare

Can someone explain what the hell is going on here? Do websites want to prevent automated tooling, as indicated by everyone putting everything behind Cloudfare and CAPTCHAs since forever, or do websites want you to be able to automate things? Because I don't see how you can have both. If I'm using Selenium it's a problem, but if I'm using Claude it's fine??

jmalicki 7mo ago
In early experiments with the Claude Chrome extension Google sites detected Claude and blocked it too. Shrug
zadikian 7mo ago
Remember when many websites had quite open public APIs? Over time this became less common, and existing things like FB added more limitations.
BeefySwain 7mo ago
Also, as someone who has tried to build tools that automate finding flights, The existing players in the space have made it nearly impossible to do. But now Google is just going to open the door for it?
parhamn 7mo ago
Is the website Stripe or NYTimes?
manveerc 7mo ago
In my opinion sites that want agent access should expose server-side MCP, server owns the tools, no browser middleman. Already works today. Sites that don’t want it will keep blocking. WebMCP doesn’t change that. Your point about selenium is absolutely right. WebMCP is an unnecessary standard. Same developer effort as server-side MCP but routed through the browser, creating a copy that drifts from the actual UI. For the long tail that won’t build any agent interface, the browser should just get smarter at reading what’s already there. Wrote about it here: https://open.substack.com/pub/manveerc/p/webmcp-false-economy-server-side-mcp-browser-apis?r=1a5vz&utm_medium=ios https://open.substack.com/pub/manveerc/p/webmcp-false-econom...
- sarkarsh 7mo ago
  [dead]
- arjunchint 7mo ago
  So... an API? Most sites don't want to expose APIs or care enough about setup and maintenance of said API.
  - manveerc 7mo ago
    Are you asking if Agents should use API?
nudpiedo 7mo ago
They will wish that you use an official API, follow the funnel they settled for you, and make purchases no matter how
loveparade 7mo ago
Not fine if you use Claude. But it's fine if you are Google Flights and the user uses Gemini. The paid version of course.
victorbjorklund 7mo ago
They wanna let you use the service the way they want. An e-commerce? Wanna automate buying your stuff - probably something they wanna allow under controlled forms Wanna scrape the site to compare prices? Maybe less so.
- candiddevmike 7mo ago
  A brave new world for fraud and returns. Also I just recently noticed Chrome now has a Klarna/BNPL thing as a built in payments option that I never asked for...
  - kylecazar 7mo ago
    Yeah it's a payment method they added to Google Pay (Google Wallet? I don't know anymore). You can turn it off in autofill settings.
chrash 7mo ago
i’m seeing this at my corporate software job now. that service that you used to have security and product approval for to even read their Swagger doc has an MCP server you can install with 2 clicks.
- politelemon 7mo ago
  Sometimes, it gets added there without your consent.
buzzerbetrayed 7mo ago
Why should a browser care about how websites want you to use them?
OsrsNeedsf2P 7mo ago
These are obviously different people you're talking about here
nojs 7mo ago
It’s weirder than that. There is a surge of companies working on how to provide automated access to things like payments, email, signup flows, etc to *Claw.
akersten 7mo ago
I'm old enough to remember discussions around the meaning of `User-Agent` and why it was important that we include it in HTTP headers. Back before it was locked to `Chromium (Gecko; Mozilla 4.0/NetScape; 147.01 ...)`. We talked about a magical future where your PDA, car, or autonomous toaster could be browsing the web on your behalf, and consuming (or not consuming) the delivered HTML as necessary. Back when we named it "user agent" on purpose. AI tooling can finally realize this for the Web, but it's a shame that so many companies who built their empires on the shoulders of those visionaries think the only valid way to browse is with a human-eyeball-to-server chain of trust.
- cameldrv 7mo ago
  Me too but it died when ads became the currency of the web. If the reason the site exists is to use ads, they’re not going to let you use an user agent that doesn’t display the ads.
  - akersten 7mo ago
    > If the reason the site exists is to use ads, they’re not going to let you use an user agent that doesn’t display the ads. They've been giving it the old college try for the better part of two decades and the only website I've had to train myself not to visit is Twitch, whose ads have invaded my sightline one time too many, and I conceded that particular adblocking battle. I don't get the sense that it's high on the priority list for most sites out there (knock on wood).
    - snackerblues 7mo ago
      Same, I just don't use Twitch when possible. Most streamers rehost their VODs on Youtube which has a better player anyway.
    - diacritical 7mo ago
      People who block ads are a minority. Sites that serve heavy content like video would care if someone wastes their resources but blocks ads, but why would a site that serves a few KBs of text spend the resources on blocking such users or making the ads beat the ad blocker in a tiresome cat and mouse game? Those users could even share or recommend the site to someone else who doesn't use ad blockers, so it actually makes sense to not try to battle ad blockers if you want to make your site more popular. This makes sense for sites that rely on network effects, like forums or classified ad sites and so on. Unless they have a near monopoly or some really valuable content, they would benefit financially if they let people block their ads. I can't back that up with data or anything, but it makes sense to me.
dawnerd 7mo ago
And what site is going to open their api up to everyone? Document endpoints already exist, why make it more complicated.
moron4hire 7mo ago
Oh, that's an easy one. LLMs have made people lose their god damned minds. It makes sense when you think about it as breaking a few eggs to get to the promised land omelette of laying off the development staff.
avaer 7mo ago
In a nutshell: Google wants your websites to be more easily used by the agents they are putting in the browser and other products. They own the user layer and models, and get to decide if your product will be used. Think search monopoly, except your site doesn't even exist as far as users are concerned, it's only used via an agent, and only if Google allows. The work of implementing this is on you. Google is building the hooks into the browser for you to do it; that's WebMCP. It's all opaque; any oopsies/dark patterns will be blamed on the AI. The profits (and future ad revenue charged for sites to show up on the LLM's radar) will be claimed by Google. The other AI companies are on board with this plan. Any questions?
- solaire_oa 7mo ago
  We should definitely feel trepidation at the prospects of any LLM guided browser, in addition to WebMCP (e.g. Claude for Chrome enters the same opaque LLM-controlled/deferred decision process, OpenClaw etc). Just one example: Prompting the browser to "register example.com" means that Google/Anthropic gets to hustle registrars for SEO-style priority. Using countermeasures like captcha locks you out of the LLM market. Google's incentive to allow you to shop around via traditional web search is decreased since traditional ads won't be as lucrative (businesses will catch on that blanket targeted ads aren't as effective as a "referral" that directs an LLM to sign-up/purchase/exchange something directly)... expect web search quality to decline, perhaps intentionally. The only way to combat this, as far as I can conceptualize, is with open models, which are not yet as good as private ones, in no small part due to the extraordinary investment subsidization. We can hope for the bubble to pop, but plan for a deader Internet. Meanwhile, trust online, at large, begins to evaporate as nobody can tell what is an LLM vs a human-conducted browser. The Internet at large is entering some very dark waters.
- oefrha 7mo ago
  The irony is Google properties are more locked down than ever. When I use a commercial VPN I get ReCAPTCHA’ed half of the time doing every single Google search; and can’t use YouTube in Incognito sometimes, “Sign in to confirm you’re not a bot”.
  - 7mo ago
medi8r 7mo ago
Both. I imagine if using this there is a tell (e.g. UA or other header). Sites can just block unauthenticated sessions using it but allow it to be used when they know who.
SilverElfin 7mo ago
I feel like this is a way to ultimately limit the ability to scrape but also the ability to use your own AI agent to take actions across the internet for you. Like how Amazon doesn’t let your agent to shop their site for you, but they’ll happily scrape every competitor’s website to enforce their anti competitive price fixing scheme. They want to allow and deny access on their terms. WebMCP will become another channel controlled by big tech and it’ll come with controls. First they’ll lure people to use this method for the situations they want to allow, and then they’ll block everything else.
aragonite 7mo ago
> Do websites want to prevent automated tooling, as indicated by everyone putting everything behind Cloudfare and CAPTCHAs since forever, or do websites want you to be able to automate things? Because I don't see how you can have both. The proposal (https://docs.google.com/document/d/1rtU1fRPS0bMqd9abMG_hc6K9OAI6soUy3Kh00toAgyk/edit?tab=t.0 https://docs.google.com/document/d/1rtU1fRPS0bMqd9abMG_hc6K9...) draws the line at headless automation. It requires a visible browsing context. > Since tool calls are handled in JavaScript, a browsing context (i.e. a browser tab or a webview) must be opened. There is no support for agents or assistive tools to call tools "headlessly," meaning without visible browser UI.
- Intermernet 7mo ago
  That really just increases the processing power required to automate it. VM running Chrome to a virtual frame buffer, point agent at frame buffer, automate session. It's clunky, but probably not that much more memory intensive than current browser automation. You could probably ditch the frame buffer as well, except for giving the browser something to write out to. It can probably be /dev/null.
bear3r 7mo ago
different threat model. cloudflare blocks automation that pretends to be human -- scraping, fake clicks, account stuffing. webmcp is a site explicitly publishing 'here are the actions i sanction.' you can block selenium on login and expose a webmcp flight search endpoint at the same time. one's unauthorized access, the other's a published api.
maximinus_thrax 7mo ago
> Do websites want to prevent automated tooling, as indicated by everyone putting everything behind Cloudfare and CAPTCHAs since forever, Not if they don't want their rankings to tank. Now you'll need to make your website machine friendly while the lords of walled gardens will relentlessly block any sort of 'rogue' automated agent from accessing their services.
fasbiner 7mo ago
I can deeply, deeply relate. X and Bluesky are both going nuts with ai and ai scams, but _both_ of them banned an advertising account because we were... using a bot to automate behavior because their APIs are only a subset of functionality. Their vision is a world where they use all the automation regardless of safety or law, and we have to jump through extra hoops and engage in manual processes with AI that literally doesn't have the tool access to do what we need and will not contact a human.
joshuanapoli 7mo ago
WebMCP should be a really easy way to add some handy automation functionality to your website. This is probably most useful for internal applications.
est 7mo ago
>Can someone explain what the hell is going on here? Someone at Chromium team is launching rapidly for an promotion
dokdev 7mo ago
I was also thinking about more or less the same thing with APIs and MCPs. The companies that didn't have any public apis are now exposing MCPs. That, to me is quite interesting. Maybe it is the FOMO effect.
sleight42 7mo ago
I can't see walled garden platforms or any website that monetizes based on ads offering WebMCP. Agents using their site represent humans who aren't.
notatoad 7mo ago
as a website operator, i want my website to not experience downtime and unreliability because of usage rates that exceed the rate at which humans load pages, and i want to not be defrauded. if you want to access my website using automated tools, that's fine. but if there's a certain automated tool that is consistently used to either break the site or attempt to defraud me, i'm going to do my best to block that tool. and sometimes that means blocking other, similar tools. if the webMCP client in chrome behaves in a reasonable way that prevents abuse, then i don't see a problem with it. if scammers discover they can use it to scam, then websites will block it too.
DrScientist 7mo ago
Obviously if you wanted people to book flights with a bot then you could have provided a public API for that long ago. I think potentially the subtlety here is a sort of cooperative mode - the computer filling out a lot of the forms and doing the grunt, but it's important that the human is still in the loop - so they need to be able to share a UI with the agent. Hence a agent friendly web page, rather than just an API.
aubanel 7mo ago
I think I have one explanation why for a website, exposing an MCP servers AND having captchas can make sense. - an agent loading the real page is waste for the server, because the data sent is a few megavytes, and you don't have the usual returns of an user seeing your ads - BUT API requests (or here, MCP) are much lighter, a few dozen kB, so that makes the ROI positive again At least that's my view : please tell me, anyone, if that reason doesn't make sense!

jmalicki 7mo ago

In early experiments with the Claude Chrome extension Google sites detected Claude and blocked it too. Shrug

zadikian 7mo ago

Remember when many websites had quite open public APIs? Over time this became less common, and existing things like FB added more limitations.

BeefySwain 7mo ago

Also, as someone who has tried to build tools that automate finding flights, The existing players in the space have made it nearly impossible to do. But now Google is just going to open the door for it?

parhamn 7mo ago

Is the website Stripe or NYTimes?

manveerc 7mo ago

In my opinion sites that want agent access should expose server-side MCP, server owns the tools, no browser middleman. Already works today. Sites that don’t want it will keep blocking. WebMCP doesn’t change that. Your point about selenium is absolutely right. WebMCP is an unnecessary standard. Same developer effort as server-side MCP but routed through the browser, creating a copy that drifts from the actual UI. For the long tail that won’t build any agent interface, the browser should just get smarter at reading what’s already there. Wrote about it here: https://open.substack.com/pub/manveerc/p/webmcp-false-economy-server-side-mcp-browser-apis?r=1a5vz&utm_medium=ios https://open.substack.com/pub/manveerc/p/webmcp-false-econom...

sarkarsh 7mo ago
[dead]
arjunchint 7mo ago
So... an API? Most sites don't want to expose APIs or care enough about setup and maintenance of said API.
- manveerc 7mo ago
  Are you asking if Agents should use API?

nudpiedo 7mo ago

They will wish that you use an official API, follow the funnel they settled for you, and make purchases no matter how

loveparade 7mo ago

Not fine if you use Claude. But it's fine if you are Google Flights and the user uses Gemini. The paid version of course.

victorbjorklund 7mo ago

They wanna let you use the service the way they want. An e-commerce? Wanna automate buying your stuff - probably something they wanna allow under controlled forms Wanna scrape the site to compare prices? Maybe less so.

candiddevmike 7mo ago
A brave new world for fraud and returns. Also I just recently noticed Chrome now has a Klarna/BNPL thing as a built in payments option that I never asked for...
- kylecazar 7mo ago
  Yeah it's a payment method they added to Google Pay (Google Wallet? I don't know anymore). You can turn it off in autofill settings.

chrash 7mo ago

i’m seeing this at my corporate software job now. that service that you used to have security and product approval for to even read their Swagger doc has an MCP server you can install with 2 clicks.

politelemon 7mo ago
Sometimes, it gets added there without your consent.

buzzerbetrayed 7mo ago

Why should a browser care about how websites want you to use them?

OsrsNeedsf2P 7mo ago

These are obviously different people you're talking about here

nojs 7mo ago

It’s weirder than that. There is a surge of companies working on how to provide automated access to things like payments, email, signup flows, etc to *Claw.

akersten 7mo ago

I'm old enough to remember discussions around the meaning of `User-Agent` and why it was important that we include it in HTTP headers. Back before it was locked to `Chromium (Gecko; Mozilla 4.0/NetScape; 147.01 ...)`. We talked about a magical future where your PDA, car, or autonomous toaster could be browsing the web on your behalf, and consuming (or not consuming) the delivered HTML as necessary. Back when we named it "user agent" on purpose. AI tooling can finally realize this for the Web, but it's a shame that so many companies who built their empires on the shoulders of those visionaries think the only valid way to browse is with a human-eyeball-to-server chain of trust.

cameldrv 7mo ago
Me too but it died when ads became the currency of the web. If the reason the site exists is to use ads, they’re not going to let you use an user agent that doesn’t display the ads.
- akersten 7mo ago
  > If the reason the site exists is to use ads, they’re not going to let you use an user agent that doesn’t display the ads. They've been giving it the old college try for the better part of two decades and the only website I've had to train myself not to visit is Twitch, whose ads have invaded my sightline one time too many, and I conceded that particular adblocking battle. I don't get the sense that it's high on the priority list for most sites out there (knock on wood).
  - snackerblues 7mo ago
    Same, I just don't use Twitch when possible. Most streamers rehost their VODs on Youtube which has a better player anyway.
  - diacritical 7mo ago
    People who block ads are a minority. Sites that serve heavy content like video would care if someone wastes their resources but blocks ads, but why would a site that serves a few KBs of text spend the resources on blocking such users or making the ads beat the ad blocker in a tiresome cat and mouse game? Those users could even share or recommend the site to someone else who doesn't use ad blockers, so it actually makes sense to not try to battle ad blockers if you want to make your site more popular. This makes sense for sites that rely on network effects, like forums or classified ad sites and so on. Unless they have a near monopoly or some really valuable content, they would benefit financially if they let people block their ads. I can't back that up with data or anything, but it makes sense to me.

dawnerd 7mo ago

And what site is going to open their api up to everyone? Document endpoints already exist, why make it more complicated.

moron4hire 7mo ago

Oh, that's an easy one. LLMs have made people lose their god damned minds. It makes sense when you think about it as breaking a few eggs to get to the promised land omelette of laying off the development staff.

avaer 7mo ago

In a nutshell: Google wants your websites to be more easily used by the agents they are putting in the browser and other products. They own the user layer and models, and get to decide if your product will be used. Think search monopoly, except your site doesn't even exist as far as users are concerned, it's only used via an agent, and only if Google allows. The work of implementing this is on you. Google is building the hooks into the browser for you to do it; that's WebMCP. It's all opaque; any oopsies/dark patterns will be blamed on the AI. The profits (and future ad revenue charged for sites to show up on the LLM's radar) will be claimed by Google. The other AI companies are on board with this plan. Any questions?

solaire_oa 7mo ago
We should definitely feel trepidation at the prospects of any LLM guided browser, in addition to WebMCP (e.g. Claude for Chrome enters the same opaque LLM-controlled/deferred decision process, OpenClaw etc). Just one example: Prompting the browser to "register example.com" means that Google/Anthropic gets to hustle registrars for SEO-style priority. Using countermeasures like captcha locks you out of the LLM market. Google's incentive to allow you to shop around via traditional web search is decreased since traditional ads won't be as lucrative (businesses will catch on that blanket targeted ads aren't as effective as a "referral" that directs an LLM to sign-up/purchase/exchange something directly)... expect web search quality to decline, perhaps intentionally. The only way to combat this, as far as I can conceptualize, is with open models, which are not yet as good as private ones, in no small part due to the extraordinary investment subsidization. We can hope for the bubble to pop, but plan for a deader Internet. Meanwhile, trust online, at large, begins to evaporate as nobody can tell what is an LLM vs a human-conducted browser. The Internet at large is entering some very dark waters.
oefrha 7mo ago
The irony is Google properties are more locked down than ever. When I use a commercial VPN I get ReCAPTCHA’ed half of the time doing every single Google search; and can’t use YouTube in Incognito sometimes, “Sign in to confirm you’re not a bot”.
- 7mo ago

medi8r 7mo ago

Both. I imagine if using this there is a tell (e.g. UA or other header). Sites can just block unauthenticated sessions using it but allow it to be used when they know who.

SilverElfin 7mo ago

I feel like this is a way to ultimately limit the ability to scrape but also the ability to use your own AI agent to take actions across the internet for you. Like how Amazon doesn’t let your agent to shop their site for you, but they’ll happily scrape every competitor’s website to enforce their anti competitive price fixing scheme. They want to allow and deny access on their terms. WebMCP will become another channel controlled by big tech and it’ll come with controls. First they’ll lure people to use this method for the situations they want to allow, and then they’ll block everything else.

aragonite 7mo ago

> Do websites want to prevent automated tooling, as indicated by everyone putting everything behind Cloudfare and CAPTCHAs since forever, or do websites want you to be able to automate things? Because I don't see how you can have both. The proposal (https://docs.google.com/document/d/1rtU1fRPS0bMqd9abMG_hc6K9OAI6soUy3Kh00toAgyk/edit?tab=t.0 https://docs.google.com/document/d/1rtU1fRPS0bMqd9abMG_hc6K9...) draws the line at headless automation. It requires a visible browsing context. > Since tool calls are handled in JavaScript, a browsing context (i.e. a browser tab or a webview) must be opened. There is no support for agents or assistive tools to call tools "headlessly," meaning without visible browser UI.

Intermernet 7mo ago
That really just increases the processing power required to automate it. VM running Chrome to a virtual frame buffer, point agent at frame buffer, automate session. It's clunky, but probably not that much more memory intensive than current browser automation. You could probably ditch the frame buffer as well, except for giving the browser something to write out to. It can probably be /dev/null.

bear3r 7mo ago

different threat model. cloudflare blocks automation that pretends to be human -- scraping, fake clicks, account stuffing. webmcp is a site explicitly publishing 'here are the actions i sanction.' you can block selenium on login and expose a webmcp flight search endpoint at the same time. one's unauthorized access, the other's a published api.

maximinus_thrax 7mo ago

> Do websites want to prevent automated tooling, as indicated by everyone putting everything behind Cloudfare and CAPTCHAs since forever, Not if they don't want their rankings to tank. Now you'll need to make your website machine friendly while the lords of walled gardens will relentlessly block any sort of 'rogue' automated agent from accessing their services.

fasbiner 7mo ago

I can deeply, deeply relate. X and Bluesky are both going nuts with ai and ai scams, but _both_ of them banned an advertising account because we were... using a bot to automate behavior because their APIs are only a subset of functionality. Their vision is a world where they use all the automation regardless of safety or law, and we have to jump through extra hoops and engage in manual processes with AI that literally doesn't have the tool access to do what we need and will not contact a human.

joshuanapoli 7mo ago

WebMCP should be a really easy way to add some handy automation functionality to your website. This is probably most useful for internal applications.

est 7mo ago

>Can someone explain what the hell is going on here? Someone at Chromium team is launching rapidly for an promotion

dokdev 7mo ago

I was also thinking about more or less the same thing with APIs and MCPs. The companies that didn't have any public apis are now exposing MCPs. That, to me is quite interesting. Maybe it is the FOMO effect.

sleight42 7mo ago

I can't see walled garden platforms or any website that monetizes based on ads offering WebMCP. Agents using their site represent humans who aren't.

notatoad 7mo ago

as a website operator, i want my website to not experience downtime and unreliability because of usage rates that exceed the rate at which humans load pages, and i want to not be defrauded. if you want to access my website using automated tools, that's fine. but if there's a certain automated tool that is consistently used to either break the site or attempt to defraud me, i'm going to do my best to block that tool. and sometimes that means blocking other, similar tools. if the webMCP client in chrome behaves in a reasonable way that prevents abuse, then i don't see a problem with it. if scammers discover they can use it to scam, then websites will block it too.

DrScientist 7mo ago

Obviously if you wanted people to book flights with a bot then you could have provided a public API for that long ago. I think potentially the subtlety here is a sort of cooperative mode - the computer filling out a lot of the forms and doing the grunt, but it's important that the human is still in the loop - so they need to be able to share a UI with the agent. Hence a agent friendly web page, rather than just an API.

aubanel 7mo ago

I think I have one explanation why for a website, exposing an MCP servers AND having captchas can make sense. - an agent loading the real page is waste for the server, because the data sent is a few megavytes, and you don't have the usual returns of an user seeing your ads - BUT API requests (or here, MCP) are much lighter, a few dozen kB, so that makes the ROI positive again At least that's my view : please tell me, anyone, if that reason doesn't make sense!