6 ms·

Dug into chrome.dll and figured out how the x-browser-validation header is generated. Full write up and PoC code here: https://github.com/dsekz/chrome-x-browser

by dsekz 1y ago

Dug into chrome.dll and figured out how the x-browser-validation header is generated. Full write up and PoC code here: https://github.com/dsekz/chrome-x-browser-validation-header https://github.com/dsekz/chrome-x-browser-validation-header Why do you think Chrome bothers with this extra headers. Anti-spoofing, bot detection, integrity or something else?

twapi 1y ago
Seems like they are using these headers only for google.com requests.
- xnx 1y ago
  Yes I think it is part of their multi level testing of for new version rollouts. In addition to all the internal unit and performance tests, they want an extra level of verification that weird things aren't happening in the wild
- AznHisoka 1y ago
  They probably are using it to block bots scraping Google results is my theory
userbinator 1y ago
Making it easier to reject "unapproved" or "unsupported" browsers and take away user freedom. Trying to make it harder for other browsers to compete.
- ajross 1y ago
  That can be done already based on User-Agent, though. Other browsers don't spoof their agent strings to look like Chrome, and never have (or, they do, but only in the sense that everyone still claims to be Mozilla). And browsers have always (for obvious reasons) been very happy to identify themselves correctly to backend sites. The purpose here is surely to detect sophisticated spoofing by non-user-browser software, like crawlers and robots. Robots are in fact required by the net's Geneva Convention equivalent to identify themselves and respect limitations, but obviously many don't. I have a hard time understanding robot detection as an issue of "user freedom" or "browser competition".
  - soulofmischief 1y ago
    It's easy to change the User Agent and we cannot handwave this fact away for the sake of argument.
  - Sayrus 1y ago
    > I have a hard time understanding robot detection as an issue of "user freedom" or "browser competition". In the name of robot detection, you can lock down device, require device attestation, prevent users from running non-standard devices/OS/software, prevent them from accessing websites (CloudFlare dislikes non-chrome browser and hates non-standard browsers, ReCaptcha blocks you out if you're not on Chrome-like/Safari/Firefox). Web Environment Integrity[1] is also a good example of where robot detection ends up affecting the end user. [1] https://en.wikipedia.org/wiki/Web_Environment_Integrity https://en.wikipedia.org/wiki/Web_Environment_Integrity
    - baybal2 1y ago
      [dead]
    - ajross 1y ago
      Aren't all those solutions even more impactful on the user experience though? Someone who cares about user freedom would think they're even worse, no?
  - jml7c5 1y ago
    >I have a hard time understanding robot detection as an issue of "user freedom" or "browser competition". The big one is that running a browser other than Chrome (or Safari) could come to mean endless captchas, degrading the experience. "Chrome doesn't have as many captchas" is a pretty good hook.
    - Forgeties79 1y ago
      Not to mention how often you can get stuck in an infinite loop where it just will not accept your captcha results and keeps making you do it over and over. Especially if you’re using a VPN. It’s maddening sometimes. Can’t even do a basic search
exiguus 1y ago
I have two questions: 1. Do I understand it correctly and the validation header is individual for each installation? 2. Is this header only in Google Chrome or also in Chromium?
- gruez 1y ago
  >1. Do I understand it correctly and the validation header is individual for each installation? I'm not sure how you got that impression. It's generated from fixed constants. https://github.com/dsekz/chrome-x-browser-validation-header?tab=readme-ov-file#how-the-header-is-made https://github.com/dsekz/chrome-x-browser-validation-header?...
  - exiguus 1y ago
    It's still not clear to me because it's called the default API key. And for me, default means that this is normally overwritten. And if overwritten, during build or during install? That's what I'm asking myself.
- dlenski 1y ago
  I had the same question (2). https://news.ycombinator.com/item?id=44560664 https://news.ycombinator.com/item?id=44560664 If it's only in the closed-source Chrome, then it seems it's intended to help Google's servers distinguish between Google's own products and others. But I've never seen a Google site which worked less-well in Chromium than in Chrome, so I'm somewhat skeptical of this. Perhaps there are exceptions
deleted 1y ago
[deleted]
wernerb 1y ago
Is it not likely that it protects against AI bot Llama?
- wut42 1y ago
  I don't see how you can "protect" against a large language model that cannot do browsing.
Avamander 1y ago
> Why do you think Chrome bothers with this extra headers. Anti-spoofing, bot detection, integrity or something else? Bot detection. It's a menace to literally everyone. Not to piss anyone off, but if you haven't dealt with it, you don't have anything of value to scrape or get access to.
- lxgr 1y ago
  Do you mean bot and non-Chrome-using human detection?
- ohdeargodno 1y ago
  Bullshit. You don't have anything of value either. Scrapers will ram through _anything_, and figure out if it's useful later.
- motorest 1y ago
  > Bot detection. It's a menace to literally everyone. Not to piss anyone off, but if you haven't dealt with it, you don't have anything of value to scrape or get access to. What leads you to believe that bit developers are unable to set a request header? They managed fine to set Chrome's user agent. Why do you think something like X-Browser-Validation is off limits?
  - Sophira 1y ago
    Because you would need to reproduce an explicit Google copyright statement which states that you don't have the right to copy it ("All rights reserved.") in order to do it fully. That presumably gives Google the legal ammunition it needs to sue you if you do it.
    - tomsonj 1y ago
      It seems like the requirement to reproduce this copyright header alone, nevermind the validation hash, would be enough to scare off scrapers?
      - 1y ago

by dsekz 1y ago

twapi 1y ago

Seems like they are using these headers only for google.com requests.

xnx 1y ago
Yes I think it is part of their multi level testing of for new version rollouts. In addition to all the internal unit and performance tests, they want an extra level of verification that weird things aren't happening in the wild
AznHisoka 1y ago
They probably are using it to block bots scraping Google results is my theory

userbinator 1y ago

Making it easier to reject "unapproved" or "unsupported" browsers and take away user freedom. Trying to make it harder for other browsers to compete.

ajross 1y ago
That can be done already based on User-Agent, though. Other browsers don't spoof their agent strings to look like Chrome, and never have (or, they do, but only in the sense that everyone still claims to be Mozilla). And browsers have always (for obvious reasons) been very happy to identify themselves correctly to backend sites. The purpose here is surely to detect sophisticated spoofing by non-user-browser software, like crawlers and robots. Robots are in fact required by the net's Geneva Convention equivalent to identify themselves and respect limitations, but obviously many don't. I have a hard time understanding robot detection as an issue of "user freedom" or "browser competition".
- soulofmischief 1y ago
  It's easy to change the User Agent and we cannot handwave this fact away for the sake of argument.
- Sayrus 1y ago
  > I have a hard time understanding robot detection as an issue of "user freedom" or "browser competition". In the name of robot detection, you can lock down device, require device attestation, prevent users from running non-standard devices/OS/software, prevent them from accessing websites (CloudFlare dislikes non-chrome browser and hates non-standard browsers, ReCaptcha blocks you out if you're not on Chrome-like/Safari/Firefox). Web Environment Integrity[1] is also a good example of where robot detection ends up affecting the end user. [1] https://en.wikipedia.org/wiki/Web_Environment_Integrity https://en.wikipedia.org/wiki/Web_Environment_Integrity
  - baybal2 1y ago
    [dead]
  - ajross 1y ago
    Aren't all those solutions even more impactful on the user experience though? Someone who cares about user freedom would think they're even worse, no?
- jml7c5 1y ago
  >I have a hard time understanding robot detection as an issue of "user freedom" or "browser competition". The big one is that running a browser other than Chrome (or Safari) could come to mean endless captchas, degrading the experience. "Chrome doesn't have as many captchas" is a pretty good hook.
  - Forgeties79 1y ago
    Not to mention how often you can get stuck in an infinite loop where it just will not accept your captcha results and keeps making you do it over and over. Especially if you’re using a VPN. It’s maddening sometimes. Can’t even do a basic search

exiguus 1y ago

I have two questions: 1. Do I understand it correctly and the validation header is individual for each installation? 2. Is this header only in Google Chrome or also in Chromium?

gruez 1y ago
>1. Do I understand it correctly and the validation header is individual for each installation? I'm not sure how you got that impression. It's generated from fixed constants. https://github.com/dsekz/chrome-x-browser-validation-header?tab=readme-ov-file#how-the-header-is-made https://github.com/dsekz/chrome-x-browser-validation-header?...
- exiguus 1y ago
  It's still not clear to me because it's called the default API key. And for me, default means that this is normally overwritten. And if overwritten, during build or during install? That's what I'm asking myself.
dlenski 1y ago
I had the same question (2). https://news.ycombinator.com/item?id=44560664 https://news.ycombinator.com/item?id=44560664 If it's only in the closed-source Chrome, then it seems it's intended to help Google's servers distinguish between Google's own products and others. But I've never seen a Google site which worked less-well in Chromium than in Chrome, so I'm somewhat skeptical of this. Perhaps there are exceptions

deleted 1y ago

[deleted]

wernerb 1y ago

Is it not likely that it protects against AI bot Llama?

wut42 1y ago
I don't see how you can "protect" against a large language model that cannot do browsing.

Avamander 1y ago

> Why do you think Chrome bothers with this extra headers. Anti-spoofing, bot detection, integrity or something else? Bot detection. It's a menace to literally everyone. Not to piss anyone off, but if you haven't dealt with it, you don't have anything of value to scrape or get access to.

lxgr 1y ago
Do you mean bot and non-Chrome-using human detection?
ohdeargodno 1y ago
Bullshit. You don't have anything of value either. Scrapers will ram through _anything_, and figure out if it's useful later.
motorest 1y ago
> Bot detection. It's a menace to literally everyone. Not to piss anyone off, but if you haven't dealt with it, you don't have anything of value to scrape or get access to. What leads you to believe that bit developers are unable to set a request header? They managed fine to set Chrome's user agent. Why do you think something like X-Browser-Validation is off limits?
- Sophira 1y ago
  Because you would need to reproduce an explicit Google copyright statement which states that you don't have the right to copy it ("All rights reserved.") in order to do it fully. That presumably gives Google the legal ammunition it needs to sue you if you do it.
  - tomsonj 1y ago
    It seems like the requirement to reproduce this copyright header alone, nevermind the validation hash, would be enough to scare off scrapers?
    - 1y ago