9 ms·

I have yet to see any bots figure out how to get past the Basic Auth protecting all links on my (zero traffic) website. Of course, any user following a link wil

I have yet to see any bots figure out how to get past the Basic Auth protecting all links on my (zero traffic) website. Of course, any user following a link will be stopped by the same login dialog (I display the credentials on the home page). The solution is to make the secrets public. ALL websites could implement the same User/Pass credentials: User: nobots Pass: nobots Can bot writers overcome this if they know the credentials?

CaptainOfCoit 11mo ago
> Can bot writers overcome this if they know the credentials? Yes, instead of doing just a HTTP request, do a HTTP request with authentication, trivial really. Probably the reason they "can't" do that now is because they haven't came across "public content behind Basic Auth with known correct credentials", so the behavior hasn't been added. But it's literally loading http://username:password@example.com http://username:password@example.com instead of http://example.com http://example.com to use Basic Auth, couldn't be simpler :)
- 8organicbits 11mo ago
  The technical side is straightforward but the legal implications of trying passwords to try to scrape content behind authentication could pose a barrier. Using credentials that aren't yours, even if they are publicly known, is (in many jurisdictions) a crime. Doing it at scale as part of a company would be quite risky.
  - Filligree 11mo ago
    Sure, it’s a crime for the bots, but it would also be a crime for the ordinary users that you want to access the website. Or if you make it clear that they’re allowed, I’m not sure you can stop the bots then.
    - CaptainOfCoit 11mo ago
      I don't think it'd be illegal for anyone. The (theoretical) scenario is: There is a website (example.com) that publishes the correct credentials, and tells users to go to example.com/authenticate and put those there. At no point is a user (or bot) bypassing anything that was meant to stop them, they're following what the website is telling them publicly.
      - 8organicbits 11mo ago
        I think this analysis is correct. The part you're missing from my comment is "at scale", which means trying to apply this scraping technique to other sites. As a contract security engineer I've found all kinds of accidentally leaked credentials; knowing if a set of credentials is accidentally leaked or are being intentionally disclosed to the public feels like a human-in-the-loop kind of thing. Getting it wrong, especially when automated at scale, is the context the bot writer needs to consider.
  - Macha 11mo ago
    The legal implications of torrenting giant ebook collections didn't seem to stop them, not sure why this would
    - 8organicbits 11mo ago
      The law doesn't directly stop anyone from doing anything, it acts much differently from a technical control. The law provides recourse to people hurt by violations and enables law enforcement action. I suspect Meta has since stopped their torrenting, and may lose the lawsuit they current face. Anyone certainly could log in to any site with credentials that are not their own, but fear of legal action may deter them.
      - worik 11mo ago
        Not criminal law There is independent enforcement that should apply
    - _heimdall 11mo ago
      Going back to Napster hasn't the gray area always been in downloading versus uploading? If anyone could show that LLM companies have been uploading torrents then they really would be in trouble. If they are only proven to have downloaded torrents they're walking the line.
  - sisizbzb 11mo ago
    There’s hundreds of billions of dollars behind these guys. Not only that, but they also have institutional power backing them. The laws don’t really matter to the worst offenders. Similar to OPs article, trying to find a technical solution here is very inefficient and just a bandaid. The people running our society are on the whole corrupt and evil. Much simpler (not easier) and more powerful to remove them.
  - CaptainOfCoit 11mo ago
    > but the legal implications of trying passwords to try to scrape content behind authentication could pose a barrier If you're doing something alike to cracking then yeah. But if the credentials are right there on the landing page, and visible to the public, it's not really cracking anymore since you already know the right password before you try it, and the website that put up the basic auth is freely sharing the password, so you aren't really bypassing anything, just using the same access methods as everyone else. Again, if you're stumbling upon basic auth and you try to crack them, I agree it's at least borderline illegal, but this was not the context in the parent comment.
    - hn8726 11mo ago
      Otoh if, as a human, you use a known (even leaked on the website) password to "bypass the security" in order to "gain access to content you're not authorized to see", I think you'd get in trouble. I'd like if the same logic aplied to bots - implement basic (albeit weak) security and only allow access to humans. This way bots have to _hack you_ to read the content
      - CaptainOfCoit 11mo ago
        > you use a known (even leaked on the website) password to "bypass the security" in order to "gain access to content you're not authorized to see", I think you'd get in trouble I agree, but if someone has a website that says "This isn't the real page, go to /real.html and when authentication pops up, enter user:password", then I'd argue that is no longer "gaining access to content you're not authorized to see", the author of the page shared the credentials themselves, and acknowledged they aren't trying to hide anything, just providing a non-typical way of accessing the (for all intents and purposes, public) content.
    - lcnPylGDnU4H9OF 11mo ago
      > freely sharing the password It doesn't have to be so free. It can be shared with the stipulation that it's not used in a bot. https://www.law.cornell.edu/uscode/text/17/1201 https://www.law.cornell.edu/uscode/text/17/1201 (a) Violations Regarding Circumvention of Technological Measures.— (1) (A) No person shall circumvent a technological measure that effectively controls access to a work protected under this title. This has been used by car manufacturers to deny diagnostic information even though the encryption key needed to decrypt the information is sitting on disk next to the encrypted data. That's since been exempted for vehicle repairs but only because they're vehicle repairs, not because the key was left in plain view. If you are only authorized to access it under certain conditions, trying to access it outside those conditions is illegal (in the US, minimally). Gaining knowledge of a password does not grant permission to use it.
  - DrewADesign 11mo ago
    The people in the mad dash to AGI are either driven by religious conviction, or pure nihilism. Nobody doing this seriously considers the law a valid impediment. They justify (earnestly or not) companies doing things like scraping independent artist’s bread and butter work to create commercial services that tank their market with garbage knockoffs by claiming we’re moving into a post-work society. Meanwhile, the US government is moving at a breakneck pace to dismantle the already insufficient safety nets we do have. None of them care. Ethical roadblocks seem to be a solved problem in tech, now.
  - throwawayffffas 11mo ago
    Same goes for human users. The real way to avoid bots is actual login credentials.
- morkalork 11mo ago
  The bot protection on low traffic sites can be hilarious in how simple and effective it can be. Just click this checkbox. That's it. But it's not a check box matching a specific pattern provided by a well-known service, so until the bot writer inspects the site and adds the case it'll work. A browser running openai operator or whatever its called would immediately figure it out though.
  - akoboldfrying 11mo ago
    > A browser running openai operator or whatever its called would immediately figure it out though. But running that costs money, which is a disincentive. (How strong of a disincentive depends on how much it costs vs. the estimated value of a scraped page, but I think it would 100x the per-page cost at least.)
throw-10-13 11mo ago
[flagged]
lfkdev 11mo ago
Not sure if I can follow you, why would credentials known by anyone stop bots?
iberator 11mo ago
[flagged]
- goodthink 11mo ago
  Three score and seven years old. How old are you? Good insane or bad? lol I can't infer it from the comment. Here are links to my 3D, multiuser, "coin-toss as a service" apps: https://chalculator.com/gaas/?world=cutcards https://chalculator.com/gaas/?world=cutcards https://chalculator.com/gaas/?world=diceroll https://chalculator.com/gaas/?world=diceroll User: croquet Pas: yadayadayada Let the bot scraping begin. (These were the impetus for the BA strategy. Some of the assets are large. And they were getting downloaded A LOT. Not anymore.)
thrance 11mo ago
Clever solution, but it will only work as long as it doesn't become mainstream, or even a tiny bit more popular.

CaptainOfCoit 11mo ago

> Can bot writers overcome this if they know the credentials? Yes, instead of doing just a HTTP request, do a HTTP request with authentication, trivial really. Probably the reason they "can't" do that now is because they haven't came across "public content behind Basic Auth with known correct credentials", so the behavior hasn't been added. But it's literally loading http://username:password@example.com http://username:password@example.com instead of http://example.com http://example.com to use Basic Auth, couldn't be simpler :)

8organicbits 11mo ago
The technical side is straightforward but the legal implications of trying passwords to try to scrape content behind authentication could pose a barrier. Using credentials that aren't yours, even if they are publicly known, is (in many jurisdictions) a crime. Doing it at scale as part of a company would be quite risky.
- Filligree 11mo ago
  Sure, it’s a crime for the bots, but it would also be a crime for the ordinary users that you want to access the website. Or if you make it clear that they’re allowed, I’m not sure you can stop the bots then.
  - CaptainOfCoit 11mo ago
    I don't think it'd be illegal for anyone. The (theoretical) scenario is: There is a website (example.com) that publishes the correct credentials, and tells users to go to example.com/authenticate and put those there. At no point is a user (or bot) bypassing anything that was meant to stop them, they're following what the website is telling them publicly.
    - 8organicbits 11mo ago
      I think this analysis is correct. The part you're missing from my comment is "at scale", which means trying to apply this scraping technique to other sites. As a contract security engineer I've found all kinds of accidentally leaked credentials; knowing if a set of credentials is accidentally leaked or are being intentionally disclosed to the public feels like a human-in-the-loop kind of thing. Getting it wrong, especially when automated at scale, is the context the bot writer needs to consider.
- Macha 11mo ago
  The legal implications of torrenting giant ebook collections didn't seem to stop them, not sure why this would
  - 8organicbits 11mo ago
    The law doesn't directly stop anyone from doing anything, it acts much differently from a technical control. The law provides recourse to people hurt by violations and enables law enforcement action. I suspect Meta has since stopped their torrenting, and may lose the lawsuit they current face. Anyone certainly could log in to any site with credentials that are not their own, but fear of legal action may deter them.
    - worik 11mo ago
      Not criminal law There is independent enforcement that should apply
  - _heimdall 11mo ago
    Going back to Napster hasn't the gray area always been in downloading versus uploading? If anyone could show that LLM companies have been uploading torrents then they really would be in trouble. If they are only proven to have downloaded torrents they're walking the line.
- sisizbzb 11mo ago
  There’s hundreds of billions of dollars behind these guys. Not only that, but they also have institutional power backing them. The laws don’t really matter to the worst offenders. Similar to OPs article, trying to find a technical solution here is very inefficient and just a bandaid. The people running our society are on the whole corrupt and evil. Much simpler (not easier) and more powerful to remove them.
- CaptainOfCoit 11mo ago
  > but the legal implications of trying passwords to try to scrape content behind authentication could pose a barrier If you're doing something alike to cracking then yeah. But if the credentials are right there on the landing page, and visible to the public, it's not really cracking anymore since you already know the right password before you try it, and the website that put up the basic auth is freely sharing the password, so you aren't really bypassing anything, just using the same access methods as everyone else. Again, if you're stumbling upon basic auth and you try to crack them, I agree it's at least borderline illegal, but this was not the context in the parent comment.
  - hn8726 11mo ago
    Otoh if, as a human, you use a known (even leaked on the website) password to "bypass the security" in order to "gain access to content you're not authorized to see", I think you'd get in trouble. I'd like if the same logic aplied to bots - implement basic (albeit weak) security and only allow access to humans. This way bots have to _hack you_ to read the content
    - CaptainOfCoit 11mo ago
      > you use a known (even leaked on the website) password to "bypass the security" in order to "gain access to content you're not authorized to see", I think you'd get in trouble I agree, but if someone has a website that says "This isn't the real page, go to /real.html and when authentication pops up, enter user:password", then I'd argue that is no longer "gaining access to content you're not authorized to see", the author of the page shared the credentials themselves, and acknowledged they aren't trying to hide anything, just providing a non-typical way of accessing the (for all intents and purposes, public) content.
  - lcnPylGDnU4H9OF 11mo ago
    > freely sharing the password It doesn't have to be so free. It can be shared with the stipulation that it's not used in a bot. https://www.law.cornell.edu/uscode/text/17/1201 https://www.law.cornell.edu/uscode/text/17/1201 (a) Violations Regarding Circumvention of Technological Measures.— (1) (A) No person shall circumvent a technological measure that effectively controls access to a work protected under this title. This has been used by car manufacturers to deny diagnostic information even though the encryption key needed to decrypt the information is sitting on disk next to the encrypted data. That's since been exempted for vehicle repairs but only because they're vehicle repairs, not because the key was left in plain view. If you are only authorized to access it under certain conditions, trying to access it outside those conditions is illegal (in the US, minimally). Gaining knowledge of a password does not grant permission to use it.
- DrewADesign 11mo ago
  The people in the mad dash to AGI are either driven by religious conviction, or pure nihilism. Nobody doing this seriously considers the law a valid impediment. They justify (earnestly or not) companies doing things like scraping independent artist’s bread and butter work to create commercial services that tank their market with garbage knockoffs by claiming we’re moving into a post-work society. Meanwhile, the US government is moving at a breakneck pace to dismantle the already insufficient safety nets we do have. None of them care. Ethical roadblocks seem to be a solved problem in tech, now.
- throwawayffffas 11mo ago
  Same goes for human users. The real way to avoid bots is actual login credentials.
morkalork 11mo ago
The bot protection on low traffic sites can be hilarious in how simple and effective it can be. Just click this checkbox. That's it. But it's not a check box matching a specific pattern provided by a well-known service, so until the bot writer inspects the site and adds the case it'll work. A browser running openai operator or whatever its called would immediately figure it out though.
- akoboldfrying 11mo ago
  > A browser running openai operator or whatever its called would immediately figure it out though. But running that costs money, which is a disincentive. (How strong of a disincentive depends on how much it costs vs. the estimated value of a scraped page, but I think it would 100x the per-page cost at least.)

throw-10-13 11mo ago

[flagged]

lfkdev 11mo ago

Not sure if I can follow you, why would credentials known by anyone stop bots?

iberator 11mo ago

[flagged]

goodthink 11mo ago
Three score and seven years old. How old are you? Good insane or bad? lol I can't infer it from the comment. Here are links to my 3D, multiuser, "coin-toss as a service" apps: https://chalculator.com/gaas/?world=cutcards https://chalculator.com/gaas/?world=cutcards https://chalculator.com/gaas/?world=diceroll https://chalculator.com/gaas/?world=diceroll User: croquet Pas: yadayadayada Let the bot scraping begin. (These were the impetus for the BA strategy. Some of the assets are large. And they were getting downloaded A LOT. Not anymore.)

thrance 11mo ago

Clever solution, but it will only work as long as it doesn't become mainstream, or even a tiny bit more popular.