6 ms·

Is it ethical to scrape when a site has explicitly blocked bots? I know a fair number of people who run small sites who are already considering closing them dow

Is it ethical to scrape when a site has explicitly blocked bots? I know a fair number of people who run small sites who are already considering closing them down because the bots are relentlessly hammering their sites and driving up hosting costs.

est 2mo ago
> Is it ethical to scrape when a site has explicitly blocked bots Ethical? ppl get jailed in China for this. "Breaching computer systems" is a felony.
- yogorenapan 2mo ago
  I know for a fact companies in China do this. One paid me directly for work in this space, and in another, I found my own OSS project in their artifact cache while working there. I've only ever gotten in trouble in the UK
maxglute 2mo ago
I suppose it's like trying to enforce no picture rule in public.
LoganDark 2mo ago
It's most definitely unethical.
cr125rider 2mo ago
I think it really depends on how the bot is run. If the bot is replacing me navigating there manually, absolutely. If it’s sucking up content to rip off and make someone else billions of dollars, no.
- fooqux 2mo ago
  Why do you feel entitled to navigate to a website you neither own nor pay for via a method the owner expressly forbids?
  - theptip 2mo ago
    Why do you feel entitled to dictate what user agent I use, if it’s well-behaved?
    - getnormality 2mo ago
      Why do your opinions about what a "well-behaved" agent is override the wishes of a site's owner?
      - Dylan16807 2mo ago
        Site owners should not have full control over what users do. Their wishes do not apply past certain boundaries. For a non-AI example I run into occasionally, any wish to stop time-shifting should not be relevant to what I choose to do as a media consumer.
        pooploop64 2mo ago
        There actually is a well defined demarcation point where the site owners wishes do apply. It's called... A demarc point! Everything that happens past this point is by definition entirely in the control of the admin. Which includes closing the door on whoever they wish. This is a technical reality, not something you can effect with opinions or wanting things more than other people want other things. It might seem contradictory but this is actually the main thing that makes the internet as free as it is.
        Dylan16807 2mo ago
        When we're talking about ethics, I think we should give the owner more than that. If they don't want to be DDoSed we shouldn't say "too bad, the attackers are outside your network, your wishes don't apply, get good".
      - theptip 2mo ago
        A request is a request. None of your business what I do with it after that.
lemagedurage 2mo ago
Use cases range from sending 100s of requests per second just to bring a website down to doing a montly request to a municipality's endpoint to get a local dashboard of when trash is picked up. I don't think you can pass a single judgement of automating web requests in general.

est 2mo ago

> Is it ethical to scrape when a site has explicitly blocked bots Ethical? ppl get jailed in China for this. "Breaching computer systems" is a felony.

yogorenapan 2mo ago
I know for a fact companies in China do this. One paid me directly for work in this space, and in another, I found my own OSS project in their artifact cache while working there. I've only ever gotten in trouble in the UK

maxglute 2mo ago

I suppose it's like trying to enforce no picture rule in public.

LoganDark 2mo ago

It's most definitely unethical.

cr125rider 2mo ago

I think it really depends on how the bot is run. If the bot is replacing me navigating there manually, absolutely. If it’s sucking up content to rip off and make someone else billions of dollars, no.

fooqux 2mo ago
Why do you feel entitled to navigate to a website you neither own nor pay for via a method the owner expressly forbids?
- theptip 2mo ago
  Why do you feel entitled to dictate what user agent I use, if it’s well-behaved?
  - getnormality 2mo ago
    Why do your opinions about what a "well-behaved" agent is override the wishes of a site's owner?
    - Dylan16807 2mo ago
      Site owners should not have full control over what users do. Their wishes do not apply past certain boundaries. For a non-AI example I run into occasionally, any wish to stop time-shifting should not be relevant to what I choose to do as a media consumer.
      - pooploop64 2mo ago
        There actually is a well defined demarcation point where the site owners wishes do apply. It's called... A demarc point! Everything that happens past this point is by definition entirely in the control of the admin. Which includes closing the door on whoever they wish. This is a technical reality, not something you can effect with opinions or wanting things more than other people want other things. It might seem contradictory but this is actually the main thing that makes the internet as free as it is.
        Dylan16807 2mo ago
        When we're talking about ethics, I think we should give the owner more than that. If they don't want to be DDoSed we shouldn't say "too bad, the attackers are outside your network, your wishes don't apply, get good".
    - theptip 2mo ago
      A request is a request. None of your business what I do with it after that.

lemagedurage 2mo ago

Use cases range from sending 100s of requests per second just to bring a website down to doing a montly request to a municipality's endpoint to get a local dashboard of when trash is picked up. I don't think you can pass a single judgement of automating web requests in general.