7 ms·

I'm a lawyer that works in the web-scraping space, and I always chuckle when I read threads like this. Almost every company that we now consider a monopolist (o

I'm a lawyer that works in the web-scraping space, and I always chuckle when I read threads like this. Almost every company that we now consider a monopolist (or their affiliates) in the tech space used scraping a part of their process to build their business, and almost every one of those same monopolists now prohibits startups and competitors from scraping their data (which, invariably, is not actually "their" data in any sort of legally cognizable sense). And so perhaps the ethics of web scraping are not so straightforward. And neither are the legal issues associated with it. I wrote an article about that last fall that got some attention here. https://news.ycombinator.com/item?id=37264676 https://news.ycombinator.com/item?id=37264676

jMyles 2y ago
> And so perhaps the ethics of web scraping are not so straightforward. It strikes me that the _ethics_ of web scraping are extremely straightforward and cognizable with a terse analysis: * You can respond however you like to my HTTP request, and I can parse your response however I like. Simple, traditional, common. This is the way that conversations have occurred since the dawn of human communication, no? > the legal issues associated with it. But aren't these, without exception, fabrics spun out of the cloth that shields established players with the threat of state violence? This is not particularly new, and seems to fit in the pathetic-and-predictable file. Moreover, the broader cheap attempt to cast this in "intellectual" property terms, and to attach that to protection of artists and creators, warrants a very particular eye-roll for its illogic.
- elicksaur 2y ago
  If I say, “Hey, please don’t text me anymore. I’m going to block this number,” and you respond by buying 500 phones in five cities and text me nonstop, is that ethical?
  - andai 2y ago
    Not sure the metaphor works here. For example most sites let Google scrape them as much as it likes, but go out of their way to block other robots. By doing so they are effectively forcing the whole world to use (or support, since smaller search engines have to piggyback on the big ones wih special status, and pay them) proprietary spyware. In your analogy, most websites block everyone except the biggest pervert known to man.
    - eli 2y ago
      Isn’t that a choice the website owner should be able to make?
      - jMyles 2y ago
        Of course it's your choice to make. Is someone forcing you to respond to requests you'd prefer to ignore?
        theamk 2y ago
        yes, people like OP who get the farms of scrapers. The website owners make their preferences clear with robots.txt, IP blocks and other antibot technology. Scrapers intentionally ignore owners' desires and force the to respond.
        paulryanrogers 2y ago
        If crawlers are stealth DDoSing my site then I lose the ability to respond entirely.
    - deleted 2y ago
      [deleted]
  - jMyles 2y ago
    It's your job to separate the wheat from the chaff at the boundary of your network interface. In fact, personal boundaries of all sorts, from informational to emotional to physical to economic, are of paramount importance in the information age. Nobody (and certainly not the state) is going to erect your personal boundaries for you by ensuring justice in the face of spammy text messages (or, for that matter, hypnotic and manipulative social media). This is your job - maybe your most important job. Just as its your job to protect your personal health and safety. Nobody (and certainly not the state) is going to do that for you. Is there something about the trajectory of evolution of the internet that suggests to you that this is incorrect? I observe continually (seemingly perpetually) increasing traffic, and continually (seemingly perpetually) increasing capacity for general purpose computing. I also observe enormous empathy and cyberpunk traditions in our communities, protecting each other. Do my eyes and ears deceive me?
    - paulryanrogers 2y ago
      Restraining orders are a thing for a reason. It's cheaper to harass someone out of business (intentionally or otherwise) than to compete on a level playing field. Being a good neighbor requires restraining oneself and making requests with consideration for the other party. Full disclosure: I worked for a price monitoring service that prided itself on crawling up to every 3 hours. Steps were always taken to mitigate the impact. Sometimes even asking hosts to allow-list the crawlers.
      - jMyles 2y ago
        > Restraining orders are a thing for a reason. Sure, but for the purposes of this conversation, saying "for a reason" regarding a function which is presently delegated to the state is fraught with all sorts of future-proofing concerns. It seems to me that, as a baseline, we have to agree to observe the apparent trend of the internet to supplant the state - to resist its censorship and influence almost entirely - as an indicator that our long-term thinking needs to put those relatively few state functions which are essential to a peaceful society (such as restraining orders) in the purview of the internet... somehow. Maybe that will prove to be unnecessary, but in the case that the state fades, we'll be happy we had the foresight. Internet traffic is barely (and arguably, already not) under human control as it is. And in another century, it will almost certainly be impossible to tell the machines 'enhance your calm or else'. Or else what? I agree wholeheartedly about your qualities of good neighbor roles. But I don't think they extrapolate the way you think they do. Consider this: at every moment, your house - your literal dwelling - is bombarded with high-level, semantic radio traffic, from way down where the messages bounce off the ionosphere all the way up to 10GHz and beyond. But this doesn't bother you. You ignore what you don't need! You draw boundaries and personally work on strengthening them - with the help of your friends and neighbors. The internet needs help taking this shape at the application layer (and really, at all layers). And that part is up to us. We can't just throw our hands up and say "<legacy state function> exists for some reason, doesn't it?"
- theamk 2y ago
  Do you apply this ethics to webs scraping only, or to all other network communications too? Because if that's your general principles, you are making the internet much shittier. I still remember the old internet with open SMTP servers, easy-to-use comment forms, and forums which did not require emails and capthas. But people with "You can respond however you like to my HTTP request" attitude ruined it with spam, scam and SEO. If you only apply this to web scraping, then where do you draw the line and why? Can you scrape at maximum rate server can support? Can you scrape if this requires active action (like account creation?) As long as you scrape, can you also post some links to improve your SEO?
  - cal85 2y ago
    > But people with "You can respond however you like to my HTTP request" attitude ruined it with spam, scam and SEO. I don’t see how those things relate. They all have separate ethical issues. You can believe it’s ok to scrape whatever info you can find online at the same time as believing it’s not ok to scam people.
    - galaxyofdoom 2y ago
      [dead]
  - jMyles 2y ago
    > Do you apply this ethics to webs scraping only, or to all other network communications too? I mean... if you're keying in at 20MHz and blasting a gigawatt of noise, then yeah you've certainly run afoul of decency and just law. You're changing the physical shape of the network environment. But if the concern is just that we don't like the bytes to which your signal decodes, or we don't like what you're doing with the response we give you, then it seems more like a speech/press issue. The internet needs to grow resilience such that annoyances in the logical layers are easy to ignore if you have the will. But that almost certainly means that you don't get to police what people do with the content you willingly hand over, pursuant to the protocol in use.
richardw 2y ago
Same thing with Facebook and identity. IIRC they leveraged Google’s address book to get traction, but will go after you if you try store FB social graph data long term for anything outside their garden. You try to block the tricks you used to get growth, basically.

jMyles 2y ago

> And so perhaps the ethics of web scraping are not so straightforward. It strikes me that the _ethics_ of web scraping are extremely straightforward and cognizable with a terse analysis: * You can respond however you like to my HTTP request, and I can parse your response however I like. Simple, traditional, common. This is the way that conversations have occurred since the dawn of human communication, no? > the legal issues associated with it. But aren't these, without exception, fabrics spun out of the cloth that shields established players with the threat of state violence? This is not particularly new, and seems to fit in the pathetic-and-predictable file. Moreover, the broader cheap attempt to cast this in "intellectual" property terms, and to attach that to protection of artists and creators, warrants a very particular eye-roll for its illogic.

elicksaur 2y ago
If I say, “Hey, please don’t text me anymore. I’m going to block this number,” and you respond by buying 500 phones in five cities and text me nonstop, is that ethical?
- andai 2y ago
  Not sure the metaphor works here. For example most sites let Google scrape them as much as it likes, but go out of their way to block other robots. By doing so they are effectively forcing the whole world to use (or support, since smaller search engines have to piggyback on the big ones wih special status, and pay them) proprietary spyware. In your analogy, most websites block everyone except the biggest pervert known to man.
  - eli 2y ago
    Isn’t that a choice the website owner should be able to make?
    - jMyles 2y ago
      Of course it's your choice to make. Is someone forcing you to respond to requests you'd prefer to ignore?
      - theamk 2y ago
        yes, people like OP who get the farms of scrapers. The website owners make their preferences clear with robots.txt, IP blocks and other antibot technology. Scrapers intentionally ignore owners' desires and force the to respond.
      - paulryanrogers 2y ago
        If crawlers are stealth DDoSing my site then I lose the ability to respond entirely.
  - deleted 2y ago
    [deleted]
- jMyles 2y ago
  It's your job to separate the wheat from the chaff at the boundary of your network interface. In fact, personal boundaries of all sorts, from informational to emotional to physical to economic, are of paramount importance in the information age. Nobody (and certainly not the state) is going to erect your personal boundaries for you by ensuring justice in the face of spammy text messages (or, for that matter, hypnotic and manipulative social media). This is your job - maybe your most important job. Just as its your job to protect your personal health and safety. Nobody (and certainly not the state) is going to do that for you. Is there something about the trajectory of evolution of the internet that suggests to you that this is incorrect? I observe continually (seemingly perpetually) increasing traffic, and continually (seemingly perpetually) increasing capacity for general purpose computing. I also observe enormous empathy and cyberpunk traditions in our communities, protecting each other. Do my eyes and ears deceive me?
  - paulryanrogers 2y ago
    Restraining orders are a thing for a reason. It's cheaper to harass someone out of business (intentionally or otherwise) than to compete on a level playing field. Being a good neighbor requires restraining oneself and making requests with consideration for the other party. Full disclosure: I worked for a price monitoring service that prided itself on crawling up to every 3 hours. Steps were always taken to mitigate the impact. Sometimes even asking hosts to allow-list the crawlers.
    - jMyles 2y ago
      > Restraining orders are a thing for a reason. Sure, but for the purposes of this conversation, saying "for a reason" regarding a function which is presently delegated to the state is fraught with all sorts of future-proofing concerns. It seems to me that, as a baseline, we have to agree to observe the apparent trend of the internet to supplant the state - to resist its censorship and influence almost entirely - as an indicator that our long-term thinking needs to put those relatively few state functions which are essential to a peaceful society (such as restraining orders) in the purview of the internet... somehow. Maybe that will prove to be unnecessary, but in the case that the state fades, we'll be happy we had the foresight. Internet traffic is barely (and arguably, already not) under human control as it is. And in another century, it will almost certainly be impossible to tell the machines 'enhance your calm or else'. Or else what? I agree wholeheartedly about your qualities of good neighbor roles. But I don't think they extrapolate the way you think they do. Consider this: at every moment, your house - your literal dwelling - is bombarded with high-level, semantic radio traffic, from way down where the messages bounce off the ionosphere all the way up to 10GHz and beyond. But this doesn't bother you. You ignore what you don't need! You draw boundaries and personally work on strengthening them - with the help of your friends and neighbors. The internet needs help taking this shape at the application layer (and really, at all layers). And that part is up to us. We can't just throw our hands up and say "<legacy state function> exists for some reason, doesn't it?"
theamk 2y ago
Do you apply this ethics to webs scraping only, or to all other network communications too? Because if that's your general principles, you are making the internet much shittier. I still remember the old internet with open SMTP servers, easy-to-use comment forms, and forums which did not require emails and capthas. But people with "You can respond however you like to my HTTP request" attitude ruined it with spam, scam and SEO. If you only apply this to web scraping, then where do you draw the line and why? Can you scrape at maximum rate server can support? Can you scrape if this requires active action (like account creation?) As long as you scrape, can you also post some links to improve your SEO?
- cal85 2y ago
  > But people with "You can respond however you like to my HTTP request" attitude ruined it with spam, scam and SEO. I don’t see how those things relate. They all have separate ethical issues. You can believe it’s ok to scrape whatever info you can find online at the same time as believing it’s not ok to scam people.
  - galaxyofdoom 2y ago
    [dead]
- jMyles 2y ago
  > Do you apply this ethics to webs scraping only, or to all other network communications too? I mean... if you're keying in at 20MHz and blasting a gigawatt of noise, then yeah you've certainly run afoul of decency and just law. You're changing the physical shape of the network environment. But if the concern is just that we don't like the bytes to which your signal decodes, or we don't like what you're doing with the response we give you, then it seems more like a speech/press issue. The internet needs to grow resilience such that annoyances in the logical layers are easy to ignore if you have the will. But that almost certainly means that you don't get to police what people do with the content you willingly hand over, pursuant to the protocol in use.

richardw 2y ago

Same thing with Facebook and identity. IIRC they leveraged Google’s address book to get traction, but will go after you if you try store FB social graph data long term for anything outside their garden. You try to block the tricks you used to get growth, basically.