8 ms·

Probably a super dumb question, but isn't this fairly unethical? The "automatic IP rotation" feature isn't there for no reason.

mcs_ 11y ago
how can be the research into internet unethical? or the automation of that unethical?
hvs 11y ago
It could certainly be used for unethical purposes, but it isn't de facto unethical.
angry-hacker 11y ago
As much as unethical as using adblock or disabling javascript. If you don't want your content to be scraped, don't put it online!
- manigandham 11y ago
  That's not a good argument. Do you ever leave your stuff lying around? I guess we can just take it then right? Just because you have access to something doesn't give you permission to access or access it in any manner possible.
  - deleted 11y ago
    [deleted]
  - yxdfasdjkljasdf 11y ago
    That is not how HTTP works; your analogy is not correct. Nobody is taking anything. If you don't want someone to access your page, then don't respond to their request.
    - manigandham 11y ago
      Since there's no easy way to always reliably identify the requester, this gets complicated. Most scrapers - including this one - advertise how they use multiple servers/locations/ips/etc to get around this.
      - yxdfasdjkljasdf 11y ago
        I fail to see a problem you are trying to present. Even if identification was hard, which is not true because of how HTTP works, it is irrelevant because HTTP doesn't discriminate. If someone does, that is their problem, and should be solved by them, and not a committee or law.
        manigandham 11y ago
        > If you don't want someone to access your page, then don't respond to their request > there's no easy way to always reliably identify the requester That's the problem: you can't identify the person to block them in the first place. Robots.txt is actually an explicit signal of intention for reputable search engines but that's all we have today and is easily ignored and does not work with these scrapers or anyone else. Not sure what your last sentence means.
    - dsjoerg 11y ago
      At a high enough frequency, scraping is indistinguishable from a DDoS attack. Do you believe DDoS attacks are OK? How do you draw the line?
      - yxdfasdjkljasdf 11y ago
        There is a clear distinction in the two. You are presenting a straw-man argument.
        dsjoerg 11y ago
        You haven't quite laid out your argument so I have to guess what it is. When you say "That is not how HTTP works" it suggests that your claim is that anything that HTTP allows is ethically OK to do. However that is clearly a ridiculous stance, since a DDoS attack is a stream of valid HTTP requests and that's clearly not OK. So I'm left wondering what your argument actually is for why unwelcome scraping is ethically OK. I find this an interesting question, because while I would love for protcols to also define ethics, I feel that would be scope creep for the poor protocol designers. There's a wide variety of conduct and ethics questions that a protocol cannot address. Where I myself draw the line is at protocol behavior intentionally designed to obscure my intentions. For example, sending my requests from a wide variety of IP addresses is behavior that is specifically designed to obscure where I'm coming from; my only intent in doing so would be to circumvent the intent of the serving machine from providing lots of content to a single requestor. At that point I'm engaging in deceptive behavior; I've crossed an ethical line.
        yxdfasdjkljasdf 11y ago
        When you say "That is not how HTTP works" it suggests that your claim is that anything that HTTP allows is ethically OK to do. However that is clearly a ridiculous stance, since a DDoS attack is a stream of valid HTTP requests and that's clearly not OK. That wasn't a response made to your comment, and you are mixing two different arguments there. You guess in not correct. So I'm left wondering what your argument actually is for why unwelcome scraping is ethically OK. I never even suggested such an argument. The behavior you described in the last paragraph is only deceptive from the eyes of an information and privacy surveillant state actor. Anonymity is not unethical, it is a human right.
  - angry-hacker 11y ago
    You have the right to disagree, but that's the way World Wide Web was built. Feel free to use alternative service(s) or stop publishing your stuff. Put it behind password or don't answer to my scrapers or browsers requests. Fair and simple. Web is for people from people, not solely for company(s) financial interests.
    - manigandham 11y ago
      What's the WWW or the way it's built have to do with it? And the web is just technology, it's not "for" anyone or anything in particular. Intentions matter - on both sides. This is what most of the legal framework of the entire world is based on. You can disagree with that but again the ability to do something doesn't grant permission to do it. You're saying the solution to that is to remove the ability, but I don't see how that's realistic.
      - fauigerzigerk 11y ago
        >What's the WWW or the way it's built have to do with it? What it has to do with it is that putting an HTTP server on the public Web signals the intention to serve up resources to anyone who sends an HTTP request. Any restrictions to this default must be implemented explicitly on top of the default. Leaving my stuff lying around does not signal my intention for anyone to take it, unless I let it lying around next to the bins. So yes, intentions matter. The question is how we learn about them. Sometimes the choice of technology implies particular intentions by default.
        manigandham 11y ago
        Putting a up a webserver that can be publically reached is not authorization to access it. I really can't say this in any other way - just because you can do something doesn't mean you are allowed to, whether it's online or offline. We already have an explicit signal called robots.txt which major search engines use. The problem is that there's no way to enforce this and there's just very little enforcement against actions on the web in general which is why people can get away with scraping but please don't mistake it for somehow being OK or allowed by the owner of that content. It's just not that simple.
  - deleted 11y ago
    [deleted]
- bduerst 11y ago
  You still have copyright to your online content and can dictate how it can be used. Websites can also have TOS for their content. If you're arguing ethics then it could be unethical to use content in a way that the copyright owner doesn't want it to be used. That doesn't mean that it doesn't happen.
  - jumperjake 11y ago
    IANAL, but copyright governs redistribution of content not consumption (That's what pirates get busted for). I aslo recall that there was a ruling that footer TOSs aren't enforceable unless the user actively and explicitly agrees to them. I agree with the GP in that public content is fair game. How do you thing google works?
    - bduerst 11y ago
      Google technically respects robots txt and noindex metatags. OP is arguing the ethics of scraping, not if people are ignoring bot meta tags. Copyright governs how the content is used, including distribution. The reason people who download videos are not liable is because you have to download the complete content to see the copyright. File sharers have already downloaded the content and are subject to copyright. Bots that scrape can interpret meta tags in the header of the dom, which is why scraping and violating copyright is unethical.
blairanderson 11y ago
nope
snowwrestler 11y ago
I don't think it's unethical to pull down a copy of public information. If you pull too fast it might be considered rude (heavy load on the server). That's why some sites reflexively block all scrapers, hence the rotating IP feature. Hopefully this tool is rate-limited so it's not rude. In terms of copyright, what matters is what you do with the scrape. If you scrape a public website for personal use, it's no different from just browsing it for personal use. If you try to republish the content for your own benefit, you'll run afoul of copyright law.
- chinathrow 11y ago
  If you disobey robots.txt it's unethical and rude.
  - snowwrestler 11y ago
    I agree, but running a scraper does not necessarily mean disobeying robots.txt.

mcs_ 11y ago

how can be the research into internet unethical? or the automation of that unethical?

hvs 11y ago

It could certainly be used for unethical purposes, but it isn't de facto unethical.

angry-hacker 11y ago

As much as unethical as using adblock or disabling javascript. If you don't want your content to be scraped, don't put it online!

manigandham 11y ago
That's not a good argument. Do you ever leave your stuff lying around? I guess we can just take it then right? Just because you have access to something doesn't give you permission to access or access it in any manner possible.
- deleted 11y ago
  [deleted]
- yxdfasdjkljasdf 11y ago
  That is not how HTTP works; your analogy is not correct. Nobody is taking anything. If you don't want someone to access your page, then don't respond to their request.
  - manigandham 11y ago
    Since there's no easy way to always reliably identify the requester, this gets complicated. Most scrapers - including this one - advertise how they use multiple servers/locations/ips/etc to get around this.
    - yxdfasdjkljasdf 11y ago
      I fail to see a problem you are trying to present. Even if identification was hard, which is not true because of how HTTP works, it is irrelevant because HTTP doesn't discriminate. If someone does, that is their problem, and should be solved by them, and not a committee or law.
      - manigandham 11y ago
        > If you don't want someone to access your page, then don't respond to their request > there's no easy way to always reliably identify the requester That's the problem: you can't identify the person to block them in the first place. Robots.txt is actually an explicit signal of intention for reputable search engines but that's all we have today and is easily ignored and does not work with these scrapers or anyone else. Not sure what your last sentence means.
  - dsjoerg 11y ago
    At a high enough frequency, scraping is indistinguishable from a DDoS attack. Do you believe DDoS attacks are OK? How do you draw the line?
    - yxdfasdjkljasdf 11y ago
      There is a clear distinction in the two. You are presenting a straw-man argument.
      - dsjoerg 11y ago
        You haven't quite laid out your argument so I have to guess what it is. When you say "That is not how HTTP works" it suggests that your claim is that anything that HTTP allows is ethically OK to do. However that is clearly a ridiculous stance, since a DDoS attack is a stream of valid HTTP requests and that's clearly not OK. So I'm left wondering what your argument actually is for why unwelcome scraping is ethically OK. I find this an interesting question, because while I would love for protcols to also define ethics, I feel that would be scope creep for the poor protocol designers. There's a wide variety of conduct and ethics questions that a protocol cannot address. Where I myself draw the line is at protocol behavior intentionally designed to obscure my intentions. For example, sending my requests from a wide variety of IP addresses is behavior that is specifically designed to obscure where I'm coming from; my only intent in doing so would be to circumvent the intent of the serving machine from providing lots of content to a single requestor. At that point I'm engaging in deceptive behavior; I've crossed an ethical line.
        yxdfasdjkljasdf 11y ago
        When you say "That is not how HTTP works" it suggests that your claim is that anything that HTTP allows is ethically OK to do. However that is clearly a ridiculous stance, since a DDoS attack is a stream of valid HTTP requests and that's clearly not OK. That wasn't a response made to your comment, and you are mixing two different arguments there. You guess in not correct. So I'm left wondering what your argument actually is for why unwelcome scraping is ethically OK. I never even suggested such an argument. The behavior you described in the last paragraph is only deceptive from the eyes of an information and privacy surveillant state actor. Anonymity is not unethical, it is a human right.
- angry-hacker 11y ago
  You have the right to disagree, but that's the way World Wide Web was built. Feel free to use alternative service(s) or stop publishing your stuff. Put it behind password or don't answer to my scrapers or browsers requests. Fair and simple. Web is for people from people, not solely for company(s) financial interests.
  - manigandham 11y ago
    What's the WWW or the way it's built have to do with it? And the web is just technology, it's not "for" anyone or anything in particular. Intentions matter - on both sides. This is what most of the legal framework of the entire world is based on. You can disagree with that but again the ability to do something doesn't grant permission to do it. You're saying the solution to that is to remove the ability, but I don't see how that's realistic.
    - fauigerzigerk 11y ago
      >What's the WWW or the way it's built have to do with it? What it has to do with it is that putting an HTTP server on the public Web signals the intention to serve up resources to anyone who sends an HTTP request. Any restrictions to this default must be implemented explicitly on top of the default. Leaving my stuff lying around does not signal my intention for anyone to take it, unless I let it lying around next to the bins. So yes, intentions matter. The question is how we learn about them. Sometimes the choice of technology implies particular intentions by default.
      - manigandham 11y ago
        Putting a up a webserver that can be publically reached is not authorization to access it. I really can't say this in any other way - just because you can do something doesn't mean you are allowed to, whether it's online or offline. We already have an explicit signal called robots.txt which major search engines use. The problem is that there's no way to enforce this and there's just very little enforcement against actions on the web in general which is why people can get away with scraping but please don't mistake it for somehow being OK or allowed by the owner of that content. It's just not that simple.
- deleted 11y ago
  [deleted]
bduerst 11y ago
You still have copyright to your online content and can dictate how it can be used. Websites can also have TOS for their content. If you're arguing ethics then it could be unethical to use content in a way that the copyright owner doesn't want it to be used. That doesn't mean that it doesn't happen.
- jumperjake 11y ago
  IANAL, but copyright governs redistribution of content not consumption (That's what pirates get busted for). I aslo recall that there was a ruling that footer TOSs aren't enforceable unless the user actively and explicitly agrees to them. I agree with the GP in that public content is fair game. How do you thing google works?
  - bduerst 11y ago
    Google technically respects robots txt and noindex metatags. OP is arguing the ethics of scraping, not if people are ignoring bot meta tags. Copyright governs how the content is used, including distribution. The reason people who download videos are not liable is because you have to download the complete content to see the copyright. File sharers have already downloaded the content and are subject to copyright. Bots that scrape can interpret meta tags in the header of the dom, which is why scraping and violating copyright is unethical.

blairanderson 11y ago

nope

snowwrestler 11y ago

I don't think it's unethical to pull down a copy of public information. If you pull too fast it might be considered rude (heavy load on the server). That's why some sites reflexively block all scrapers, hence the rotating IP feature. Hopefully this tool is rate-limited so it's not rude. In terms of copyright, what matters is what you do with the scrape. If you scrape a public website for personal use, it's no different from just browsing it for personal use. If you try to republish the content for your own benefit, you'll run afoul of copyright law.

chinathrow 11y ago
If you disobey robots.txt it's unethical and rude.
- snowwrestler 11y ago
  I agree, but running a scraper does not necessarily mean disobeying robots.txt.