5 ms·

I thought most AWS IP addresses were blocked by most admins. Does any one have experience scraping using Amazon.

abronte 15y ago
We have a pretty big e-commerce scraping project, and we don't really run into many problems regarding blacklisted ip's. There are a few sites who we get consistent bans from, but with elastic ip's it's pretty much a non-issue. I have yet to see a site that ban's all amazon ip's.
- mattgreenrocks 15y ago
  It is annoying when it happens and you're on something like Heroku.
- sjwright 15y ago
  I run http://whirlpool.net.au http://whirlpool.net.au and I religiously check the Amazon EC2 forum announcements[1] for new IP ranges to ban. [1] https://forums.aws.amazon.com/ann.jspa?annID=1030 https://forums.aws.amazon.com/ann.jspa?annID=1030
  - stoney 15y ago
    > I run http://whirlpool.net.au http://whirlpool.net.au An excellent and very useful forum! It seems like whatever topic I'm searching for, google(.com.au) returns a useful result on your site.
  - aw3c2 15y ago
    Would you mind telling why?
  - maaku 15y ago
    Why? Name me one good reason. Name me one.
    - sjwright 15y ago
      Shitloads of rogue bots doing "social media monitoring". Shitloads of rogue bots stealing content for black-hat SEO. Shitloads of rogue bots harvesting email addresses. Shitloads of rogue bots submitting spammy replies.
      - nupark2 15y ago
        That's unfortunate. I don't know what the answer is, but real people do route their traffic out of AWS endpoints.
        sjwright 15y ago
        Far fewer than you might think.
      - maaku 15y ago
        So maintain a blacklist of elastic IPs. If it's too big for you, make it a community effort. Those are bad reasons to close your site to all of AWS. As nupark2 mentioned, there are legitimate users routing traffic through EC2, even some bots that you'd want to visit your site. Archive.org comes to mind (many of there scrapers are or were behind AWS). Closing your site or app to a large swath of the web is the wrong solution. It's like killing a spider with a bazooka.
        sjwright 15y ago
        Unlike the assumptions you're limited to making, I know how much of my AWS traffic is human, and it's really very very very small. The sad reality is I'm sick and tired of rogue bots, and the tiny sliver of collateral damage can fill out the CAPTCHA validation every so often. (I also blacklist GWS, rackspace, linode, softlayer, reliablehosting, ovh.net, node4, netdirect, layer42, all TOR exits... it's actually a pretty huge list.)
        sjwright 15y ago
        I whitelist archive.org, and they've never hit through AWS.
ben1040 15y ago
I don't scrape, but I use an EC2 micro instance as a VPN endpoint when I am on an unsecured wifi network. It's a little annoying that some sites block me solely because I'm coming in from AWS (StackOverflow, I'm looking at you).
- koko775 15y ago
  I suggest using prgmr.com instead. $6/mo is worth a Xen instance with IPv6 and everything.
mike-cardwell 15y ago
Most admins? Most admins probably don't even know what AWS is. Most admins probably don't even block a single IP. I would be surprised if as many as 0.1% of admins block AWS.
semanticist 15y ago
In my experience no UK newspapers block AWS IP addresses.
Rickasaurus 15y ago
I usually use Tor when I scrape anyway :)

We have a pretty big e-commerce scraping project, and we don't really run into many problems regarding blacklisted ip's. There are a few sites who we get consistent bans from, but with elastic ip's it's pretty much a non-issue. I have yet to see a site that ban's all amazon ip's.

mattgreenrocks 15y ago
It is annoying when it happens and you're on something like Heroku.
sjwright 15y ago
I run http://whirlpool.net.au http://whirlpool.net.au and I religiously check the Amazon EC2 forum announcements[1] for new IP ranges to ban. [1] https://forums.aws.amazon.com/ann.jspa?annID=1030 https://forums.aws.amazon.com/ann.jspa?annID=1030
- stoney 15y ago
  > I run http://whirlpool.net.au http://whirlpool.net.au An excellent and very useful forum! It seems like whatever topic I'm searching for, google(.com.au) returns a useful result on your site.
- aw3c2 15y ago
  Would you mind telling why?
- maaku 15y ago
  Why? Name me one good reason. Name me one.
  - sjwright 15y ago
    Shitloads of rogue bots doing "social media monitoring". Shitloads of rogue bots stealing content for black-hat SEO. Shitloads of rogue bots harvesting email addresses. Shitloads of rogue bots submitting spammy replies.
    - nupark2 15y ago
      That's unfortunate. I don't know what the answer is, but real people do route their traffic out of AWS endpoints.
      - sjwright 15y ago
        Far fewer than you might think.
    - maaku 15y ago
      So maintain a blacklist of elastic IPs. If it's too big for you, make it a community effort. Those are bad reasons to close your site to all of AWS. As nupark2 mentioned, there are legitimate users routing traffic through EC2, even some bots that you'd want to visit your site. Archive.org comes to mind (many of there scrapers are or were behind AWS). Closing your site or app to a large swath of the web is the wrong solution. It's like killing a spider with a bazooka.
      - sjwright 15y ago
        Unlike the assumptions you're limited to making, I know how much of my AWS traffic is human, and it's really very very very small. The sad reality is I'm sick and tired of rogue bots, and the tiny sliver of collateral damage can fill out the CAPTCHA validation every so often. (I also blacklist GWS, rackspace, linode, softlayer, reliablehosting, ovh.net, node4, netdirect, layer42, all TOR exits... it's actually a pretty huge list.)
      - sjwright 15y ago
        I whitelist archive.org, and they've never hit through AWS.

ben1040 15y ago

I don't scrape, but I use an EC2 micro instance as a VPN endpoint when I am on an unsecured wifi network. It's a little annoying that some sites block me solely because I'm coming in from AWS (StackOverflow, I'm looking at you).

koko775 15y ago
I suggest using prgmr.com instead. $6/mo is worth a Xen instance with IPv6 and everything.

mike-cardwell 15y ago

Most admins? Most admins probably don't even know what AWS is. Most admins probably don't even block a single IP. I would be surprised if as many as 0.1% of admins block AWS.

semanticist 15y ago

In my experience no UK newspapers block AWS IP addresses.

Rickasaurus 15y ago

I usually use Tor when I scrape anyway :)