7 ms·

The State of Web Scraping 2022

291 points by Ian_Kerins 5y ago

JJxFile 5y ago
The web scraping ecosystem is growing, with more libraries, frameworks and products available than ever before to simplify our web scraping headaches so the future is looking bright.
joe_91 5y ago
I'm scraping about 30 sites for work at the moment, but have a few that are using Cloudflare which has been a b*tch to deal with. Tried numerous libraries and different proxy providers, but reliability is patchy. Previous fixes like https://github.com/Anorov/cloudflare-scrape https://github.com/Anorov/cloudflare-scrape don't seem to work anymore after Cloudflare updates, so I've switched to using a pretty optimised headless browser with good proxies instead.
- Ian_Kerins 5y ago
  This has a lot of good info on how to cloudflare and others work, and more creative ways to bypass them if the easier options don't work https://incolumitas.com/2021/05/20/avoid-puppeteer-and-playwright-for-scraping/ https://incolumitas.com/2021/05/20/avoid-puppeteer-and-playw...
- nanna 5y ago
  I'm finding that Cloudflare is even blocking my RSS reader from requesting feeds behind their service. It's not even just scrapers at this point.
- valar_m 5y ago
  Do you have any recommendations for the "good proxies" you mentioned?
- emptysea 5y ago
  Is the “pretty optimized headless browser” an off the shelf thing, or something custom? Are you using playwright/puppeteer to drive it?
  - mycall 5y ago
    Headless Chrome [0] and alpine-Chrome [1] are pretty popular. Some variations also include V2Ray, Shadowsocks and other VPNs. [0] https://hub.docker.com/r/justinribeiro/chrome-headless/ https://hub.docker.com/r/justinribeiro/chrome-headless/ [1] https://github.com/Zenika/alpine-chrome https://github.com/Zenika/alpine-chrome
  - rozenmd 5y ago
    There are plugins for Puppeteer: https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth https://github.com/berstend/puppeteer-extra/tree/master/pack...
- temp8964 5y ago
  I think it will eventually goes to like stock trading. If you have a good strategy, you don't want to share with the world, because it will render your strategy useless.
- nsonha 5y ago
  > optimised headless browser with good proxies instead are you saying you only had problem because you didn't use headless browser before and now with both headless and proxy it generally suffices to not be seen as scrapper?
Ian_Kerins 5y ago
If anyone has anything else they think was missed or should be included then let me know!
newsbinator 5y ago
Like most here, I am very good at web scraping and automated form fills. I keep trying to figure out a profitable side project or business idea to make out of it and keep coming up with nothing that works. Any good ideas?
- stef25 5y ago
  For a while I had a hobby project that would scrape real estate websites listing properties in my city. Goal was to try and figure out trends, pricing data, find good deals. Eventually the site added those features itself (heatmaps based on prices, for example) With all that data you can do stuff like make heatmaps from pricing data, figure out the most attractive areas for certain profiles (singles, families, ...). You could then mash up that data to produce things like a "Walkscore" or let people indicate what's important for them (green areas, bars & restaurants, time & distance to other destinations, even crime levels) and then show real estate that meets their criteria. Some sites in the US already show this but in other countries that's not the case, while the data's all there just to grab. Most likely it wouldn't be legal and certainly not if you made money from it. But it's incredibly fun and hugely useful. Maybe that could get you started on some ideas!
- Fantosism 5y ago
  I know many people that follow limited/exclusive releases for things like Yeezy/Air Jordan sneakers as well as PS5's and graphics cards. They pay $500/mo for access to a bot that will allow them to make these purchases. Most of the community lives on discord.
  - mschuster91 5y ago
    I understand people using bots to snipe PS5s and GPUs, these have real economic value and actual usage. But what other than artificial scarcity drives people to spend hundreds of dollars on bots to snipe sneakers?!
    - pjc50 5y ago
      Same as NFTs: hype and resale value. At least you can wear the sneakers once you've stopped flipping them.
ok_coo 5y ago
Time for me to advocate again for people to use Common Crawl. Please don't slam peoples' websites, look for alternatives before scraping. There are probably other, better options. APIs, data set downloads, etc. https://commoncrawl.org/ https://commoncrawl.org/
- dewey 5y ago
  I'd guess that for the many popular scraping uses cases this is not really useful as it's usually about being quick and up to date (job postings, availability information, e-commerce, serps,...) not about having a big corpus of historic data.
- mycall 5y ago
  I wish web.archive.org had an index by someone like common crawl. There is lots of great stuff on archive.org
  - wumpus 5y ago
    web.archive.org has a CDX index, similar to Common Crawl. Since I use both of these archives together, I wrote this code to iron out the differences between them: https://github.com/cocrawler/cdx_toolkit https://github.com/cocrawler/cdx_toolkit
    - kevinsundar 5y ago
      Hey! I was using your tool a couple months ago. It was super helpful for my project.
      - wumpus 5y ago
        Thanks! I rarely hear from users, great to hear from you!
  - kevinsundar 5y ago
    They do and its better than common crawl's by my testing.
NDizzle 5y ago
I still have a daily job running a web scraper I first wrote with Scrapy back in 2017. I think I've had to update it 3 times over the years for changes to the site and web standards. Good old government sites - rarely change!
cblconfederate 5y ago
Cloudflare's blocks get in the way of many websites who are simply trying to get a "link preview" of the page, even if it is only a single request from a new IP. I wish they would offer some kind of alternative for the pages they serve instead of a captcha block.
fareesh 5y ago
My toolbox of choice for web scraping is either Nokogiri or puppeteer Can someone sell me on beautiful soup or scrapy or any of the others? Do they provide any advantages or features that I'd be missing out on?
- edmundsauto 5y ago
  One great scrapy feauture is caching the page content. So you can essentially write a crawler, and when that’s running, you write your extraction code. Then, if you want to go back, you can add more extractors and run it against your local copy.
  - fareesh 5y ago
    Ah interesting, I end up doing this manually, i.e. File.write followed by what I want to scrape
    - edmundsauto 5y ago
      I believe scrapy has somewhat intelligent cache control options - maybe it could be recreated in a few dozens of lines of code, maybe a few hundred. But there are a huge number of these types of features - it’s basically a Swiss Army knife. Examples include rotating proxies, rotating user agent headers. Hooks to add in middleware for processing pipelines. CLI switches to change your data output format. Nice debugging and logging. Other large scale features include distributed crawlers. Scheduling. Monitoring UI so you can see progress via a web UI. It’s what I reach for first, because you can be up and running with your first scraper in an hour. By hand, that’s maybe 10 minutes - but if you want to iterate, and your first scraper is a v1 rather than final effort… i think it’s definitely worth it.
slvrspoon 5y ago
for those in this thread with super-serious experience scraping and automating at scale, looking for work (ethical!) please contact me directly.
mellosouls 5y ago
With the right combination of proxies, user agents and browsers, you can scrape every website. Even those that seem unscrapable. : This outcome was great news for web scrapers, as it means that so long as a websites has made their data public you are not in violation of the CFAA when you scrape the data even if it is prohibited in some other way (T&Cs, robots.txt, etc). Just because you can, doesn't mean you should. It would be better I think if there was a treatment of the ethics here, rather than a seemingly "ra-ra go bots" attitude, as though the only consideration is commercial.
- bryanrasmussen 5y ago
  this sort of implies that the 'ethics' would end up meaning that you shouldn't scrape if it is not wanted, although I suppose there can be ethics or other than commercial requirements that mean that you should.
- Ian_Kerins 5y ago
  100% agree, when scraping it should always be done respectfully. - If they provide a API, then use it. - Don't slam a website, ideally spread it out over hours of the day when there target audience is least active (night time). - If you can get cached data from somewhere that works, then use that. Most developers are respectful and only scrape what they really need, not only from an ethical point of view but also a cost and resources point of view. Scraping data is resource intensive and proxy costs can quickly rise to $1,000-$10,000 per month. So most only scrape the minimum they need. The other thing here as well, is that a lot of the most popular sites being scraped, are also massive scrapers themselves. The big ecommerce sites are being scraped, but they are also scraping their competitors too.
  - travisporter 5y ago
    Don’t get my home address, name, family members names, salary, cell phone number, aggregate and sell them and claim “it’s all publically available anyway”
    - RobSm 5y ago
blantonl 5y ago
I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opinion of it is that those that are doing it to my platforms are doing so solely to steal data and build businesses on top of other's hard work.
- jeroenhd 5y ago
  Scraping itself isn't universally unethical. Google and Bing scraping websites to make information easier to access is fine, and scraping and analysing government data is even better. Public data should be public, after all. However, the disgusting data brokers that employ most of the custom scrapers, are usually unethical. That's why I don't trust any person or company that admits being involved professionally in "scraping", because most of the time that means "we collect personal information that got leaked elsewhere and sell them on".
- yashasolutions 5y ago
  Google is web scrapper number one, as any search engine. Making web scrapping illegal mean making search engine illegal. You do not want information to be public and/or free? Put it under login and charge for it. You want to prevent people to reuse the data you publish to build other (potentially competitive) products, then use licensing and copyright, and the law. However, banning a technological mean because what a minority could potentially do with it? Then make the internet illegal then and the problem is fixed altogether.
  - FinanceAnon 5y ago
    What if Google didn't scrape websites automatically, and waited till users submit their domains to them, to mark that they want to be scraped? I think in that case, most users would still submit their domains there, because they want to come up in Google search. You might want your website to be scraped by some people/companies and not by others, but not have to put everything behind a login screen (which some determined scrapers would still try to breach in some way).
KieranMac 5y ago
As a lawyer whose primary focus is in web scraping, this article is in many ways misleading and inaccurate. While it is true that the Van Buren case is generally positive for web scraping, the overall legal landscape is still murky. The main battleground for web scraping legal issues is shifting from the CFAA to breach of contract and various state-law issues, including misappropriation, unjust enrichment, and trespass to chattels. In my opinion, 2021 was a bad year for the law as it relates to web scraping. The Supreme Court remanded hiQ Labs, and many high-profile lower-court cases ended badly for web scrapers. It's a darker shade of gray than it was in 2020. It can be navigated, but it's tricky.
- digitcatphd 5y ago
  Good take, IMO ethically speaking we should not penalize scrapers themselves but do so based on their use. Scraping Facebook to make a clone of profiles shouldn’t be held to the same scrutiny of scraping Facebook to do an internal analysis of user demographics for research purposes.
  - ForHackernews 5y ago
    Why should either be discouraged?
    - antonf 5y ago
      With cloned profiles (or any data obtained and shared without your consent) it will be harder for you to exercise your right to be forgotten, for example.
    - zamadatix 5y ago
      Cloning profiles is what seems likely wrong to me but I'm not sure how that being done via scraping or not should matter.
- RobSm 5y ago
  How many contracts google breaches scraping billions of pages every month?
bobblywobbles 5y ago
Not a lawyer, but many terms of service prohibit interacting with their website in an automated fashion, as well as collecting their data. In my understanding, scraping a site with these terms already puts you in the wrong.
- akersten 5y ago
  > many terms of service prohibit interacting with their website in an automated fashion, Ignoring the fact that I didn't agree to anything just by virtue of requesting a page from a webserver (and, your server sent me the data!), that's such a meaningless phrase that it's certainly unenforceable. What is an automated fashion? Do I have to manually craft my HTTP request by hand-pulsing a voltage on an Ethernet cable, or do I have your permission to let Chrome automate that for me?
  - RobSm 5y ago
    This is so exactly. People do not realize that when they use chrome to view website, chrome is their 'scraper'. And the goal of webs craping is not to get illegal data, but to have efficiency and performance by not doing something manually but letting computer do the repetitive tasks. It's a productivity tool. You can't make something illegal just because it's an automation instead of 'manual' operation.
    - nsonha 5y ago
      are you a lawyer? Your opinion doesn't really mean anything if you still lose the case at the end. By your logic there isn't a clear way to define DDoS either. Sounds like there is though?
      - Chris2048 5y ago
        > there isn't a clear way to define DDoS either It isn't clear to me that there is. The difference seems to lie in intent. You could maybe nail a group making many requests without using the data for anything as making many spurious requests and hence having ill-intent, I suppose. Maybe having dedicated servers for such a tasks prove it even more?
coverj 5y ago
I have been interested in web scraping lately but never really dived too deep. Did anyone have more indepth resources (github projects, blogs, forums, etc) than the tutorials that are basically install beautiful soup and get data from a tag?
- JimBlackwood 5y ago
  Genuine question but, what more do you need?
gmanis 5y ago
What does HN think of web scraping for the purpose of price comparison? I’m asking this because I run a small side project to show prices across retailers for a very small niche. The users are very very happy. Even the vendors started contacting to be listed on the comparison. But I am unable to make a business out of it other than few affiliate commission.
- magixx 5y ago
  I worked for a company that did exactly this many years ago. (They were even able to parter with some retailer). Their product worked well yet they still went out of business long ago. To be honest, I don't see much value in such a service, not that it doesn't exist, it's just hard to justify paying for this data.
darepublic 5y ago
Separate from web scraping, there is the use of automation to perform normal allowable user actions on the site. That should be considered distinct from large scale data extraction no

291 points by Ian_Kerins 5y ago

The web scraping ecosystem is growing, with more libraries, frameworks and products available than ever before to simplify our web scraping headaches so the future is looking bright.

joe_91 5y ago

I'm scraping about 30 sites for work at the moment, but have a few that are using Cloudflare which has been a b*tch to deal with. Tried numerous libraries and different proxy providers, but reliability is patchy. Previous fixes like https://github.com/Anorov/cloudflare-scrape https://github.com/Anorov/cloudflare-scrape don't seem to work anymore after Cloudflare updates, so I've switched to using a pretty optimised headless browser with good proxies instead.

Ian_Kerins 5y ago
This has a lot of good info on how to cloudflare and others work, and more creative ways to bypass them if the easier options don't work https://incolumitas.com/2021/05/20/avoid-puppeteer-and-playwright-for-scraping/ https://incolumitas.com/2021/05/20/avoid-puppeteer-and-playw...
nanna 5y ago
I'm finding that Cloudflare is even blocking my RSS reader from requesting feeds behind their service. It's not even just scrapers at this point.
valar_m 5y ago
Do you have any recommendations for the "good proxies" you mentioned?
emptysea 5y ago
Is the “pretty optimized headless browser” an off the shelf thing, or something custom? Are you using playwright/puppeteer to drive it?
- mycall 5y ago
  Headless Chrome [0] and alpine-Chrome [1] are pretty popular. Some variations also include V2Ray, Shadowsocks and other VPNs. [0] https://hub.docker.com/r/justinribeiro/chrome-headless/ https://hub.docker.com/r/justinribeiro/chrome-headless/ [1] https://github.com/Zenika/alpine-chrome https://github.com/Zenika/alpine-chrome
- rozenmd 5y ago
  There are plugins for Puppeteer: https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth https://github.com/berstend/puppeteer-extra/tree/master/pack...
temp8964 5y ago
I think it will eventually goes to like stock trading. If you have a good strategy, you don't want to share with the world, because it will render your strategy useless.
nsonha 5y ago
> optimised headless browser with good proxies instead are you saying you only had problem because you didn't use headless browser before and now with both headless and proxy it generally suffices to not be seen as scrapper?

Ian_Kerins 5y ago

If anyone has anything else they think was missed or should be included then let me know!

newsbinator 5y ago

Like most here, I am very good at web scraping and automated form fills. I keep trying to figure out a profitable side project or business idea to make out of it and keep coming up with nothing that works. Any good ideas?

stef25 5y ago
For a while I had a hobby project that would scrape real estate websites listing properties in my city. Goal was to try and figure out trends, pricing data, find good deals. Eventually the site added those features itself (heatmaps based on prices, for example) With all that data you can do stuff like make heatmaps from pricing data, figure out the most attractive areas for certain profiles (singles, families, ...). You could then mash up that data to produce things like a "Walkscore" or let people indicate what's important for them (green areas, bars & restaurants, time & distance to other destinations, even crime levels) and then show real estate that meets their criteria. Some sites in the US already show this but in other countries that's not the case, while the data's all there just to grab. Most likely it wouldn't be legal and certainly not if you made money from it. But it's incredibly fun and hugely useful. Maybe that could get you started on some ideas!
Fantosism 5y ago
I know many people that follow limited/exclusive releases for things like Yeezy/Air Jordan sneakers as well as PS5's and graphics cards. They pay $500/mo for access to a bot that will allow them to make these purchases. Most of the community lives on discord.
- mschuster91 5y ago
  I understand people using bots to snipe PS5s and GPUs, these have real economic value and actual usage. But what other than artificial scarcity drives people to spend hundreds of dollars on bots to snipe sneakers?!
  - pjc50 5y ago
    Same as NFTs: hype and resale value. At least you can wear the sneakers once you've stopped flipping them.

ok_coo 5y ago

Time for me to advocate again for people to use Common Crawl. Please don't slam peoples' websites, look for alternatives before scraping. There are probably other, better options. APIs, data set downloads, etc. https://commoncrawl.org/ https://commoncrawl.org/

dewey 5y ago
I'd guess that for the many popular scraping uses cases this is not really useful as it's usually about being quick and up to date (job postings, availability information, e-commerce, serps,...) not about having a big corpus of historic data.
mycall 5y ago
I wish web.archive.org had an index by someone like common crawl. There is lots of great stuff on archive.org
- wumpus 5y ago
  web.archive.org has a CDX index, similar to Common Crawl. Since I use both of these archives together, I wrote this code to iron out the differences between them: https://github.com/cocrawler/cdx_toolkit https://github.com/cocrawler/cdx_toolkit
  - kevinsundar 5y ago
    Hey! I was using your tool a couple months ago. It was super helpful for my project.
    - wumpus 5y ago
      Thanks! I rarely hear from users, great to hear from you!
- kevinsundar 5y ago
  They do and its better than common crawl's by my testing.

NDizzle 5y ago

I still have a daily job running a web scraper I first wrote with Scrapy back in 2017. I think I've had to update it 3 times over the years for changes to the site and web standards. Good old government sites - rarely change!

cblconfederate 5y ago

Cloudflare's blocks get in the way of many websites who are simply trying to get a "link preview" of the page, even if it is only a single request from a new IP. I wish they would offer some kind of alternative for the pages they serve instead of a captcha block.

fareesh 5y ago

My toolbox of choice for web scraping is either Nokogiri or puppeteer Can someone sell me on beautiful soup or scrapy or any of the others? Do they provide any advantages or features that I'd be missing out on?

edmundsauto 5y ago
One great scrapy feauture is caching the page content. So you can essentially write a crawler, and when that’s running, you write your extraction code. Then, if you want to go back, you can add more extractors and run it against your local copy.
- fareesh 5y ago
  Ah interesting, I end up doing this manually, i.e. File.write followed by what I want to scrape
  - edmundsauto 5y ago
    I believe scrapy has somewhat intelligent cache control options - maybe it could be recreated in a few dozens of lines of code, maybe a few hundred. But there are a huge number of these types of features - it’s basically a Swiss Army knife. Examples include rotating proxies, rotating user agent headers. Hooks to add in middleware for processing pipelines. CLI switches to change your data output format. Nice debugging and logging. Other large scale features include distributed crawlers. Scheduling. Monitoring UI so you can see progress via a web UI. It’s what I reach for first, because you can be up and running with your first scraper in an hour. By hand, that’s maybe 10 minutes - but if you want to iterate, and your first scraper is a v1 rather than final effort… i think it’s definitely worth it.

slvrspoon 5y ago

for those in this thread with super-serious experience scraping and automating at scale, looking for work (ethical!) please contact me directly.

mellosouls 5y ago

With the right combination of proxies, user agents and browsers, you can scrape every website. Even those that seem unscrapable. : This outcome was great news for web scrapers, as it means that so long as a websites has made their data public you are not in violation of the CFAA when you scrape the data even if it is prohibited in some other way (T&Cs, robots.txt, etc). Just because you can, doesn't mean you should. It would be better I think if there was a treatment of the ethics here, rather than a seemingly "ra-ra go bots" attitude, as though the only consideration is commercial.

bryanrasmussen 5y ago
this sort of implies that the 'ethics' would end up meaning that you shouldn't scrape if it is not wanted, although I suppose there can be ethics or other than commercial requirements that mean that you should.
Ian_Kerins 5y ago
100% agree, when scraping it should always be done respectfully. - If they provide a API, then use it. - Don't slam a website, ideally spread it out over hours of the day when there target audience is least active (night time). - If you can get cached data from somewhere that works, then use that. Most developers are respectful and only scrape what they really need, not only from an ethical point of view but also a cost and resources point of view. Scraping data is resource intensive and proxy costs can quickly rise to $1,000-$10,000 per month. So most only scrape the minimum they need. The other thing here as well, is that a lot of the most popular sites being scraped, are also massive scrapers themselves. The big ecommerce sites are being scraped, but they are also scraping their competitors too.
- travisporter 5y ago
  Don’t get my home address, name, family members names, salary, cell phone number, aggregate and sell them and claim “it’s all publically available anyway”
  - RobSm 5y ago

blantonl 5y ago

I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opinion of it is that those that are doing it to my platforms are doing so solely to steal data and build businesses on top of other's hard work.

jeroenhd 5y ago
Scraping itself isn't universally unethical. Google and Bing scraping websites to make information easier to access is fine, and scraping and analysing government data is even better. Public data should be public, after all. However, the disgusting data brokers that employ most of the custom scrapers, are usually unethical. That's why I don't trust any person or company that admits being involved professionally in "scraping", because most of the time that means "we collect personal information that got leaked elsewhere and sell them on".
yashasolutions 5y ago
Google is web scrapper number one, as any search engine. Making web scrapping illegal mean making search engine illegal. You do not want information to be public and/or free? Put it under login and charge for it. You want to prevent people to reuse the data you publish to build other (potentially competitive) products, then use licensing and copyright, and the law. However, banning a technological mean because what a minority could potentially do with it? Then make the internet illegal then and the problem is fixed altogether.
- FinanceAnon 5y ago
  What if Google didn't scrape websites automatically, and waited till users submit their domains to them, to mark that they want to be scraped? I think in that case, most users would still submit their domains there, because they want to come up in Google search. You might want your website to be scraped by some people/companies and not by others, but not have to put everything behind a login screen (which some determined scrapers would still try to breach in some way).

KieranMac 5y ago

As a lawyer whose primary focus is in web scraping, this article is in many ways misleading and inaccurate. While it is true that the Van Buren case is generally positive for web scraping, the overall legal landscape is still murky. The main battleground for web scraping legal issues is shifting from the CFAA to breach of contract and various state-law issues, including misappropriation, unjust enrichment, and trespass to chattels. In my opinion, 2021 was a bad year for the law as it relates to web scraping. The Supreme Court remanded hiQ Labs, and many high-profile lower-court cases ended badly for web scrapers. It's a darker shade of gray than it was in 2020. It can be navigated, but it's tricky.

digitcatphd 5y ago
Good take, IMO ethically speaking we should not penalize scrapers themselves but do so based on their use. Scraping Facebook to make a clone of profiles shouldn’t be held to the same scrutiny of scraping Facebook to do an internal analysis of user demographics for research purposes.
- ForHackernews 5y ago
  Why should either be discouraged?
  - antonf 5y ago
    With cloned profiles (or any data obtained and shared without your consent) it will be harder for you to exercise your right to be forgotten, for example.
  - zamadatix 5y ago
    Cloning profiles is what seems likely wrong to me but I'm not sure how that being done via scraping or not should matter.
RobSm 5y ago
How many contracts google breaches scraping billions of pages every month?

bobblywobbles 5y ago

Not a lawyer, but many terms of service prohibit interacting with their website in an automated fashion, as well as collecting their data. In my understanding, scraping a site with these terms already puts you in the wrong.

akersten 5y ago
> many terms of service prohibit interacting with their website in an automated fashion, Ignoring the fact that I didn't agree to anything just by virtue of requesting a page from a webserver (and, your server sent me the data!), that's such a meaningless phrase that it's certainly unenforceable. What is an automated fashion? Do I have to manually craft my HTTP request by hand-pulsing a voltage on an Ethernet cable, or do I have your permission to let Chrome automate that for me?
- RobSm 5y ago
  This is so exactly. People do not realize that when they use chrome to view website, chrome is their 'scraper'. And the goal of webs craping is not to get illegal data, but to have efficiency and performance by not doing something manually but letting computer do the repetitive tasks. It's a productivity tool. You can't make something illegal just because it's an automation instead of 'manual' operation.
  - nsonha 5y ago
    are you a lawyer? Your opinion doesn't really mean anything if you still lose the case at the end. By your logic there isn't a clear way to define DDoS either. Sounds like there is though?
    - Chris2048 5y ago
      > there isn't a clear way to define DDoS either It isn't clear to me that there is. The difference seems to lie in intent. You could maybe nail a group making many requests without using the data for anything as making many spurious requests and hence having ill-intent, I suppose. Maybe having dedicated servers for such a tasks prove it even more?

coverj 5y ago

I have been interested in web scraping lately but never really dived too deep. Did anyone have more indepth resources (github projects, blogs, forums, etc) than the tutorials that are basically install beautiful soup and get data from a tag?

JimBlackwood 5y ago
Genuine question but, what more do you need?

gmanis 5y ago

What does HN think of web scraping for the purpose of price comparison? I’m asking this because I run a small side project to show prices across retailers for a very small niche. The users are very very happy. Even the vendors started contacting to be listed on the comparison. But I am unable to make a business out of it other than few affiliate commission.

magixx 5y ago
I worked for a company that did exactly this many years ago. (They were even able to parter with some retailer). Their product worked well yet they still went out of business long ago. To be honest, I don't see much value in such a service, not that it doesn't exist, it's just hard to justify paying for this data.

darepublic 5y ago

Separate from web scraping, there is the use of automation to perform normal allowable user actions on the site. That should be considered distinct from large scale data extraction no