TheEspion

16 ms·

1.
by TheEspion 12y ago
Yes!
2.
by TheEspion 12y ago
The entire anonymous IP infrastructure is indeed built-in. Yes, performance is lower than non-JS-enabled scraping. To counter this we offer a variety of options to turn on/off performance trade-offs. They can be set per job and per pag
3.
by TheEspion 12y ago
That's a feature we never thought of. I can't promise it will be available at release time, but quite possibly later next year.
4.
by TheEspion 12y ago
We approach this the same way e-commerce or gaming companies approach fraud: we'll actively monitor the infrastructure and set up safeguards to prevent spam and other illegitimate activities.
5.
by TheEspion 12y ago
Very interesting question. HTML5 audio and video are definitely a possibility, if there's a strong use case. If you have a specific idea, I'd be interested to hear it. I don't expect Flash or Java support. Flash and Java apps
6.
by TheEspion 12y ago
You gain access to the page's own DOM and JavaScript, which lets you call the site's functions to fetch data if it helps you. You can also code your scraper with the same techniques you use when building a page: jQuery, CSS select
7.
by TheEspion 12y ago
Actually no, and I don't think our location would protect us from legal liability. Our servers are outside Mauritius, hosting locally would be very expensive and induce latency.
8.
by TheEspion 12y ago
It's more complex than that. Whether the site's TOS are enforceable is a matter of jurisdiction and may depend on the intent of the web scraper. Ryanair for example has lost several cases where they tried to forbid scraping their
9.
by TheEspion 12y ago
Espion is similar to PhantomJS in that they're both headless browsers that you can inject JavaScript into. Espion comes with a lot more. First there is the infrastructure: processing power, storage, connectivity and IP addresses that y
10.
by TheEspion 12y ago
CAPTCHAs and IP blacklistings are things you encounter routinely in perfectly legitimate and legal web scraping projects. A typical example are businesses that want to monitor their competitors' prices or the second hand market for the
11.
by TheEspion 12y ago
Thanks, we'll definitely do that.
12.
by TheEspion 12y ago
The packages are for a set number of pages, with no timeframe. I don't have prices yet for a load in the millions of pages per hour. What kind of system do you feed data into?
13.
by TheEspion 12y ago
We target businesses that want to fulfil their web data extraction needs in-house rather than hiring a third-party provider – which we originally are. Espion was built for our own needs at first and we’re still in the customer discovery pha
14.
Show HN: Scrape the web by injecting JavaScript into web pages(espion.io)
109 points by TheEspion 12y ago|54 comments