dmn001

6 ms·

1.
by dmn001 9y ago
I find there is really no need to hide or mask the IP address when web scraping. The use of proxies or Tor to do so is completely unnecessary and maybe prohibitive e.g. try using Google in Tor.
2.
by dmn001 9y ago
By caching, I just mean storing of data locally so you don't have to request it again under a certain timeframe. I use my own caching scripts written in Python, if you use a 3rd party library then data deletion does not matter too much
3.
by dmn001 9y ago
There is no issue with parsing and scraping in the same loop as long as there is caching in there as well. You don't want to be hitting the server repeatedly whilst you're debugging. A project like Scrapy should have caching on by
4.
by dmn001 9y ago
On the contrary, I have found lxml suitable for all of my scraping projects where the objective is to write some XPath to parse or extract some data from some element.
5.
by dmn001 9y ago
That may be fine for javascript heavy websites for a site with a few pages, but for anything with more than say 1,000 pages it is much more efficient to scrape using requests with lxml. The requests can be made concurrently, are scalable an
6.
by dmn001 9y ago
SVG flowcharts of many gamebooks including FF and Lone Wolf: http://outspaced.fightingfantasy.net/SVG_Flowcharts/main.htm...
7.
by dmn001 9y ago
Should be fixed now.
8.
by dmn001 9y ago
I made .svg diagrams for the Fighting Fantasy books Warlock of Firetop Mountain and Deathtrap dungeon a while back on my old blog: https://daveman.wordpress.com/2010/01/08/how-to-create-svg-l...
9.
by dmn001 9y ago
No. Most websites don't do this.
10.
by dmn001 9y ago
It's extremely rare to be ip-blocked by any website just for using the Google's user agent from a non-specific range. IP's get re-used and you can switch to a new one easily, so it's really not common or good practice fo
11.
by dmn001 9y ago
SEEKING WORK - England,UK/Remote Over 7 years experience with Python software development, cloud services, data mining, web crawling, databases. Want to extract or crawl data from a website such as business listings, sports data, gover
12.
by dmn001 10y ago
The first part seems like a very long-winded way to say "don't use the default user agent". The captcha was unusually simple to solve, in most cases the best strategy is to avoid seeing it in the first place.
13.
by dmn001 10y ago
Right, if you're just matching a substring in a list of strings then you don't need regex for that. Also, lxml is over 20x faster than BeautifulSoup, although it doesn't really matter with such a low frequency of requests, it
14.
by dmn001 10y ago
SEEKING WORK - UK/Remote Over 7 years experience with Python and Perl web scraping, cloud services, databases and Linux administration. Want to extract or crawl data from a website such as business listings, sports data, directories, e
15.
by dmn001 10y ago
I don't think I have the other videos, IIRC he removed or made private the previous videos shortly after the later ones were uploaded and that was the impetus for downloading them in the first place. Btw, here is an another video of Ga
16.
by dmn001 10y ago
I found some of Vova's videos: https://www.youtube.com/watch?v=_zG_QpwbV3E https://www.youtube.com/watch?v=9csBR4AcSAQ https://www.youtube.com/watch?v=XfGbWbWSr00 https://ww
17.
by dmn001 10y ago
It would just be much easier to use the Facebook graph api, there is an official Python module and is well documented, and would be less likely to hit rate limits or other blocks - ironically that was one of the reasons that the author used
18.
by dmn001 11y ago
SEEKING WORK - UK/Remote Over 5 years experience with Python and Perl web scraping, cloud services, databases and Linux administration. Want to extract or crawl data from a website such as business listings, sports data, directories, e
19.
by dmn001 11y ago
Location: UK Remote: Yes. Willing to relocate: Yes. Technologies: Python, Perl, Matlab, R, MongoDB, MySQL, Git Résumé/CV: available on request Email: dmn001 gmail
20.
by dmn001 11y ago
Btw, I have no association with this company, other than I like to keep track of their blog and events regularly as it ties in quite closely with my line of work. Thought I'd post as it's an article that fits in the startup cultur
21.
What happens when you turn up for work and find out you've been evicted(blog.import.io)
8 points by dmn001 11y ago|4 comments
22.
by dmn001 12y ago
SEEKING WORK - UK/Remote 5+ years experience with Python and Perl web scraping, cloud services, databases and Linux administration. Want to extract or crawl data from a website such as business listings, sports data, directories, eithe
23.
by dmn001 12y ago
> the entire content of The Pirate Bay > Not the P2P content that people download I think you're confusing different definitions of 'content' here. Also, there are many more sites that have publicly available indexes of
24.
by dmn001 12y ago
SEEKING WORK - remote or UK 5+ years experience with Python and Perl web scraping, cloud services, databases and Linux administration. Need to extract or crawl data from a website such as business listings, sports data, directories, either
25.
by dmn001 12y ago
Yes, I heard that one is expensive. Just out of interest, what are you using it for?
26.
by dmn001 12y ago
Is there any documentation for that api, from the url it looks that they created it specifically for their official mobile app only?
27.
by dmn001 12y ago
Other browser based screen scrapers that are in the space are 80 legs, kiminolabs, Mozenda and OutWit Hub, I'm sure there are more. Last time I checked, import.io was a fairly lightweight browser wrapper. I also write web scrapers usin
28.
by dmn001 12y ago
Looks like it just uses browser-side js requests to get the search pages, so it would use minimal bandwidth.
29.
by dmn001 12y ago
The problem with Leap is that to do anything remotely accurately for a moderate period of time takes an huge amount of effort moving your arm in mid-air with only visual feedback. It will be marginally worse than using a wii controller to t
30.
by dmn001 12y ago
Predictive typing for single words is usually based on a locally stored dictionary, and for sentences it can be done with a model like prediction by partial matching. The device functionality is similar to a system used for speech recogniti

More ›