4 ms·
This is very interesting, although anyone aiming to crawl Wikipedia should make sure they read this section on the Database download page. http://en.wikipedia.
by decad 14y ago
This is very interesting, although anyone aiming to crawl Wikipedia should make sure they read this section on the Database download page.
http://en.wikipedia.org/wiki/Wikipedia:Database_download#Why_not_just_retrieve_data_from_wikipedia.org_at_runtime.3F http://en.wikipedia.org/wiki/Wikipedia:Database_download#Why...
Everything should be fine as long as you respect their 1 request per second rule and their robots.txt
- kefs 14y agoA quick skim of the source shows that rate-limiting is not implemented, and the code is non-compliant with Wikipedia's crawling rules.
- BenjaminCoe 14y agoThanks for the heads up. I'll add rate-limiting directly into the API.