6 ms·
WhaleEye SEO Spider - http://www.whalei.com http://www.whalei.com I created the same thing a year ago. I underestimated the CPU load and number of pages on man
by primaryobjects 13y ago
WhaleEye SEO Spider - http://www.whalei.com http://www.whalei.com
I created the same thing a year ago. I underestimated the CPU load and number of pages on many sites (especially e-commerce sites). It doesn't take many crawls to quickly fill a db.