5 ms·
How do you determine which sites get crawled?
by jrruethe 6y ago
How do you determine which sites get crawled?
- nsturtz 6y agoI use the CLI command (php spider.php -u <url> -f (full) -l (leave domain) -r (re-crawl) that is How I crawl. I crawl websites that I use on a daily basis, I also target websites that are for open source projects and the likes, but once I start my spider on a website it will just keep going and going and going hopping different websites. and if you want you can submit a url to nate.sturtz@sturtz.cf it just has to be a SFW website. I hope this answered your question.
- nsturtz 6y ago@jruethe did this answer your question?