5 ms·
Are there any details about how they're crawling the web? I've never encountered the BraveBot User-Agent and never heard of such crawler.
by kaivi 3y ago
Are there any details about how they're crawling the web? I've never encountered the BraveBot User-Agent and never heard of such crawler.
- EMM_386 3y agoI'm not sure about the crawler itself, but they do get some of their data from the Web Discovery Project. What percentage of it, I have no idea. https://support.brave.com/hc/en-us/articles/4409406835469-What-is-the-Web-Discovery-Project- https://support.brave.com/hc/en-us/articles/4409406835469-Wh...