5 ms·

How does BuiltWith decides which sites to crawl? assuming it is too complex to crawl everything. And how is the data/queries being sold without compromising it

• nedwin 11y ago
  I believe that they started via user submission of URLs to crawl. I used to submit tons of sites that I was curious about and eventually got their Chrome browser extension so I could find out the tech behind a site by clicking that little button. Not sure if this is how they scaled out to the volume of sites they now cover.