Thanks for the feedback! I’ll keep that in mind as this is built out. Fortunately the initial bootstrapping uses data from the Internet Archive and the crawls a

Thanks for the feedback! I’ll keep that in mind as this is built out. Fortunately the initial bootstrapping uses data from the Internet Archive and the crawls afterwards is to check for updates (at a reasonable rate). The number of URLs being hit is much much lower in the end than you would think.