5 ms·
It’s not that hard. I’ve done this. The list of URLs for a crawl is several hundred gigs. Easily fits in a lookup index on a single instance.
by binarymax 2mo ago
It’s not that hard. I’ve done this. The list of URLs for a crawl is several hundred gigs. Easily fits in a lookup index on a single instance.