7 ms·
750k items in their content management sysem. N independent labs crawling wanting to check that every day could easily give bursts of millions per hour Million
by procaryote 2mo ago
750k items in their content management sysem. N independent labs crawling wanting to check that every day could easily give bursts of millions per hour
Millions per hour is tens per second though; perhaps the fix is performance improvements
- Jabbles 2mo ago1,000,000 / 3,600 = ~278
- setupminimal 2mo agoWe have put in a number of performance improvements, yes. The nice thing about those is that they also make the site snappier when it's not under load. Right now we have just over a million items in our CMS, plus our publicly available mailing list archives, which are much larger, even if they're less frequently referenced.
- andai 2mo agoYeah we can just rewrite the web servers in Rust ;) That'll be great until.. they rewrite the scrapers in Rust! Then we're really hosed!