throwawayscrapd

5 ms·

1. 1.▲
   by throwawayscrapd 1y ago

   And maybe you could Bing and learn what "cache eviction" is and why that happens when a crawler systematically hits every page on your site.
2. 2.▲
   by throwawayscrapd 1y ago

   I mean given that you've had this problem repeatedly, I'd call it "past-proofing", but I suppose you know your codebase better than I do.
3. 3.▲
   by throwawayscrapd 1y ago

   Did you ever consider refactoring the code so that you don't have to do shotgun surgery every time you make this kind of change?
4. 4.▲
   by throwawayscrapd 1y ago

   I work on the kind of big online scientific database that this article is about. 100% of our data is available from a clearly marked "Download" page. We still have scraper bots running through the whole site constantly. We are not