7 ms·
Techniques scraping vulnerability data with over 100 crawlers (without LLMs)
- louisstow 1y agoI wanted to share some of my techniques for managing and scaling 100+ unique web crawlers. I've tried a few approaches but these techniques stand out as having the most impact in terms of quick iteration and being resilient to breaking changes. Interested in hearing different approaches.