7 ms·
My instinct was also to use LLMs for this, but it was way to slow and still expensive if you want to scrape millions of pages.
by vetler 6mo ago
My instinct was also to use LLMs for this, but it was way to slow and still expensive if you want to scrape millions of pages.
- andrew_zhong 6mo agoPut things to perspective - Gemini 2.5 flash is 0.3/1M tokens - assuming each page is 700 tokens and output is not much you are looking at $210 for 1M pages
- vetler 6mo agoYou will absolutely struggle to get all the info you need into 700 tokens per page. Edit: There's also the added complexity of running a browser against 1M pages, or more.
- andrew_zhong 6mo agoI agree that When pages have similar structure, for one time extraction as it is (not reasoning from context), scraping with selectors is the way to go. This library also supports HTML as input so running a browser is not required.
- vetler 6mo agoCame back here to say I was wrong! I have been experimenting, and it is doable. I have been experimenting with setting up a scraping pipeline with LLM enrichment since I wrote the comment above, and have very positive results so far. :)