6 ms·
Ask HN: Have you used a good general purpose LLM-powered scraper?
I'm looking to either build or use a model or SaaS that could be used with a prompt like:
Find the 10 biggest realtors in [CITY], then extract all their listings within [BUDGET] in JSON with all the data you can find.
I'm looking for a tool that can use search inputs on any website, click next page buttons or handle infinite scroll.
Is there a tool like that on the market that actually delivers?What I've seen for now doesn't seem good enough.
- coderatlarge 3y agoOne precursor question would be whether an LLM can extract the data you want from raw html even when copy-pasted manually. In my limited experience we’re not quite there yet, but I’d be curious to hear of others have different experience - or better yet, actual measurements against a baseline scraper.
- FrenchDevRemote 3y agoOh it definitely can. I tried with GPT-4 and Anthropic(claude 2), it was relatively good, although I had to tweak the prompts to get everything I wanted(sometimes it forgot images or urls, because my prompt was too vague) You could also write a simple function that turn HTML into a json that only contained innerText, src and hrefs, and ask the LLM to only keep the relevant data
- mdev23 3y agoI am working on an LLM agents tool that could probably help with this: https://recurai.com https://recurai.com Reach out and i'll try to get you setup: matt@recurai.com