7 ms·
Show HN: POC to scrape and structure HTML into JSON for RAG
Hey all,
I built a quick PoC that scrapes a webpage, sends the content to Gemini Flash, and outputs a clean, structured JSON — ready for RAG workflows.
In my case, I’ll use this structured data to enhance models by integrating external knowledge sources during the generation process.
Curious if you think this has potential or if there are any use cases I might have missed.
Happy to share more details if there's interest!
- mahi_novice 1y agoDo you mind sharing more about the implementation details? Any safeguards you have for the urls and all?
- nirvanist 1y agoBasically, I use a headless Chromium with Puppeteer to render the page. Then, some logic extracts and cleans the HTML content. Finally, I use Gemini with a specific schema to return a JSON response.
- mahi_novice 1y agoOkay, thanks!
- mahi_novice 1y agoHow do you plan to use it?
- nirvanist 1y agoAt the moment, I use it in client projects to build agents for their chat systems by adding RAG to models
- mahi_novice 1y agoOkay