5 ms·
Show HN: Pulpie – Models for Cleaning the Web
Hey HN, I'm Shreyash, founder of Feyn. We built Pulpie, a family of Pareto optimal models for cleaning the web. Pulpie strips boilerplate (ads, footers, sidebars) from raw HTML and returns just the main content as HTML or Markdown.
We match SOTA extraction quality while being 20x cheaper. Cleaning 1 billion webpages costs $7,900 with Pulpie versus $159,000 with Dripper, the current leading extractor.
The gains come from architecture. Today's leading extractors are decoders that generate output one token at a time. Each step reads the full model from memory to produce a single token. Conversely, Pulpie models are encoders. They run one forward pass over the full input HTML and label each block as boilerplate or content. As a result, Pulpie is compute-bound while decoders are memory-bound. Cheaper GPUs have relatively more compute than memory bandwidth. This makes Pulpie easy to run optimally.
Here's Pulpie and Dripper cleaning the same pages side by side: https://www.youtube.com/watch?v=ibd-tIiQECo https://www.youtube.com/watch?v=ibd-tIiQECo. You can try a side-by-side comparison yourself: https://huggingface.co/spaces/feyninc/pulpie https://huggingface.co/spaces/feyninc/pulpie
Our motivation for Pulpie came from building a deep research harness. Every search API returns noisy content that contains ads, nav elements, and sidebars. In one instance, an ad for "Gemini on Pixel" slipped into our search results, got passed into LLM context, and ended up in the final answer served to the user. Pretty embarrassing moment for us but it helped us realize how bad data kills model intelligence. We built Pulpie to get clean data for cheap.
All models are open source on Hugging Face. You can read about our training process and how to use Pulpie here: https://usefeyn.com/blog/pulpie-pareto-optimal-models-for-cleaning-the-web/#get-started https://usefeyn.com/blog/pulpie-pareto-optimal-models-for-cl...
Happy to answer any questions!
- goldenjm 2mo agoThis model looks very interesting. How does it compare to heuristic-based approaches like Defuddle and Readability?
- andy_parhelia 2mo ago[dead]
- deleted 2mo ago[deleted]
- lnenad 2mo agoVery nice! Thank you for building this.
- rambambram 2mo ago[flagged]
- kocamaz 2mo agoIt's good looking, and I liked it. The trial page accessed from the hugging face website is a very inefficient experience when I use Mozilla and the dark theme, FYI.
- snyy 2mo agoFixed. Try again. Let me know if any other issues
- esafak 2mo agoWhy does the 'Quality vs Cost of Web Content Extraction' chart not have zero cost at the origin? Up to the right does not have to mean better; we can read.
- snyy 2mo agoFunnily enough, that wasn't my first choice either. I A/B tested it with a small group and people understood "up and to the right is better" faster.
- rishav2580 2mo ago[flagged]
- snyy 2mo agoThanks! Good questions: We haven't run a targeted eval against SEO spam yet. However, with Pulpie, each block gets labeled by what the text actually says rather than what the tags look like. Wrapping boilerplate in semantic tags fools rule based extractors precisely because they judge structure. Pulpie doesn't. The closest benchmark we have for this is the WebMainBench difficulty split, where pulpie-orange-small holds 0.813 on the hard subset. For comparison, trafilatura scores a 0.526. For quantization, we haven't benchmarked INT8 or FP8. Everything in the post ran on L4 and A100. That said, I expect it to go well for a few reasons. It's a single forward pass over the page, so the workload is compute bound rather than bandwidth bound, which is why the L4 held up so well against the A100 and why cheaper cards should degrade gracefully. At 210M params the small model is roughly 420MB in FP16 and half that in INT8. So it fits on any consumer GPU with room to spare. Also, one pass classification tends to survive 8 bit quantization better than autoregressive generation since there is no error accumulation across decode steps.
- andy_parhelia 2mo ago[flagged]
- andrethegiant 2mo agoWhy not use a plain old html → markdown converter? You can easily strip out ads using CSS /jQuery-like selectors. That would cost zero dollars.
- spelk 2mo agoIf I had to reckon, it's because the web comes in very many shapes, and outsourcing that work to a generalist LLM/SLM like GPT Nano is expensive, and doing it deterministically will never catch all the edge cases as well as a purpose-built encoder when run at webscale.
- dracyr 2mo agoLooks like they are including Trafilatura in the comparison tables, which I've used before with pretty decent results, but it still has trouble with some pages. Looks like the pulpie f1 scores are quite a bit better, especially for the hard cases. Would be curious how it runs on more modest hardware though, I'm using it for a small bookmark archiving tool and being able to run it on my small mini-pc homelab would be nice.
- snyy 2mo agoWe see far better performance with models. Heuristics break on richer content like codeblocks, formulae, quotes, etc. In our testing, our model was 25 F1 points better than Trafilatura.
- nullsanity 2mo ago[dead]
- andrethegiant 2mo agoI think instead of "performance" you must mean "accuracy". Traditional deterministic conversion will always be faster and cheaper than running through a model, even if it is less accurate.
- snyy 2mo ago
- zaptheimpaler 2mo agoSo this is tailored towards kind of a "reader view" for models right? Can it handle images, tables, shadow DOMs too? Like there are 3 use cases I have now - one is a simple text view for models to understand it, one is a "web clip" mode which would ideally preserve images and media, and one is to extract tabular data from web pages. Which ones is this good at?
- snyy 2mo agoImages pass through as they are considered main content. Same with tables. Pulpie will return all main content on a page as HTML/Markdown. I’m not sure I fully understand “which one this is good at?”. perhaps you can try the model on hugging face and let me know if the results look good? https://huggingface.co/spaces/feyninc/pulpie https://huggingface.co/spaces/feyninc/pulpie
- tyzoid 2mo agoHow does this work on pages that require JavaScript in order to render?
- philipkglass 2mo agoYou'd typically use a headless browser to generate the fully rendered page, then capture the rendered output for use with the model.
- snyy 2mo agoExactly this. Thank you for answering!
- cpill 2mo agoI did some research on this about 10 years ago. I spent 2 days hand labelling data from scraped news sites. Then built a good old fashioned Random Forest model to classify html nodes based on some feature engineering. turns out the P tag and the number-of-words threshold get you 90% of the way there, on news sites anyway. Great thing about RF models is they tell you which features are the most important. fun little project (apart from the 2 days of data labelling).
- rnagulapalle 2mo ago[flagged]
- geniium 2mo agoAmazing I was just looking for something like this to be able to import web page content into Whisperit
- wiradikusuma 2mo agoDoes it work with ecommerce for product scraping? E.g. Amazon, or Shopee (big in SEA)
- snyy 2mo agoYes. I tried it with https://www.allbirds.com/products/womens-cruiser-canvas https://www.allbirds.com/products/womens-cruiser-canvas on our HF space and Pulpie worked great. HF Space: https://huggingface.co/spaces/feyninc/pulpie https://huggingface.co/spaces/feyninc/pulpie
- danielmeskin 2mo agoWell allbirds is an AI company now so I guess that makes sense
- emblemapp 2mo agoThis looks really cool
- vishalkundar 2mo agoVery interesting
- keynha 2mo ago[dead]
- vladsiu 2mo ago[dead]
- grillermo 2mo agoI’m implementing this for readitsoon my web to kindle app. Thanks for this!
- ibrameme 2mo ago[dead]
- i18nagentai 2mo ago[flagged]