4 ms·

Have you made this open source? I really want something like this. Bonus points if it lets me add filters, etc. I want to read my local newspaper, but like h

by raddan 1mo ago

Have you made this open source? I really want something like this. Bonus points if it lets me add filters, etc. I want to read my local newspaper, but like half of the stories are about high school sports, of which I have absolutely zero interest.

neo1 1mo ago
It's called w3m (or lynx). No seriously. Not kidding. While I'm trying to be snarky, I recently discovered how well it works to do exactly what you are asking. Sites that need javascript, or are paranoid cloudflare prisons won't work, but otherwise it works quite well. For walls of text (long substack articles) I still prefer epub, and use https://codeberg.org/neo1/mkebook https://codeberg.org/neo1/mkebook which I wrote, but if you love the terminal: w3m. Filters are done by your rss reader, newsboat has boat loads of options, but any decent one should.
- theshrike79 1mo ago
  Defuddle (https://github.com/kepano/defuddle https://github.com/kepano/defuddle) by @kepano (CEO of Obsidian) is pretty damn good at getting a text-only version of random sites.
  - neo1 1mo ago
    Defuddle looks interesting, I will try it out. Thanks for pointing it out! There is another tool that on the surface appears similar to Defuddle called trafilatura: https://github.com/adbar/trafilatura https://github.com/adbar/trafilatura It was OK, but I wasn't happy with it, as it kept stripping the comments sections.
theshrike79 1mo ago
It's 110% vibe engineered, so I'm rather hesitant to share it to anyone else - I don't need the drama =) But it's pretty much a bespoke handler for every site with some common bits (caching, feed fetching etc) For example: the HN handler uses the Algolia API to grab "front page" articles, filters them by score and comment count, grabs the link and gets any opengraph data from the link. Then the feed item contains some context for the linked site + post score + comment count. This way I can make the decision whether to open it or hit next right in my RSS reader. I have similar handlers for lobsters, tildes, youtube etc. along with a "meh" attempt of a daily AI generated summary for noisy sites like Verge, Ars Technica etc. who publish a dozen or more articles per day. The system grabs all those, combines the ones about the same subject and summarises the content. But yea, it's open source. 99% of the stuff I build is public on my Github with a permissive license.

by raddan 1mo ago

neo1 1mo ago

It's called w3m (or lynx). No seriously. Not kidding. While I'm trying to be snarky, I recently discovered how well it works to do exactly what you are asking. Sites that need javascript, or are paranoid cloudflare prisons won't work, but otherwise it works quite well. For walls of text (long substack articles) I still prefer epub, and use https://codeberg.org/neo1/mkebook https://codeberg.org/neo1/mkebook which I wrote, but if you love the terminal: w3m. Filters are done by your rss reader, newsboat has boat loads of options, but any decent one should.

theshrike79 1mo ago
Defuddle (https://github.com/kepano/defuddle https://github.com/kepano/defuddle) by @kepano (CEO of Obsidian) is pretty damn good at getting a text-only version of random sites.
- neo1 1mo ago
  Defuddle looks interesting, I will try it out. Thanks for pointing it out! There is another tool that on the surface appears similar to Defuddle called trafilatura: https://github.com/adbar/trafilatura https://github.com/adbar/trafilatura It was OK, but I wasn't happy with it, as it kept stripping the comments sections.

theshrike79 1mo ago

It's 110% vibe engineered, so I'm rather hesitant to share it to anyone else - I don't need the drama =) But it's pretty much a bespoke handler for every site with some common bits (caching, feed fetching etc) For example: the HN handler uses the Algolia API to grab "front page" articles, filters them by score and comment count, grabs the link and gets any opengraph data from the link. Then the feed item contains some context for the linked site + post score + comment count. This way I can make the decision whether to open it or hit next right in my RSS reader. I have similar handlers for lobsters, tildes, youtube etc. along with a "meh" attempt of a daily AI generated summary for noisy sites like Verge, Ars Technica etc. who publish a dozen or more articles per day. The system grabs all those, combines the ones about the same subject and summarises the content. But yea, it's open source. 99% of the stuff I build is public on my Github with a permissive license.