5 ms·

I set this up a few months ago based on asciimoo's comments on HN, and barely used it at first, but I realized not too long ago that it could be a pretty useful

by wmchen 25d ago

I set this up a few months ago based on asciimoo's comments on HN, and barely used it at first, but I realized not too long ago that it could be a pretty useful research tool for one of my hobbies (award travel), that revolves around being in the know around various concepts and quirks. I scraped and imported posts from the blogs I regularly reference for award travel, then hooked it up to OpenCode/Codex as an MCP server and used that corpus for research on those topics. So I can ask things like "has anyone ever mentioned running into this problem before?" [1] If you have a hobby or working situation that requires you to regularly reference a core set of websites or reference materials, Hister provides almost all the tools out of the box to start a search engine against it. The default datasets they promote include the Python Stlib, MDN and RFC corpus, as an example. [2] [1]: https://wmchen.com/blog/revisiting-hister/ https://wmchen.com/blog/revisiting-hister/ [2]: https://hister.org/datasets https://hister.org/datasets

asciimoo 25d ago
Wow, this is a really inspiring use case and blog post. Thanks for sharing it. What tools or features would Hister need to support your complete search workflow?
- wmchen 25d ago
  Thank you so much for building this! It's a really awesome piece of work and I'm grateful for your work. I'd love to sponsor you on Github in the near future. The only thing that I think would be interesting to see is native support for crawling via a sitemap.xml instead of recursively. I worked around this by implementing a basic scraper that fetched pages exclusively from the sitemap.xml to add into Hister. I think you're already aware of this, but I also experienced some data loss during the import because I was running a concurrent reindex. I clocked it pretty quickly so I didn't think too much of it. [1] [1] "TODO store new documents in both indexes while running reindex to guarantee not losing any data." @ https://github.com/asciimoo/hister/blob/master/server/indexer/indexer.go https://github.com/asciimoo/hister/blob/master/server/indexe...
Aboutplants 25d ago
Oh this gets my brain spinning! Thanks for the tips

by wmchen 25d ago

asciimoo 25d ago

Wow, this is a really inspiring use case and blog post. Thanks for sharing it. What tools or features would Hister need to support your complete search workflow?

wmchen 25d ago
Thank you so much for building this! It's a really awesome piece of work and I'm grateful for your work. I'd love to sponsor you on Github in the near future. The only thing that I think would be interesting to see is native support for crawling via a sitemap.xml instead of recursively. I worked around this by implementing a basic scraper that fetched pages exclusively from the sitemap.xml to add into Hister. I think you're already aware of this, but I also experienced some data loss during the import because I was running a concurrent reindex. I clocked it pretty quickly so I didn't think too much of it. [1] [1] "TODO store new documents in both indexes while running reindex to guarantee not losing any data." @ https://github.com/asciimoo/hister/blob/master/server/indexer/indexer.go https://github.com/asciimoo/hister/blob/master/server/indexe...

Aboutplants 25d ago

Oh this gets my brain spinning! Thanks for the tips