11 ms·

Hister – A private, full content search index that you control

497 points by auraham 1mo ago

lnenad 26d ago
I LOVE the concept. I will play around with the execution, if it works as described this is a great product.
CodeWithLeo 25d ago
[flagged]
asciimoo 26d ago
Ohi, author here! Thanks for posting Hister. Feel free to A.M.A. My first free software search project was Searx, a privacy respecting metasearch engine, but because of the limitations of the metasearch concept, I've decided to take a different approach. Hister builds a personal search index from pages you visit, bookmarks, browser history, local files, and crawled websites. It stores extracted content with offline result previews, so information remains searchable even when the original page changes or disappears. It supports full text and semantic search, can run entirely on your own machine, and includes a web interface, command line tools, and an MCP endpoint for assistant integrations. Project page: https://github.com/asciimoo/hister https://github.com/asciimoo/hister Tiny read-only demo: https://demo.hister.org/ https://demo.hister.org/
- zuzululu 26d ago
  interesting you used AGPL 3 licensing, are you planning a hosted version? would've been great with a more liberal license
  - asciimoo 26d ago
    > are you planning a hosted version Not in the near term. Right now I am focused on developing Hister rather than operating a hosted service. There are already plenty of centralized hosted search engines, so my longer term interest is in federation and distributed search. I want to make the core system mature first. > would've been great with a more liberal license It depends on how do you define liberal. =] I chose AGPLv3+ because I want Hister to remain free software and available to their users.
    - zargon 26d ago
      AGPLv3 is awesome, great choice!
    - zuzululu 26d ago
      it prevents certain innovations to be derived from it but with LLMs I do think these licensing are pretty much moot. i still think it could've benefited by Apache 2.0 which more or less gets you to your goals
    - stavros 25d ago
      I agree with AGPLv3, I don't really understand why people have an objection. All it prevents is someone taking your work and making a business out of it.
      - zuzululu 25d ago
        with AGPL if you modify any part of it you have to open source the entire thing and you may not want to do that or your company prevents it. In fact many companies avoid AGPLv3 licensing for this reason. > All it prevents is someone taking your work and making a business out of it. Not really. I say licensing is moot now because of LLMs.
  - polymer8563 26d ago
    it would not! asciimoo, thanks for using AGPL, I've been using hister for months now and it helped me a lot
  - nairboon 25d ago
    Can you give an example of a reasonable thing that you want to do with the Hister source code that the AGPL currently prevents you to do?
- dented42 26d ago
  What’s the safari story look like right now?
  - asciimoo 26d ago
    Safari support is currently stalled in this PR: https://github.com/asciimoo/hister/pull/46 https://github.com/asciimoo/hister/pull/46. I do not have access to macOS, so help with development and testing would be very welcome.
- pbronez 26d ago
  I've been using LinkDing + SingleFile for this. Nice to have another option!
  - asciimoo 26d ago
    That is a great combo. My main friction with it is having to manually capture pages I want to keep.
- Groxx 26d ago
  >Hister builds a personal search index from pages you visit, bookmarks, browser history, local files, and crawled websites. Immediately interested and will check it out, thank you! I've wanted a "search stuff you've seen online" tool for a long time, but everything seems to be research-oriented or "archive but don't search" or some weird combination that means it's nigh useless to me. I've got decades of bookmarks and archives and I've kinda been stuck grepping them at best (it's rare but I do sometimes want a page I saw once three years ago and I love having that option), while hoping someone would build something better. One question if ya don't mind, while I explore: any chance of singlefile support? Content-extraction is useful in lots of situations (e.g. wallabag) and it's a great default, but sometimes it fails and sometimes you really do want the page, relatively close to how it actually was. Singlefile does that much better than most, and it does so well enough (and manually-handle-able enough if needed) that I don't feel any desire to switch to WARCs or similar. Though specifically I'm probably looking for something like "content-extract everything" + "key combo to save singlefile version too" + "upload singlefile archives to backfill / recover". Like 99% of the time content extraction is preferred, and I'm glad to see it... it's just not always enough, and having to go elsewhere for exceptions breaks a lot of the utility.
  - asciimoo 26d ago
    Exactly! I had the very same issues before Hister. > One question if ya don't mind, while I explore: any chance of singlefile support? Yes, partially. Hister can already import HTML files created by SingleFile, but there is no direct integration yet. In the longer term, I would like the SingleFile extension to be able to send snapshots directly to Hister.
    - Groxx 26d ago
      I assume it content-extracts that on upload? I'd really like to move storage into hister too, if possible. That way you could also switch from an extracted view to a "full" view in the UI. Though I assume that'd be fairly simple to build later. Overall I really like what I'm seeing, it ticks a lot of important boxes for me and it's pleasantly straightforward. Hopefully I'll find time to contribute!
- oulipo 26d ago
  You should integrate with Karakeep.app, it's only natural that you'd want both a search engine, and a nice "archive" and "article pretty view" features :)
  - asciimoo 26d ago
    What do you mean by integration? Hister already supports importing data from Karakeep, Linkding, Linkwarden, Readeck, Shaarli, and wallabag: https://hister.org/docs/import https://hister.org/docs/import
- Jon_m 26d ago
  [dead]
- iambateman 26d ago
  Very cool. Does it work to connect my phone and laptop to the same firehose? I often find myself irritated because I read an article on my phone 6 months ago and the history is gone.
  - asciimoo 26d ago
    You can host Hister on a home server and access it from multiple devices. Automatic page capture on mobile currently requires Firefox, since mobile Chrome does not support browser extensions.
    - alchemist1e9 25d ago
      Orion browser hopefully can run the Firefox extension on iOS. Then joining the mobile device to tailscale network where Hister is hosted on a server on a host in the network … probably this will work at least that is my plan to try. Thank you for an amazing project. I’m a heavy msgvault user and adding Hister to my browsing will be hugely complementary.
    - porridgeraisin 25d ago
      Can you support kiwi browser mobile? It has extension support and is chromium based. Happy to contribute the feature if needed
- mfld 26d ago
  What would be a typical size of the search index, let's say after 5 years of intense browsing?
  - asciimoo 26d ago
    It depends on what you consider intense browsing. An indexed document uses about 100KB on average because Hister stores the full original HTML for offline previews. If storage is a concern, you can disable offline previews, reducing the average document size significantly.
- mirshko 26d ago
  Do you plan on adding filterable tags to the ui? Rather than just metadata.
  - stavros 25d ago
    If I understand what you mean correctly, https://historio.us https://historio.us does that.
- vzaliva 26d ago
  I was thinking about making a "personal data search engine". I have almost 30GB of email archives and a bunch of Google Drive and offline files. I am thinking about downloading it all from the cloud and storing it on a large RAID array, with an indexing and search interface (and possibly a local MCP server). Would Hister be suitable for this? Can it index mbox files? Would it handle this amount of data? Does it have a search API so I can build an MCP server?
  - asciimoo 26d ago
    > Would Hister be suitable for this? The indexer it uses (Bleve) can handle millions of records according to their docs, but sure that Hister would be the best choice for this task. I'd probably use Meilisearch (https://www.meilisearch.com/ https://www.meilisearch.com/). > Can it index mbox files? Not yet. > Does it have a search API so I can build an MCP server? It has both search API and MCP server endpoints.
    - asciimoo 25d ago
      EDIT: I wanted to write "*not* sure" in the previous message.
  - alchemist1e9 24d ago
    Take a look at msgvault. It might handle your email archives very well.
- moostee 26d ago
  I'd love to be able to use a cloud storage backend (eg. Google Drive) so that my laptop and phone drive space isn't at risk and backups are automatic.
- MasterYoda 26d ago
  This sounds very interesting. Wanted something similar, so going to check it out:) One thing at first look, like a little feature request ;). Personally I think it would be very nice if it was possible to index different sites to different search indexs. So you could separate different stuff, like job, different projects you have, and other stuff and then last everything else etc to there on search index db. So it would be possible to have like "profiles" you could easy choose from. So in the addon, when you click on the hister icon on the toolbar there would be a list of profiles that you could choose where to index the site to. But also a setting to choose which domains and sites that always should index to one search index profils. And all sites that dont is added to a profile is added to the default index instead. For me this would make it much more easy to find stuff and sort out what Im looking for (for me things get very "thing"/project based what I want to find), so having this.. 1 - The "profiles" would become like an important filter. When you search you could easy mark one or more "profiles" which you would search from, and you remove alot of unwanted data automatically, specially when you index every site you visit by automatically. 2 - It would also make it easy if the db gets big over time to remove data that is less important later on and that you dont need to be index anymore. 3 - It would also make it easier to backup only the most important search index DB:s if the DB for the differnt search index:s where in different folders/files. I guess the "default index" that index all sites could easy get big, but there it would be alot of not important data. So would be nice to be able to easy just use a backup program and only backup the only important search databases, to save space. Thanks
  - asciimoo 25d ago
    You can create multiple users for different use-cases. Also, it is possible to automatically assign labels to documents coming from different sources and use these labels to filter results. E.g. using multiple browser profiles where different profiles apply different labels to submitted documents.
- javier2 26d ago
  I think its kinda cool, but I dont really get it? So I can gather and search my browser history + most websites i have visited?
- hankbond 25d ago
  I'm glad this came up again because the last time is was posted I never got around to installing it on my Unraid. Just did!
- poopiokaka 25d ago
  [dead]
- p-e-w 25d ago
  I used to be interested in the prophecies of Nostradamus a lifetime ago. You may be surprised to learn that “Hister” is a name mentioned in those prophecies that is commonly claimed by believers to refer to the future Adolf Hitler. Academics usually take it to mean the Danube river instead, but in conspiracy/New Age contexts, the association with Hitler is prevalent, and mentioned in several pop culture works, including at least one feature film.
  - mickdarling 25d ago
    Literally the first thing I thought of. Is he trying to reference an Antichrist?
    - p-e-w 25d ago
      I’m pretty sure it’s just an unintentional name collision. The vast majority of people are unaware of this name’s connotation in Nostradamus, unless they happen to have read or watched some pop culture work that references it.
  - asciimoo 25d ago
    The name is coming from "HISTory on STERoids". When I checked the name, I found only a beetle and part of the Danube called "Hister".
- human_llm 25d ago
  Thanks you for making Hister. I have been using it since May and find it immensely useful. It's a piece of mind to know that I always will be able to easily find what I looked at. So I don't feel the need to bookmark things, which I used to do, but then never looked at anyways ;).
- idontneedcoffee 25d ago
  Thanks for your contribution to the Universe! :) I'm hacking on a in-process DB on top of LMDB+Lance(for now, hilbert space kung-fu with a custom matryoshka embedding setup with separate spatial + temporal + internal and content derived anchors will replace that ~last-century~ last-year tech) + roaring bitmaps as the primary indexing engine with the same or similar goal[0] and will definitely deep-dive into yours. I'm also trying to index users unstructured documents and workflows(tabs, emails, files, notes, identities etc) - Organize them into semantically meaningful user or agent created context or directory-like virtual trees (the same photo of a nice kitchen may be surfaced under `/travel/barcelona` and `/arch/interieour/kitches`) - ..where tree nodes are mapped to bitmaps - `/travel/barcelona` does a fast and cheap `travel` AND `barcelona`, want to "zoom-out" you just go one directory up to `/travel` and see all documents tagged with travel) - You can use multiple timelines - extract that fancy md-converted en-wiki hf dataset into a wikipedia db dataset + timeline, tag your personal timeline as "personal" - wanna know the zeitgeist of your grandmothers birth date - search for it with timelines personal + wikipedia in layered mode and you'll get everything that happened or was happening during that time. - You can have long-running stateful query sessions and refine your searches dynamically - search for "winter" and get all documents with a winter scenery or mentioning winter - refine with "nice view" then "laptop" - citing a recent example[1] - Documents have relations that would be cumbersome to map in a virtual tree structure(worth an experiment due to the zoom-in/out you get with context bitmap trees though) - hence on top of the initial structure you can use graph edges(also powered by bitmaps - as most indexes are) - All vector queries always run on top of a candidate set you get by the bitmap/bitmap-based filter algebra hence searching through 100k+ docs is usually pretty fast Anyhow, let me stop here, thank you once again! [0] https://github.com/canvas-ui/canvas-synapsd https://github.com/canvas-ui/canvas-synapsd (sorry for the sloppy ai readme, no time to resurrect my old one with the updated APIs) [1] https://demo.cnvs.ai/pub/c/aks6zaf8 https://demo.cnvs.ai/pub/c/aks6zaf8
- cure_42 25d ago
  Is the extracted content stored in a way that could be easily converted to markdown? Seems like this could be integrated incredibly well with obsidian as a way to automatically expand your knowledge base, as well as a better search for it.
  - asciimoo 25d ago
    The extracted content is stored and displayed as HTML, so converting it to markdown doesn't guarantee lossless transformation. But, the other direction works well: Hister can live track and import markdown files providing full text search and rendered previews for all your files.
- veemjeem 23d ago
  I'm running hister locally, I see in the MCP docs that a third party can query the server, but is there a way to import data as well? I find that I often do research using pi, and would love to save the context to hister to query later on. The closest thing I see is the "hister import file" command, which is great, but inconvenient. If MCP allowed third parties to dump text data in, that would be ideal.
Carrok 26d ago
I tried this out this week and liked it but really wish this project had some form of auth. Opening the contents of every page you’ve ever visited, even to the local network, is not the best idea.
- asciimoo 26d ago
  Hister supports token based, password based, and OIDC/OAuth authentications with optional multi-user handling. Details about user handling can be found here: https://hister.org/docs/user-handling https://hister.org/docs/user-handling It also has a "public mode" where anyone can search the indexed content, but only authenticated users can add or modify it.
  - Carrok 26d ago
    It seemed like the public mode was the default when I set it up. If so, that’s a fairly dangerous default as keeping a “clean” history with no secrets leaked seems neigh impossible.
    - asciimoo 26d ago
      The default configuration binds only to localhost, and a fresh installation starts with an empty database/index. Could you clarify which specific attack surface you are concerned about in that scenario?
      - Carrok 26d ago
        I’m not concerned about an attack scenario. I’m just saying that using the docker image, if someone (or their agent) isn’t careful, they could expose their browsing history publicly fairly easily. It might just be nice to default to at least a user and pass login rather than just wide open.
- androidu 26d ago
  I've set it up today with OIDC AuthN (Keycloak), removed the username/password fields & kept only OIDC, it's working flawless. Also indexed data is persisted on a per-user basis, so you got this isolation and certainty that your searches will not be polluted by your family's
shevy-java 26d ago
This is like Hipster. But missing a 'p'.
robotmay 26d ago
Ooh I’ve been thinking about this idea for years, I’m glad someone beat me to it. Guess I have something to play with over the rest of the weekend!
renegat0x0 26d ago
I have created something similar, yet different. I maintain my own Internet domain index. https://github.com/rumca-js/Internet-Places-Database https://github.com/rumca-js/Internet-Places-Database Also I maintain android app that can be used to search places. https://f-droid.org/en/packages/io.github.rumcajs.offlinewebsearch https://f-droid.org/en/packages/io.github.rumcajs.offlineweb... I believe hister you have to fill in with your data, right?
wmchen 26d ago
I set this up a few months ago based on asciimoo's comments on HN, and barely used it at first, but I realized not too long ago that it could be a pretty useful research tool for one of my hobbies (award travel), that revolves around being in the know around various concepts and quirks. I scraped and imported posts from the blogs I regularly reference for award travel, then hooked it up to OpenCode/Codex as an MCP server and used that corpus for research on those topics. So I can ask things like "has anyone ever mentioned running into this problem before?" [1] If you have a hobby or working situation that requires you to regularly reference a core set of websites or reference materials, Hister provides almost all the tools out of the box to start a search engine against it. The default datasets they promote include the Python Stlib, MDN and RFC corpus, as an example. [2] [1]: https://wmchen.com/blog/revisiting-hister/ https://wmchen.com/blog/revisiting-hister/ [2]: https://hister.org/datasets https://hister.org/datasets
- asciimoo 26d ago
  Wow, this is a really inspiring use case and blog post. Thanks for sharing it. What tools or features would Hister need to support your complete search workflow?
  - wmchen 26d ago
    Thank you so much for building this! It's a really awesome piece of work and I'm grateful for your work. I'd love to sponsor you on Github in the near future. The only thing that I think would be interesting to see is native support for crawling via a sitemap.xml instead of recursively. I worked around this by implementing a basic scraper that fetched pages exclusively from the sitemap.xml to add into Hister. I think you're already aware of this, but I also experienced some data loss during the import because I was running a concurrent reindex. I clocked it pretty quickly so I didn't think too much of it. [1] [1] "TODO store new documents in both indexes while running reindex to guarantee not losing any data." @ https://github.com/asciimoo/hister/blob/master/server/indexer/indexer.go https://github.com/asciimoo/hister/blob/master/server/indexe...
- Aboutplants 26d ago
  Oh this gets my brain spinning! Thanks for the tips
thataccount 26d ago
[flagged]
- wolfy1993 26d ago
  I'll be honest I wouldn't blame OP for not finding this (I found a species of hister beetle before anything to do with Hitler) I feel this is a weird spin on the "Clicks to Hitler" game
  - thataccount 26d ago
    I was not using Wikipedia to get there. This is something I immediately recognized from research about World War II and possible prophetic fulfillment. I definitely wasn't playing a game when I came across this information either, but I get your drift.
    - ButlerianJihad 26d ago
      Interestingly, when I searched "hister" on English Wikipedia, I was sent to https://en.wikipedia.org/wiki/Danube#Names_and_etymology https://en.wikipedia.org/wiki/Danube#Names_and_etymology Where it says that “Hister” is a variant of its Latin name. I think the name is lovely with no reason to change. Surely there are few people who would connect this with silly medieval superstition. Perhaps you can take part in wiping out that association.
      - thataccount 26d ago
        [flagged]
  - hardgeus 25d ago
    I've been on site site for years. I created an account just to comment on this thread. Holy crap, the only time I've ever heard 'Hister' was in a book I read on Nostradamus like 30 years ago.
    - 25d ago
exe34 26d ago
Can it import browser bookmarks? It says browser history and a bunch of other bookmark services but not specifically browser bookmarks.
- asciimoo 26d ago
  Yes, it can import data from various sources, including browser history, and popular self-hosted bookmarking apps: https://hister.org/docs/import https://hister.org/docs/import
  - exe34 26d ago
    Is this reply from Gemini? Because I'm used to gemini gaslighting me. The answer is the opposite of my question.
    - asciimoo 26d ago
      Doh, sorry, the answer was coming from me who did not read the question properly. It currently cannot import browser bookmarks, but it is a good idea. Added to my TODO.
      - exe34 26d ago
        Ah sorry didn't mean to be mean, I just wondered if you had automated it haha!
      - exe34 24d ago
        Figured I can just export the bookmarks and then hister index the urls.
alexnewman 26d ago
Time to shameless plug my own easy to host your own content search index Webtm.io All open source and small enough to deploy. I deploy to cf webworkers so it’s the only place it’s tested. One cool thing is we work on iOS, chrome and friends, Firefox and pretty much everywhere. We do require you bring your own LLM though.
quasigod 26d ago
I just finished setting this up yesterday and I'm kind of obsessed with it. I was previously a heavy user of Karakeep, but I hated forgetting to save something and losing it. I also think Hister's semantic search is a better solution than AI generated summaries and tags. Support for local docs is super cool too, I have it set up to index my org notes directory.
ebr4him 26d ago
Is the design vibe coded in Codex by any chance? Seeing a lot of similar designs out there and it’s really starting to annoy me.
- hallole 26d ago
  Really? The website, you mean? :( I thought to myself, "what a cute, well designed webpage!" when I saw it. Maybe with time this site will stand out like those Claudeslop ones.
- lucaaa 26d ago
  same
- asciimoo 26d ago
  The whole design is created by a main contributor and IIRC the style is called neobrutalism.
- tolerance 25d ago
  I think that the tells, if they are in fact legitimate, are negligible at best or subtle at worst. It looks like the site uses Tailwind, so if AI was involved it wasn't generating the design from scratch. If it did then I'm impressed because usually it sucks. This is alright.
cowmanOG 26d ago
Would be really cool to have Zotero library integration / import support!
knowaveragejoe 26d ago
I love the idea overall, but something about browser extensions give me a bad taste. Am I overthinking it? I realize the extension is open source, of course.
kinibha 26d ago
Any way to add for mobile?
- quasigod 25d ago
  If you're on android the extension works for Firefox
jbellis 25d ago
Love to see this. I started something similar a couple years ago but didn't quite have the patience to make the browser extensions really bulletproof. Planning to contribute significant improvements to the vector search side here.
mmargenot 25d ago
This is so cool! For the time being I’m still locked into notion for my handwritten knowledge base, but I love this for incorporating external information. For the semantic search is there any chunking/processing that happens with the content or do you need to be diligent about having a large embedding context (and/or small content)?
triilman 25d ago
can it search .pdf content as easy as searching content in txt file?
0n0n0m0uz 25d ago
Cant wait to check this out
vivzkestrel 25d ago
- lots of stupid questions to the author from a guy who has no idea about search engines - let us say I want to index every blog ever listed on HN - should be a small subset of the 400 billion pages out there on the internet no? - First I need to gather data, what do you use to load so many webpages rapidly? asyncio with aiohttp in python? are there better options? - how do you handle proxies? rotation? are there libraries you recommend for this? - what about pages that use cloudflare? or block your request or present a captcha or a challenge of some kind? - what are the filetypes you collect? only html or media as well? - where and in what format do you store all these collected files? flat file storage? duckdb? postgres? hstore? something else? - what is the frequency at which you refresh each page? once a day? once a week? something else? - what kind of pre-processing do you use on the collected data? remove extra spaces? special characters? some kind of complex regex pipeline? LLM? - how do you match the incoming query with processed data? simple text matching? regex? vector embedding match? something else?
- 9935c101ab17a66 22d ago
  A lot of these can be answered by checking the documentation: https://hister.org https://hister.org. Others could be answered by examining the source code, as it is open source. It's good etiquette to check existing documentation and resources before badgering OSS project teams with a lengthy list of questions.
vladshiyan 25d ago
[flagged]
stayux 25d ago
Thanks, this will pair wonderfully with my Capcat.org project.
ndom91 25d ago
What's the disk space usage like? i.e. average per-day/week additional storage in your usage?
uygar 25d ago
[dead]
dmd 25d ago
It’s always been bizarre to me how bad browsers themselves are at searching their own history. This looks great.
phendrenad2 25d ago
Ister (Latin Hister, Ancient Greek Ἴστρος / Istros) is the ancient classical name used by the Greeks and Romans for the Danube River, specifically referring to its lower course.
evanjrowley 25d ago
While I love Karakeep, the truth is that I'm using it to solve this exact same problem, even though its project goals are not as well aligned. I would seriously consider giving Hister a try because it seems like a better fit. Bonus points for being written in Go. I suppose the one thing I need would be an easy way to share links into Hister. Hopefully we will see both Android/iOS mobile apps to send links into Hister and better support for Safari. I wonder what could be possible with the extensions support available on mobile browsers too (Safari, MS Edge, Firefox).
- bpavuk 25d ago
  the Hister extension is available on Firefox for Android :) https://addons.mozilla.org/en-US/firefox/addon/hister/ https://addons.mozilla.org/en-US/firefox/addon/hister/
retrochameleon 25d ago
Any chance I could integrate or import data from Readwise Reader? (Link archival tool)

497 points by auraham 1mo ago

lnenad 26d ago

I LOVE the concept. I will play around with the execution, if it works as described this is a great product.

CodeWithLeo 25d ago

[flagged]

asciimoo 26d ago

Ohi, author here! Thanks for posting Hister. Feel free to A.M.A. My first free software search project was Searx, a privacy respecting metasearch engine, but because of the limitations of the metasearch concept, I've decided to take a different approach. Hister builds a personal search index from pages you visit, bookmarks, browser history, local files, and crawled websites. It stores extracted content with offline result previews, so information remains searchable even when the original page changes or disappears. It supports full text and semantic search, can run entirely on your own machine, and includes a web interface, command line tools, and an MCP endpoint for assistant integrations. Project page: https://github.com/asciimoo/hister https://github.com/asciimoo/hister Tiny read-only demo: https://demo.hister.org/ https://demo.hister.org/

zuzululu 26d ago
interesting you used AGPL 3 licensing, are you planning a hosted version? would've been great with a more liberal license
- asciimoo 26d ago
  > are you planning a hosted version Not in the near term. Right now I am focused on developing Hister rather than operating a hosted service. There are already plenty of centralized hosted search engines, so my longer term interest is in federation and distributed search. I want to make the core system mature first. > would've been great with a more liberal license It depends on how do you define liberal. =] I chose AGPLv3+ because I want Hister to remain free software and available to their users.
  - zargon 26d ago
    AGPLv3 is awesome, great choice!
  - zuzululu 26d ago
    it prevents certain innovations to be derived from it but with LLMs I do think these licensing are pretty much moot. i still think it could've benefited by Apache 2.0 which more or less gets you to your goals
  - stavros 25d ago
    I agree with AGPLv3, I don't really understand why people have an objection. All it prevents is someone taking your work and making a business out of it.
    - zuzululu 25d ago
      with AGPL if you modify any part of it you have to open source the entire thing and you may not want to do that or your company prevents it. In fact many companies avoid AGPLv3 licensing for this reason. > All it prevents is someone taking your work and making a business out of it. Not really. I say licensing is moot now because of LLMs.
- polymer8563 26d ago
  it would not! asciimoo, thanks for using AGPL, I've been using hister for months now and it helped me a lot
- nairboon 25d ago
  Can you give an example of a reasonable thing that you want to do with the Hister source code that the AGPL currently prevents you to do?
dented42 26d ago
What’s the safari story look like right now?
- asciimoo 26d ago
  Safari support is currently stalled in this PR: https://github.com/asciimoo/hister/pull/46 https://github.com/asciimoo/hister/pull/46. I do not have access to macOS, so help with development and testing would be very welcome.
pbronez 26d ago
I've been using LinkDing + SingleFile for this. Nice to have another option!
- asciimoo 26d ago
  That is a great combo. My main friction with it is having to manually capture pages I want to keep.
Groxx 26d ago
>Hister builds a personal search index from pages you visit, bookmarks, browser history, local files, and crawled websites. Immediately interested and will check it out, thank you! I've wanted a "search stuff you've seen online" tool for a long time, but everything seems to be research-oriented or "archive but don't search" or some weird combination that means it's nigh useless to me. I've got decades of bookmarks and archives and I've kinda been stuck grepping them at best (it's rare but I do sometimes want a page I saw once three years ago and I love having that option), while hoping someone would build something better. One question if ya don't mind, while I explore: any chance of singlefile support? Content-extraction is useful in lots of situations (e.g. wallabag) and it's a great default, but sometimes it fails and sometimes you really do want the page, relatively close to how it actually was. Singlefile does that much better than most, and it does so well enough (and manually-handle-able enough if needed) that I don't feel any desire to switch to WARCs or similar. Though specifically I'm probably looking for something like "content-extract everything" + "key combo to save singlefile version too" + "upload singlefile archives to backfill / recover". Like 99% of the time content extraction is preferred, and I'm glad to see it... it's just not always enough, and having to go elsewhere for exceptions breaks a lot of the utility.
- asciimoo 26d ago
  Exactly! I had the very same issues before Hister. > One question if ya don't mind, while I explore: any chance of singlefile support? Yes, partially. Hister can already import HTML files created by SingleFile, but there is no direct integration yet. In the longer term, I would like the SingleFile extension to be able to send snapshots directly to Hister.
  - Groxx 26d ago
    I assume it content-extracts that on upload? I'd really like to move storage into hister too, if possible. That way you could also switch from an extracted view to a "full" view in the UI. Though I assume that'd be fairly simple to build later. Overall I really like what I'm seeing, it ticks a lot of important boxes for me and it's pleasantly straightforward. Hopefully I'll find time to contribute!
oulipo 26d ago
You should integrate with Karakeep.app, it's only natural that you'd want both a search engine, and a nice "archive" and "article pretty view" features :)
- asciimoo 26d ago
  What do you mean by integration? Hister already supports importing data from Karakeep, Linkding, Linkwarden, Readeck, Shaarli, and wallabag: https://hister.org/docs/import https://hister.org/docs/import
Jon_m 26d ago
[dead]
iambateman 26d ago
Very cool. Does it work to connect my phone and laptop to the same firehose? I often find myself irritated because I read an article on my phone 6 months ago and the history is gone.
- asciimoo 26d ago
  You can host Hister on a home server and access it from multiple devices. Automatic page capture on mobile currently requires Firefox, since mobile Chrome does not support browser extensions.
  - alchemist1e9 25d ago
    Orion browser hopefully can run the Firefox extension on iOS. Then joining the mobile device to tailscale network where Hister is hosted on a server on a host in the network … probably this will work at least that is my plan to try. Thank you for an amazing project. I’m a heavy msgvault user and adding Hister to my browsing will be hugely complementary.
  - porridgeraisin 25d ago
    Can you support kiwi browser mobile? It has extension support and is chromium based. Happy to contribute the feature if needed
mfld 26d ago
What would be a typical size of the search index, let's say after 5 years of intense browsing?
- asciimoo 26d ago
  It depends on what you consider intense browsing. An indexed document uses about 100KB on average because Hister stores the full original HTML for offline previews. If storage is a concern, you can disable offline previews, reducing the average document size significantly.
mirshko 26d ago
Do you plan on adding filterable tags to the ui? Rather than just metadata.
- stavros 25d ago
  If I understand what you mean correctly, https://historio.us https://historio.us does that.
vzaliva 26d ago
I was thinking about making a "personal data search engine". I have almost 30GB of email archives and a bunch of Google Drive and offline files. I am thinking about downloading it all from the cloud and storing it on a large RAID array, with an indexing and search interface (and possibly a local MCP server). Would Hister be suitable for this? Can it index mbox files? Would it handle this amount of data? Does it have a search API so I can build an MCP server?
- asciimoo 26d ago
  > Would Hister be suitable for this? The indexer it uses (Bleve) can handle millions of records according to their docs, but sure that Hister would be the best choice for this task. I'd probably use Meilisearch (https://www.meilisearch.com/ https://www.meilisearch.com/). > Can it index mbox files? Not yet. > Does it have a search API so I can build an MCP server? It has both search API and MCP server endpoints.
  - asciimoo 25d ago
    EDIT: I wanted to write "*not* sure" in the previous message.
- alchemist1e9 24d ago
  Take a look at msgvault. It might handle your email archives very well.
moostee 26d ago
I'd love to be able to use a cloud storage backend (eg. Google Drive) so that my laptop and phone drive space isn't at risk and backups are automatic.
MasterYoda 26d ago
This sounds very interesting. Wanted something similar, so going to check it out:) One thing at first look, like a little feature request ;). Personally I think it would be very nice if it was possible to index different sites to different search indexs. So you could separate different stuff, like job, different projects you have, and other stuff and then last everything else etc to there on search index db. So it would be possible to have like "profiles" you could easy choose from. So in the addon, when you click on the hister icon on the toolbar there would be a list of profiles that you could choose where to index the site to. But also a setting to choose which domains and sites that always should index to one search index profils. And all sites that dont is added to a profile is added to the default index instead. For me this would make it much more easy to find stuff and sort out what Im looking for (for me things get very "thing"/project based what I want to find), so having this.. 1 - The "profiles" would become like an important filter. When you search you could easy mark one or more "profiles" which you would search from, and you remove alot of unwanted data automatically, specially when you index every site you visit by automatically. 2 - It would also make it easy if the db gets big over time to remove data that is less important later on and that you dont need to be index anymore. 3 - It would also make it easier to backup only the most important search index DB:s if the DB for the differnt search index:s where in different folders/files. I guess the "default index" that index all sites could easy get big, but there it would be alot of not important data. So would be nice to be able to easy just use a backup program and only backup the only important search databases, to save space. Thanks
- asciimoo 25d ago
  You can create multiple users for different use-cases. Also, it is possible to automatically assign labels to documents coming from different sources and use these labels to filter results. E.g. using multiple browser profiles where different profiles apply different labels to submitted documents.
javier2 26d ago
I think its kinda cool, but I dont really get it? So I can gather and search my browser history + most websites i have visited?
hankbond 25d ago
I'm glad this came up again because the last time is was posted I never got around to installing it on my Unraid. Just did!
poopiokaka 25d ago
[dead]
p-e-w 25d ago
I used to be interested in the prophecies of Nostradamus a lifetime ago. You may be surprised to learn that “Hister” is a name mentioned in those prophecies that is commonly claimed by believers to refer to the future Adolf Hitler. Academics usually take it to mean the Danube river instead, but in conspiracy/New Age contexts, the association with Hitler is prevalent, and mentioned in several pop culture works, including at least one feature film.
- mickdarling 25d ago
  Literally the first thing I thought of. Is he trying to reference an Antichrist?
  - p-e-w 25d ago
    I’m pretty sure it’s just an unintentional name collision. The vast majority of people are unaware of this name’s connotation in Nostradamus, unless they happen to have read or watched some pop culture work that references it.
- asciimoo 25d ago
  The name is coming from "HISTory on STERoids". When I checked the name, I found only a beetle and part of the Danube called "Hister".
human_llm 25d ago
Thanks you for making Hister. I have been using it since May and find it immensely useful. It's a piece of mind to know that I always will be able to easily find what I looked at. So I don't feel the need to bookmark things, which I used to do, but then never looked at anyways ;).
idontneedcoffee 25d ago
Thanks for your contribution to the Universe! :) I'm hacking on a in-process DB on top of LMDB+Lance(for now, hilbert space kung-fu with a custom matryoshka embedding setup with separate spatial + temporal + internal and content derived anchors will replace that ~last-century~ last-year tech) + roaring bitmaps as the primary indexing engine with the same or similar goal[0] and will definitely deep-dive into yours. I'm also trying to index users unstructured documents and workflows(tabs, emails, files, notes, identities etc) - Organize them into semantically meaningful user or agent created context or directory-like virtual trees (the same photo of a nice kitchen may be surfaced under `/travel/barcelona` and `/arch/interieour/kitches`) - ..where tree nodes are mapped to bitmaps - `/travel/barcelona` does a fast and cheap `travel` AND `barcelona`, want to "zoom-out" you just go one directory up to `/travel` and see all documents tagged with travel) - You can use multiple timelines - extract that fancy md-converted en-wiki hf dataset into a wikipedia db dataset + timeline, tag your personal timeline as "personal" - wanna know the zeitgeist of your grandmothers birth date - search for it with timelines personal + wikipedia in layered mode and you'll get everything that happened or was happening during that time. - You can have long-running stateful query sessions and refine your searches dynamically - search for "winter" and get all documents with a winter scenery or mentioning winter - refine with "nice view" then "laptop" - citing a recent example[1] - Documents have relations that would be cumbersome to map in a virtual tree structure(worth an experiment due to the zoom-in/out you get with context bitmap trees though) - hence on top of the initial structure you can use graph edges(also powered by bitmaps - as most indexes are) - All vector queries always run on top of a candidate set you get by the bitmap/bitmap-based filter algebra hence searching through 100k+ docs is usually pretty fast Anyhow, let me stop here, thank you once again! [0] https://github.com/canvas-ui/canvas-synapsd https://github.com/canvas-ui/canvas-synapsd (sorry for the sloppy ai readme, no time to resurrect my old one with the updated APIs) [1] https://demo.cnvs.ai/pub/c/aks6zaf8 https://demo.cnvs.ai/pub/c/aks6zaf8
cure_42 25d ago
Is the extracted content stored in a way that could be easily converted to markdown? Seems like this could be integrated incredibly well with obsidian as a way to automatically expand your knowledge base, as well as a better search for it.
- asciimoo 25d ago
  The extracted content is stored and displayed as HTML, so converting it to markdown doesn't guarantee lossless transformation. But, the other direction works well: Hister can live track and import markdown files providing full text search and rendered previews for all your files.
veemjeem 23d ago
I'm running hister locally, I see in the MCP docs that a third party can query the server, but is there a way to import data as well? I find that I often do research using pi, and would love to save the context to hister to query later on. The closest thing I see is the "hister import file" command, which is great, but inconvenient. If MCP allowed third parties to dump text data in, that would be ideal.

Carrok 26d ago

I tried this out this week and liked it but really wish this project had some form of auth. Opening the contents of every page you’ve ever visited, even to the local network, is not the best idea.

asciimoo 26d ago
Hister supports token based, password based, and OIDC/OAuth authentications with optional multi-user handling. Details about user handling can be found here: https://hister.org/docs/user-handling https://hister.org/docs/user-handling It also has a "public mode" where anyone can search the indexed content, but only authenticated users can add or modify it.
- Carrok 26d ago
  It seemed like the public mode was the default when I set it up. If so, that’s a fairly dangerous default as keeping a “clean” history with no secrets leaked seems neigh impossible.
  - asciimoo 26d ago
    The default configuration binds only to localhost, and a fresh installation starts with an empty database/index. Could you clarify which specific attack surface you are concerned about in that scenario?
    - Carrok 26d ago
      I’m not concerned about an attack scenario. I’m just saying that using the docker image, if someone (or their agent) isn’t careful, they could expose their browsing history publicly fairly easily. It might just be nice to default to at least a user and pass login rather than just wide open.
androidu 26d ago
I've set it up today with OIDC AuthN (Keycloak), removed the username/password fields & kept only OIDC, it's working flawless. Also indexed data is persisted on a per-user basis, so you got this isolation and certainty that your searches will not be polluted by your family's

shevy-java 26d ago

This is like Hipster. But missing a 'p'.

robotmay 26d ago

Ooh I’ve been thinking about this idea for years, I’m glad someone beat me to it. Guess I have something to play with over the rest of the weekend!

renegat0x0 26d ago

I have created something similar, yet different. I maintain my own Internet domain index. https://github.com/rumca-js/Internet-Places-Database https://github.com/rumca-js/Internet-Places-Database Also I maintain android app that can be used to search places. https://f-droid.org/en/packages/io.github.rumcajs.offlinewebsearch https://f-droid.org/en/packages/io.github.rumcajs.offlineweb... I believe hister you have to fill in with your data, right?

wmchen 26d ago

I set this up a few months ago based on asciimoo's comments on HN, and barely used it at first, but I realized not too long ago that it could be a pretty useful research tool for one of my hobbies (award travel), that revolves around being in the know around various concepts and quirks. I scraped and imported posts from the blogs I regularly reference for award travel, then hooked it up to OpenCode/Codex as an MCP server and used that corpus for research on those topics. So I can ask things like "has anyone ever mentioned running into this problem before?" [1] If you have a hobby or working situation that requires you to regularly reference a core set of websites or reference materials, Hister provides almost all the tools out of the box to start a search engine against it. The default datasets they promote include the Python Stlib, MDN and RFC corpus, as an example. [2] [1]: https://wmchen.com/blog/revisiting-hister/ https://wmchen.com/blog/revisiting-hister/ [2]: https://hister.org/datasets https://hister.org/datasets

asciimoo 26d ago
Wow, this is a really inspiring use case and blog post. Thanks for sharing it. What tools or features would Hister need to support your complete search workflow?
- wmchen 26d ago
  Thank you so much for building this! It's a really awesome piece of work and I'm grateful for your work. I'd love to sponsor you on Github in the near future. The only thing that I think would be interesting to see is native support for crawling via a sitemap.xml instead of recursively. I worked around this by implementing a basic scraper that fetched pages exclusively from the sitemap.xml to add into Hister. I think you're already aware of this, but I also experienced some data loss during the import because I was running a concurrent reindex. I clocked it pretty quickly so I didn't think too much of it. [1] [1] "TODO store new documents in both indexes while running reindex to guarantee not losing any data." @ https://github.com/asciimoo/hister/blob/master/server/indexer/indexer.go https://github.com/asciimoo/hister/blob/master/server/indexe...
Aboutplants 26d ago
Oh this gets my brain spinning! Thanks for the tips

thataccount 26d ago

[flagged]

wolfy1993 26d ago
I'll be honest I wouldn't blame OP for not finding this (I found a species of hister beetle before anything to do with Hitler) I feel this is a weird spin on the "Clicks to Hitler" game
- thataccount 26d ago
  I was not using Wikipedia to get there. This is something I immediately recognized from research about World War II and possible prophetic fulfillment. I definitely wasn't playing a game when I came across this information either, but I get your drift.
  - ButlerianJihad 26d ago
    Interestingly, when I searched "hister" on English Wikipedia, I was sent to https://en.wikipedia.org/wiki/Danube#Names_and_etymology https://en.wikipedia.org/wiki/Danube#Names_and_etymology Where it says that “Hister” is a variant of its Latin name. I think the name is lovely with no reason to change. Surely there are few people who would connect this with silly medieval superstition. Perhaps you can take part in wiping out that association.
    - thataccount 26d ago
      [flagged]
- hardgeus 25d ago
  I've been on site site for years. I created an account just to comment on this thread. Holy crap, the only time I've ever heard 'Hister' was in a book I read on Nostradamus like 30 years ago.
  - 25d ago

exe34 26d ago

Can it import browser bookmarks? It says browser history and a bunch of other bookmark services but not specifically browser bookmarks.

asciimoo 26d ago
Yes, it can import data from various sources, including browser history, and popular self-hosted bookmarking apps: https://hister.org/docs/import https://hister.org/docs/import
- exe34 26d ago
  Is this reply from Gemini? Because I'm used to gemini gaslighting me. The answer is the opposite of my question.
  - asciimoo 26d ago
    Doh, sorry, the answer was coming from me who did not read the question properly. It currently cannot import browser bookmarks, but it is a good idea. Added to my TODO.
    - exe34 26d ago
      Ah sorry didn't mean to be mean, I just wondered if you had automated it haha!
    - exe34 24d ago
      Figured I can just export the bookmarks and then hister index the urls.

alexnewman 26d ago

Time to shameless plug my own easy to host your own content search index Webtm.io All open source and small enough to deploy. I deploy to cf webworkers so it’s the only place it’s tested. One cool thing is we work on iOS, chrome and friends, Firefox and pretty much everywhere. We do require you bring your own LLM though.

quasigod 26d ago

I just finished setting this up yesterday and I'm kind of obsessed with it. I was previously a heavy user of Karakeep, but I hated forgetting to save something and losing it. I also think Hister's semantic search is a better solution than AI generated summaries and tags. Support for local docs is super cool too, I have it set up to index my org notes directory.

ebr4him 26d ago

Is the design vibe coded in Codex by any chance? Seeing a lot of similar designs out there and it’s really starting to annoy me.

hallole 26d ago
Really? The website, you mean? :( I thought to myself, "what a cute, well designed webpage!" when I saw it. Maybe with time this site will stand out like those Claudeslop ones.
lucaaa 26d ago
same
asciimoo 26d ago
The whole design is created by a main contributor and IIRC the style is called neobrutalism.
tolerance 25d ago
I think that the tells, if they are in fact legitimate, are negligible at best or subtle at worst. It looks like the site uses Tailwind, so if AI was involved it wasn't generating the design from scratch. If it did then I'm impressed because usually it sucks. This is alright.

cowmanOG 26d ago

Would be really cool to have Zotero library integration / import support!

knowaveragejoe 26d ago

I love the idea overall, but something about browser extensions give me a bad taste. Am I overthinking it? I realize the extension is open source, of course.

kinibha 26d ago

Any way to add for mobile?

quasigod 25d ago
If you're on android the extension works for Firefox

jbellis 25d ago

Love to see this. I started something similar a couple years ago but didn't quite have the patience to make the browser extensions really bulletproof. Planning to contribute significant improvements to the vector search side here.

mmargenot 25d ago

This is so cool! For the time being I’m still locked into notion for my handwritten knowledge base, but I love this for incorporating external information. For the semantic search is there any chunking/processing that happens with the content or do you need to be diligent about having a large embedding context (and/or small content)?

triilman 25d ago

can it search .pdf content as easy as searching content in txt file?

0n0n0m0uz 25d ago

Cant wait to check this out

vivzkestrel 25d ago

- lots of stupid questions to the author from a guy who has no idea about search engines - let us say I want to index every blog ever listed on HN - should be a small subset of the 400 billion pages out there on the internet no? - First I need to gather data, what do you use to load so many webpages rapidly? asyncio with aiohttp in python? are there better options? - how do you handle proxies? rotation? are there libraries you recommend for this? - what about pages that use cloudflare? or block your request or present a captcha or a challenge of some kind? - what are the filetypes you collect? only html or media as well? - where and in what format do you store all these collected files? flat file storage? duckdb? postgres? hstore? something else? - what is the frequency at which you refresh each page? once a day? once a week? something else? - what kind of pre-processing do you use on the collected data? remove extra spaces? special characters? some kind of complex regex pipeline? LLM? - how do you match the incoming query with processed data? simple text matching? regex? vector embedding match? something else?

9935c101ab17a66 22d ago
A lot of these can be answered by checking the documentation: https://hister.org https://hister.org. Others could be answered by examining the source code, as it is open source. It's good etiquette to check existing documentation and resources before badgering OSS project teams with a lengthy list of questions.

vladshiyan 25d ago

[flagged]

stayux 25d ago

Thanks, this will pair wonderfully with my Capcat.org project.

ndom91 25d ago

What's the disk space usage like? i.e. average per-day/week additional storage in your usage?

uygar 25d ago

[dead]

dmd 25d ago

It’s always been bizarre to me how bad browsers themselves are at searching their own history. This looks great.

phendrenad2 25d ago

Ister (Latin Hister, Ancient Greek Ἴστρος / Istros) is the ancient classical name used by the Greeks and Romans for the Danube River, specifically referring to its lower course.

evanjrowley 25d ago

While I love Karakeep, the truth is that I'm using it to solve this exact same problem, even though its project goals are not as well aligned. I would seriously consider giving Hister a try because it seems like a better fit. Bonus points for being written in Go. I suppose the one thing I need would be an easy way to share links into Hister. Hopefully we will see both Android/iOS mobile apps to send links into Hister and better support for Safari. I wonder what could be possible with the extensions support available on mobile browsers too (Safari, MS Edge, Firefox).

bpavuk 25d ago
the Hister extension is available on Firefox for Android :) https://addons.mozilla.org/en-US/firefox/addon/hister/ https://addons.mozilla.org/en-US/firefox/addon/hister/

retrochameleon 25d ago

Any chance I could integrate or import data from Readwise Reader? (Link archival tool)