7 ms·
Very interesting project! I guess it could be even better if you didn't have to ingest the session data into a database but just build an index on top. I have a
by malandin 3mo ago
Very interesting project! I guess it could be even better if you didn't have to ingest the session data into a database but just build an index on top. I have an idea how to do it
- luca-ctx 3mo agoThanks and this is a very interesting idea! We considered this, but the main thing you gain from this tradeoff is some disk space and cleaner retention semantics from not having to duplicate all of the searchable text. But you still have to do the parsing and ingestion work to build the index in the first place, so CPU time does not go away. And you still have to store the indexes and enough metadata to map results back to the raw session files, which bounds the benefit of not duplicating the data. The main downside is flexibility (you would lose the ability to do arbitrary SQL queries, semantic search on top of structured corpus, etc) But I would love to see if I can be proven wrong on this!
- mkornaukhov 3mo agoWe've been working on remote search indexing in our project and it works pretty good. since we are building a Postgres-compatible database, everything is pure SQL. I'd say we could join forces if you're up for it.
- luca-ctx 3mo agoVery cool! Do you have any of it OSS? Or drop me an email: luca@ctx.rs