5 ms·
Knowledge Compressor
- Mappleton 24d agoHow far can documentation be compressed without losing its meaning?
- neoncontrails 24d agoThis is a good question! The answer is, unfortunately, uncomputable – or so we think. But if you manage to find a more satisfying answer, you'll have solved one of the more beguiling problems of the last century, which is the basis for most of modern Information Theory. You can learn more about it here: https://homepages.cwi.nl/~paulv/kolmogorov.html https://homepages.cwi.nl/~paulv/kolmogorov.html
- idan 24d agoHello! Idan (from Next) here, and others from the team will drop by the thread if you have questions!
- adsharma 24d agoDid you consider extracting knowledge from the doc into a graph database (while passing the Q&A) and compressing the graph instead?
- adsharma 24d agohttps://huggingface.co/datasets/ladybugdb/github-knowledge-compressor/tree/main https://huggingface.co/datasets/ladybugdb/github-knowledge-c... These methods could not beat the 480 token result because of the small dataset. But for a sufficiently large corpus with graph reordering and columnar parquet compression, they could be quite competitive. The parquet files are self explanatory. For knowledge.lbdb.zst, first decompress with zstd and then run a cypher such as: lbug> MATCH (a:E)-[t:TRIPLE]->(b:E) RETURN a.name AS subj, t.predicate AS pred, b.name AS obj, t.props AS props, t.scope AS scope;