7 ms·
Did you consider extracting knowledge from the doc into a graph database (while passing the Q&A) and compressing the graph instead?
by adsharma 22d ago
Did you consider extracting knowledge from the doc into a graph database (while passing the Q&A) and compressing the graph instead?
- adsharma 22d agohttps://huggingface.co/datasets/ladybugdb/github-knowledge-compressor/tree/main https://huggingface.co/datasets/ladybugdb/github-knowledge-c... These methods could not beat the 480 token result because of the small dataset. But for a sufficiently large corpus with graph reordering and columnar parquet compression, they could be quite competitive. The parquet files are self explanatory. For knowledge.lbdb.zst, first decompress with zstd and then run a cypher such as: lbug> MATCH (a:E)-[t:TRIPLE]->(b:E) RETURN a.name AS subj, t.predicate AS pred, b.name AS obj, t.props AS props, t.scope AS scope;