5 ms·
Here’s an even simpler take: just embed everything the first time, then track what was changed. Use a cheap model to summarize and clean up the documents/chats
by refactor_master 22d ago
Here’s an even simpler take: just embed everything the first time, then track what was changed. Use a cheap model to summarize and clean up the documents/chats with summary and keywords. Unless you have entire libraries of books to embed it’s going to be a few hundred dollars of API calls.
Then, throw it all in BigQuery. Handles all the vector stuff natively.
Sprinkle an agentic bot UI thing on top to make it appear all-knowing and magical.
I assume other vendors than Google have a similar batteries-included approach you can just plug in.
- cpursley 22d agoYep, lock into some vendor from day 1. Great idea!
- usernametaken29 22d ago> embed everything the first time This assumes your text is small. Try embedding pdf reports - though luck. It surely won’t fit into most embeddings. I can think of many more examples: books, news articles, medical reports, insurance claims etc. they’re all too big to “index it all at once”
- robrorcroptrer 22d agoWhat about splitting bigger content into chunks before embedding?
- freakynit 22d agoHow are you gonna handle the relations that span across individual chunks... if a later chunk refers something from 2 chunks before using `it`, rather than proper name, how will you handle that? Because at query time, that later chunk would not match.
- refactor_master 22d agoHumans usually have ways around that in longer documents eg page numbers, paragraphs, links. If someone gave me a report, in my hands, that said “see ‘it’” I’d also be confused.
- gf000 22d agoAbsolutely a novice in this topic, but I would imagine that by simply having sufficiently big chunks it's simply not a problem? You surely have enough information in like a couple of paragraphs to denote in vector space roughly what it is about. So that both chunks would get found by a vector search, and then whatever is the logic it may put the whole original text of those chunks into context, but in any case enough so that an LLM can "reason" about the references in-between the two.
- harlanji 22d agoChunks can only be as large as the embedding model’s token limit, about 512-1024 tokens usually. Anything longer gets truncated. Natural language processing could expanded references, but it starts to get tricky. Do you use Graph RAG, embed another version of the chunk that is distinct from the full text version, etc.. Another layer of processing and data to keep in sync if the source dan be updated.
- gf000 22d ago(assuming English text) 512 tokens ≈ 350–400 words ≈ a long paragraph or two. 1024 tokens ≈ 700–800 words ≈ about a page and a half to two pages. I would be very surprised if that amount of text is not enough to encode a general topic into the embedding (otherwise, what would be the whole point of them?). So if there is a meaningful reference in C referring to A (assuming A-B-C consecutive 1-2 paragraphs), I would expect that the content of the two at least superficially resemble each other, and a vector query for one would return both. (And also, if I am including A in the context after retrieval, then I better give some context before-after as well -- and references tend to be local). But feel free to prove me wrong, I'm mostly just guessing, not even an educated (in the given topic) guess here.
- khalic 22d agoyou won't get anything out of a whole book embedding anyway, even a structured page is too much
- j0selit0 22d agoI'm sorry is this ironic or not? doesn't sounds simple at all