I’ve done some experiments and the ideal seems to be semantic chunks of max 175 tokens. But only if those chunks are already very dense (use summarisation to ge

I’ve done some experiments and the ideal seems to be semantic chunks of max 175 tokens. But only if those chunks are already very dense (use summarisation to get there)