5 ms·
You might want to take a look at https://github.com/segment-any-text/wtpsplit https://github.com/segment-any-text/wtpsplit It uses a similar approach but the f
by mathis-l 1y ago
You might want to take a look at https://github.com/segment-any-text/wtpsplit https://github.com/segment-any-text/wtpsplit
It uses a similar approach but the focus is on sentence/paragraph segmentation generally and not specifically focused on RAG. It also has some benchmarks. Might be a good source of inspiration for where to take chonky next.
- hessdalenlight 1y ago[dead]
- vunderba 1y agoThis is the library that I use, mainly around very noisy IRC chat transcripts and it works pretty well. OP I'd love to see a paragraph matching comparison benchmark against wtpsplit to see how well Chonky stacks up.