chiccomagnus

6 ms·

1.
by chiccomagnus 1y ago
IMHO, your article is missing an important point: 90% of implementations today flatten documents to plain text before chunking them. Why not consider the visual appearance that the human gave to the document? Using layout information combin
2.
by chiccomagnus 1y ago
How does credits works?
3.
by chiccomagnus 1y ago
If you don't want to reinvent the wheel, we have built exactly that, goggle "Preprocess"
4.
by chiccomagnus 1y ago
Have you compared this solution with tools like Preprocess, Reducto, etc.. ? I'm curious about the performance gain you can achieve with your approach
5.
by chiccomagnus 2y ago
It seems it tries to always extract tables even if the content is just text. Is not working and at a similar price you can get high performing solutions like preprocess.co and similars.
6.
by chiccomagnus 2y ago
I'm curious to know how you handle the chunking of complex documents
7.
by chiccomagnus 2y ago
If for "documents" you mean PDF, Word, Powerpoints, Excel you should try preprocess.co If that’s not what you meant, can you please clarify?
8.
by chiccomagnus 2y ago
Do you use naive chunking? Have you tried something else?
9.
by chiccomagnus 2y ago
By chance, have you tried preprocess.co for text extraction + chucking?
10.
by chiccomagnus 2y ago
I see a huge missing point here: real world files are PDF, Word, PowerPoint and Excel, not only plain text.
11.
by chiccomagnus 2y ago
There are more sophisticated chunking strategies, you will lose lots of context like that
12.
by chiccomagnus 2y ago
Well-written article, missing key considerations: - Titles matter, a lot: if you add the title of the section at the start of each chunk you will get 10x better embeddings and so more accurate results. - The size doesn't matter: It dep
13.
by chiccomagnus 2y ago
The real point that has been completely ignored is that data for real applications come from Office and PDF files, and making them plain text throws all the visual information.
14.
by chiccomagnus 3y ago
Sorry, but it is really strange how haphazard it is. Am I doing something wrong? It's missing more or less all the titles The order of the text is casual The system chunks at random points in the text https://mlops.community
15.
by chiccomagnus 3y ago
check out this https://preprocess.co
16.
by chiccomagnus 3y ago
The datasets are composed by Documents like PDFs and/or Office files?
17.
by chiccomagnus 3y ago
That's a good point, and documents too needs different chunking techniques. You don't want to split a word file the same way you split an excel...
18.
by chiccomagnus 3y ago
You are both right about chunking, and i think is one of the main challenges. About more intelligent chunking approaches, i think you have to give a try to to preprocess.co It's able to preprocess and chunk PDFs, Office Files, and HTML
19.
Seeking Datasets for Evaluating File Chunking Strategies for RAG/LLM
1 points by chiccomagnus 3y ago|3 comments
20.
by chiccomagnus 3y ago
doesn't the basic chunking you are using for splitting documents have any impact on results?
21.
by chiccomagnus 3y ago
It's an interesting topic i'm working on at the moment. Given the ones you shared, I figured out additional challenges. Especially for PDF and Office documents, it's really important to consider the original layout. You shoul
22.
Does someone have ProductHunt Invite?
1 points by chiccomagnus 10y ago|0 comments