nhirschfeld

8 ms·

1.
by nhirschfeld 4mo ago
Hi HN! I'm the maintainer of Kreuzberg, an open-source document intelligence library ( https://github.com/kreuzberg-dev/kreuzberg ). Some of you may have used it for RAG ingestion. We're launching Kreuzberg Clo
2.
Show HN: Kreuzberg Cloud – ultra fast content intelligence – in public beta(kreuzberg.dev)
5 points by nhirschfeld 4mo ago|4 comments
3.
Show HN: Liter-LLM, Universal LLM client in Rust with bindings for 11 languages(github.com)
2 points by nhirschfeld 6mo ago|0 comments
4.
Show HN: Kreuzberg Comparative Benchmarks(kreuzberg.dev)
1 points by nhirschfeld 7mo ago|0 comments
5.
Show HN: Kreuzberg v3.0 – Modern Python Document Extraction
5 points by nhirschfeld 1y ago|0 comments
6.
Ask HN: Interest in a pgvector-based RAG system library?
3 points by nhirschfeld 2y ago|2 comments
7.
by nhirschfeld 2y ago
You'll need to use a different OCR engine. Look at easy ocr
8.
by nhirschfeld 2y ago
Yes, there have already been several suggestions here for other backend etc. You should try using a different PSM to see if you get better results. If it's scientific texts specifically, look at grobid
9.
by nhirschfeld 2y ago
You still need to write it to file to process it via pandoc/tesseract etc. There are alternative options to tesseract ofc.
10.
by nhirschfeld 2y ago
thats why Kreuzberg also exposes a sync API for you to consume.
11.
by nhirschfeld 2y ago
didnt know this!
12.
by nhirschfeld 2y ago
I haven't, testing it out is on my todo list for sure
13.
by nhirschfeld 2y ago
I google this for a while...
14.
by nhirschfeld 2y ago
I'm actually considering another library with optional API called `Kreuzköln` - probably without the Umlaut!
15.
by nhirschfeld 2y ago
Retrieval Augmented Generation. Its a class of techniques for generating content using LLMs. I'd recommend Googling this.
16.
by nhirschfeld 2y ago
Thanks for asking! It's both. The OCR part is ofc CPU bound, but the entire text extraction involves reading files, or writing and then reading files. Without async, these simply block. As for efficiency - if you're working in an
17.
by nhirschfeld 2y ago
Amazing, would be interested in reading your experience
18.
by nhirschfeld 2y ago
Sorry to hear...
19.
by nhirschfeld 2y ago
Yup, easy OCR is good. My reasons for using Tesseract - easy OCR is larger, and it has a significant cold start. It benchmarks better for many OCR tasks though, so I'm thinking of adding it as an alternative backend.
20.
by nhirschfeld 2y ago
interesting!
21.
by nhirschfeld 2y ago
lol ;). But seriously, in 13 years living here, only one guy tried to pick pocket me.
22.
by nhirschfeld 2y ago
Thanks, I'll check these links. In my tests I found tesseract quite good for regular text documents. For other kinds of texts it's not great. As for using models - there are some good small language models as well, and of course L
23.
by nhirschfeld 2y ago
That's my neighborhood in Berlin, which I love
24.
by nhirschfeld 2y ago
So, for PDF we need to distinguish between two types of text extraction- 1. Text extraction from a searchable PDF. 2. OCR. For 1. Kreuzberg uses pypdfium2, which is a python binding for pdfium - the chromium PDF engine. In this regard Kreuz
25.
Show HN: Kreuzberg – Modern async Python library for document text extraction(github.com)
197 points by nhirschfeld 2y ago|75 comments
26.
by nhirschfeld 3y ago
We're working very hard on shipping it
27.
by nhirschfeld 5y ago
Seems that urls are not embedded in text fields, maybe the do in comments? Repo: https://github.com/Goldziher/starlite Docs: https://goldziher.github.io/starlite/ Article: https://itnex
28.
Show HN: Starlite, the little ASGI framework that can
3 points by nhirschfeld 5y ago|1 comments
29.
Show HN: Starlite – A New Python ASGI API Framework(github.com)
2 points by nhirschfeld 5y ago|0 comments