5 ms·

Daniel, your work is changing the world. More power to you. I setup a pipeline for inference with OCR, full text search, embedding and summarization of land re

Daniel, your work is changing the world. More power to you. I setup a pipeline for inference with OCR, full text search, embedding and summarization of land records dating back 1800s. All powered by the GGUF's you generate and llama.cpp. People are so excited that they can now search the records in multiple languages that a 1 minute wait to process the document seems nothing. Thank you!

wok4899 6mo ago
This is a very interesting project. If it's publicly available, would you mind sharing it? I would love to understand how it works. Ps: found your other comments, thanks.
danielhanchen 6mo ago
Oh appreciate it! Oh nice! That sounds fantastic! I hope Gemma-4 will make it even better! The small ones 2B and 4B are shockingly good haha!
- qingcharles 6mo ago
  Just switched from 3.1 Flash Lite to Gemma-4 31B on the AI Studio API since there is a generous 1500/day on non-billed projects. It's doing fantastic.
polishdude20 6mo ago
Hey in really interested in your pipeline techniques. I've got some pdfs I need to get processed but processing them in the cloud with big providers requires redaction. Wondering if a local model or a self hosted one would work just as well.
- jorl17 6mo ago
  Seconded, would also love to hear your story if you would be willing
- evilelectron 6mo ago
  I run llama.cpp with Qwen3-VL-8B-Instruct-Q4_K_S.gguf with mmproj-F16.gguf for OCR and translation. I also run llama.cpp with Qwen3-Embedding-0.6B-GGUF for embeddings. Drupal 11 with ai_provider_ollama and custom provider ai_provider_llama (heavily derived from ai_provider_ollama) with PostreSQL and pgvector. People on site scan the documents and upload them for archival. The directory monitor looks for new files in the archive directories and once a new file is available, it is uploaded to Drupal. Once a new content is created in Drupal, Drupal triggers the translation and embedding process through llama.cpp. Qwen3-VL-8B is also used for chat and RAG. Client is familiar with Drupal and CMS in general and wanted to stay in a similar environment. If you are starting new I would recommend looking at docling.
  - lwhi 6mo ago
    Are you linking any of the processes using the Drupal AI module suite?
    - evilelectron 6mo ago
      Yes, they are all linked using Drupal's AI modules. I have an OpenCV application that removes the old paper look, enhances the contrast and fixes the orientation of the images before they hit llama.cpp for OCR and translation.
- chrisweekly 6mo ago
  Disclaimer: I'm an AI novice relative to many here. FWIW last wknd I spent a couple hours setting up self-hosted n8n with ollama and gemma3:4b [EDIT: not Qwen-3.5], using PDF content extraction for my PoC. 100% local workflow, no runtime dependency on cloud providers. I doubt it'd scale very well (macbook air m4, measly 16GB RAM), but it works as intended.
  - polishdude20 6mo ago
    How do you extract the content? OCR? Pdf to text then feed into qwen? I tried something similar where I needed a bunch of tables extracted from the pdf over like 40 pages. It was crazy slow on my MacBook and innacurate
    - philipkglass 6mo ago
      If you have a basic ARM MacBook, GLM-OCR is the best single model I have found for OCR with good table extraction/formatting. It's a compact 0.9b parameter model, so it'll run on systems with only 8 GB of RAM. https://github.com/zai-org/GLM-OCR https://github.com/zai-org/GLM-OCR Use mlx-vlm for inference: https://github.com/zai-org/GLM-OCR/blob/main/examples/mlx-deploy/README.md https://github.com/zai-org/GLM-OCR/blob/main/examples/mlx-de... Then you can run a single command to process your PDF: glmocr parse example.pdf Loading images: example.pdf Found 1 file(s) Starting Pipeline... Pipeline started! GLM-OCR initialized in self-hosted mode Using Pipeline (enable_layout=true)... === Parsing: example.pdf (1/1) === My test document contains scanned pages from a law textbook. It's two columns of text with a lot of footnotes. It took 60 seconds to process 5 pages on a MBP with M4 Max chip. After it's done, you'll have a directory output/example/ that contains .md and .json files. The .md file will contain a markdown rendition of the complete document. The .json file will contain individual labeled regions from the document along with their transcriptions. If you get all the JSON objects with "label": "table" from the JSON file, you can get an HTML-formatted table from each "content" section of these objects. It might still be inaccurate -- I don't know how challenging your original tables are -- but it shouldn't be terribly slow. The tables it produced for me were good. I have also built more complex work flows that use a mixture of OCR-specialized models and general purpose VLM models like Qwen 3.5, along with software to coordinate and reconcile operations, but GLM-OCR by itself is the best first thing to try locally.
- tehologist 6mo ago
  Python pdftools to convert to images and tesseract to ocr them to text files. Fast free and can run on CPU.
irishcoffee 6mo ago
> your work is changing the world I realize this may have been hyperbole, but it sure isn't changing the world.
- a96 6mo ago
  For relatively small values of changing or world, it sure is. In the world of local models, Unsloth is one of the most significant projects there is.
Breza 6mo ago
I'm very active in family history and this kind of project is massively helpful, thank you

wok4899 6mo ago

This is a very interesting project. If it's publicly available, would you mind sharing it? I would love to understand how it works. Ps: found your other comments, thanks.

danielhanchen 6mo ago

Oh appreciate it! Oh nice! That sounds fantastic! I hope Gemma-4 will make it even better! The small ones 2B and 4B are shockingly good haha!

qingcharles 6mo ago
Just switched from 3.1 Flash Lite to Gemma-4 31B on the AI Studio API since there is a generous 1500/day on non-billed projects. It's doing fantastic.

polishdude20 6mo ago

Hey in really interested in your pipeline techniques. I've got some pdfs I need to get processed but processing them in the cloud with big providers requires redaction. Wondering if a local model or a self hosted one would work just as well.

jorl17 6mo ago
Seconded, would also love to hear your story if you would be willing
evilelectron 6mo ago
I run llama.cpp with Qwen3-VL-8B-Instruct-Q4_K_S.gguf with mmproj-F16.gguf for OCR and translation. I also run llama.cpp with Qwen3-Embedding-0.6B-GGUF for embeddings. Drupal 11 with ai_provider_ollama and custom provider ai_provider_llama (heavily derived from ai_provider_ollama) with PostreSQL and pgvector. People on site scan the documents and upload them for archival. The directory monitor looks for new files in the archive directories and once a new file is available, it is uploaded to Drupal. Once a new content is created in Drupal, Drupal triggers the translation and embedding process through llama.cpp. Qwen3-VL-8B is also used for chat and RAG. Client is familiar with Drupal and CMS in general and wanted to stay in a similar environment. If you are starting new I would recommend looking at docling.
- lwhi 6mo ago
  Are you linking any of the processes using the Drupal AI module suite?
  - evilelectron 6mo ago
    Yes, they are all linked using Drupal's AI modules. I have an OpenCV application that removes the old paper look, enhances the contrast and fixes the orientation of the images before they hit llama.cpp for OCR and translation.
chrisweekly 6mo ago
Disclaimer: I'm an AI novice relative to many here. FWIW last wknd I spent a couple hours setting up self-hosted n8n with ollama and gemma3:4b [EDIT: not Qwen-3.5], using PDF content extraction for my PoC. 100% local workflow, no runtime dependency on cloud providers. I doubt it'd scale very well (macbook air m4, measly 16GB RAM), but it works as intended.
- polishdude20 6mo ago
  How do you extract the content? OCR? Pdf to text then feed into qwen? I tried something similar where I needed a bunch of tables extracted from the pdf over like 40 pages. It was crazy slow on my MacBook and innacurate
  - philipkglass 6mo ago
    If you have a basic ARM MacBook, GLM-OCR is the best single model I have found for OCR with good table extraction/formatting. It's a compact 0.9b parameter model, so it'll run on systems with only 8 GB of RAM. https://github.com/zai-org/GLM-OCR https://github.com/zai-org/GLM-OCR Use mlx-vlm for inference: https://github.com/zai-org/GLM-OCR/blob/main/examples/mlx-deploy/README.md https://github.com/zai-org/GLM-OCR/blob/main/examples/mlx-de... Then you can run a single command to process your PDF: glmocr parse example.pdf Loading images: example.pdf Found 1 file(s) Starting Pipeline... Pipeline started! GLM-OCR initialized in self-hosted mode Using Pipeline (enable_layout=true)... === Parsing: example.pdf (1/1) === My test document contains scanned pages from a law textbook. It's two columns of text with a lot of footnotes. It took 60 seconds to process 5 pages on a MBP with M4 Max chip. After it's done, you'll have a directory output/example/ that contains .md and .json files. The .md file will contain a markdown rendition of the complete document. The .json file will contain individual labeled regions from the document along with their transcriptions. If you get all the JSON objects with "label": "table" from the JSON file, you can get an HTML-formatted table from each "content" section of these objects. It might still be inaccurate -- I don't know how challenging your original tables are -- but it shouldn't be terribly slow. The tables it produced for me were good. I have also built more complex work flows that use a mixture of OCR-specialized models and general purpose VLM models like Qwen 3.5, along with software to coordinate and reconcile operations, but GLM-OCR by itself is the best first thing to try locally.
tehologist 6mo ago
Python pdftools to convert to images and tesseract to ocr them to text files. Fast free and can run on CPU.

irishcoffee 6mo ago

> your work is changing the world I realize this may have been hyperbole, but it sure isn't changing the world.

a96 6mo ago
For relatively small values of changing or world, it sure is. In the world of local models, Unsloth is one of the most significant projects there is.

Breza 6mo ago

I'm very active in family history and this kind of project is massively helpful, thank you