5 ms·

I gave a blanket ban on pdftotext to my agents. The output can get so mangled that a smart human wouldn’t untangle it. Did you try understanding the output from

I gave a blanket ban on pdftotext to my agents. The output can get so mangled that a smart human wouldn’t untangle it. Did you try understanding the output from pdftotext yourself? My approach is just ocr-ing with Terra or Gemini flash + checking citations with source both ways. But if I wanted to avoid llm calls, I’d just tell Fable to build a pdf reader directly from pdf binary format. Should be way more robust.

rayiner 5d ago
A PDF is a command stream designed for rendering. Interpreting the command stream to get the positions of each glyph is deterministic and existing libraries (I use both pdf_oxide and lopdf) do that fine. Once you have glyph positions, you need to use various heuristics to reconstruct words, paragraphs, columns, headers and footers, etc. For example, in a patent document, there's two columns with a gutter in the middle of line numbers. If you interpret the document as having a single line, you'll get numbers mixed up with the text, which can throw off efforts to find particular phrases. PDF builders also insert all sorts of weird crap into the OCR layers that has to get normalized out. It's just a pretty pedestrian data-munging problem where there's no closed form perfect solution and you have to use various heuristics to get the right result.
- keeda 5d ago
  Yes, but the edge cases are infinite and so heuristics don't scale well. As an example, at some point you would likely find yourself with "dueling" heuristics, forcing you to tune them, which is brittle, or find yet another heuristic as a tie-breaker, which ratchets up the complexity. (I just spent a lot of time on an adjacent but much simpler problem before finally giving up on churning heuristics!) As an example, many times it is impossible to determine the order of some words from just position data without considering the meanings of those words. This is why LLMs / VLMs are so much better at this task, because they can look at the document holistically like we can. Also, funny that you mention patents, something I've worked on in the past as well! If you're looking only at US Patents, the USPTO data resource is much, much better: https://data.uspto.gov/home https://data.uspto.gov/home -- they provide the text in XML format (https://www.uspto.gov/learning-and-resources/xml-resources https://www.uspto.gov/learning-and-resources/xml-resources) which is also pretty complex but wayyyy easier to parse than PDFs!
  - rayiner 5d ago
    > This is why LLMs / VLMs are so much better at this task, because they can look at the document holistically like we can. Totally agreed. But in this use case, PDFs are the working format, not just an archival format. An offline batch process to ingest the PDFs isn’t feasible. Unless there are some super fast LLMs I’m not aware of that can handle tens of PDF pages per second. It seems like Grok and Claude don’t try to read the PDF directly, they use pdftotext or some Python wrapper over pdfium. But maybe I’m missing something!
- hollerith 5d ago
  >It's just a pretty pedestrian data-munging problem In other words, how complicated could it be? Well, if Adobe has been introducing complications and making the format brittle and inflexible over the last 35 years to make it hard for its competitors to write software to process PDF files, quite complicated.
- kolinko 5d ago
  Yeah that’s why heuristics should work on the lowest possible layer, not on pdftotext. If you use pdftotext you’re stripping positional data and other stuff. Do you use a public set of documents? I bet I could almost oneshot this with my harness :p
  - rayiner 5d ago
    Yes, the tool I’m describing works on what the pdf_oxide crate returns, which includes glyph positions. Here’s a public appendix from a recent Federal Circuit case. It has a representative assortment of documents (opinions, briefs, patents, transcripts) but contains only cited pages to the appendix pagination is non-consecutive: https://www.courtlistener.com/docket/68048163/15/ecofactor-inc-v-google-llc/ https://www.courtlistener.com/docket/68048163/15/ecofactor-i... The underlying docket is usually 100 to 1,000 times larger than this but it’s similar types of files and the appendix shows the various types of headers and footers that can exist.
    - kolinko 4d ago
      Here’s the oneshot, not sure if it’s slop or not though :) https://kolinko.eu/pdf-reading-order/ https://kolinko.eu/pdf-reading-order/ But I wonder about your opinion.

rayiner 5d ago

A PDF is a command stream designed for rendering. Interpreting the command stream to get the positions of each glyph is deterministic and existing libraries (I use both pdf_oxide and lopdf) do that fine. Once you have glyph positions, you need to use various heuristics to reconstruct words, paragraphs, columns, headers and footers, etc. For example, in a patent document, there's two columns with a gutter in the middle of line numbers. If you interpret the document as having a single line, you'll get numbers mixed up with the text, which can throw off efforts to find particular phrases. PDF builders also insert all sorts of weird crap into the OCR layers that has to get normalized out. It's just a pretty pedestrian data-munging problem where there's no closed form perfect solution and you have to use various heuristics to get the right result.

keeda 5d ago
Yes, but the edge cases are infinite and so heuristics don't scale well. As an example, at some point you would likely find yourself with "dueling" heuristics, forcing you to tune them, which is brittle, or find yet another heuristic as a tie-breaker, which ratchets up the complexity. (I just spent a lot of time on an adjacent but much simpler problem before finally giving up on churning heuristics!) As an example, many times it is impossible to determine the order of some words from just position data without considering the meanings of those words. This is why LLMs / VLMs are so much better at this task, because they can look at the document holistically like we can. Also, funny that you mention patents, something I've worked on in the past as well! If you're looking only at US Patents, the USPTO data resource is much, much better: https://data.uspto.gov/home https://data.uspto.gov/home -- they provide the text in XML format (https://www.uspto.gov/learning-and-resources/xml-resources https://www.uspto.gov/learning-and-resources/xml-resources) which is also pretty complex but wayyyy easier to parse than PDFs!
- rayiner 5d ago
  > This is why LLMs / VLMs are so much better at this task, because they can look at the document holistically like we can. Totally agreed. But in this use case, PDFs are the working format, not just an archival format. An offline batch process to ingest the PDFs isn’t feasible. Unless there are some super fast LLMs I’m not aware of that can handle tens of PDF pages per second. It seems like Grok and Claude don’t try to read the PDF directly, they use pdftotext or some Python wrapper over pdfium. But maybe I’m missing something!
hollerith 5d ago
>It's just a pretty pedestrian data-munging problem In other words, how complicated could it be? Well, if Adobe has been introducing complications and making the format brittle and inflexible over the last 35 years to make it hard for its competitors to write software to process PDF files, quite complicated.
kolinko 5d ago
Yeah that’s why heuristics should work on the lowest possible layer, not on pdftotext. If you use pdftotext you’re stripping positional data and other stuff. Do you use a public set of documents? I bet I could almost oneshot this with my harness :p
- rayiner 5d ago
  Yes, the tool I’m describing works on what the pdf_oxide crate returns, which includes glyph positions. Here’s a public appendix from a recent Federal Circuit case. It has a representative assortment of documents (opinions, briefs, patents, transcripts) but contains only cited pages to the appendix pagination is non-consecutive: https://www.courtlistener.com/docket/68048163/15/ecofactor-inc-v-google-llc/ https://www.courtlistener.com/docket/68048163/15/ecofactor-i... The underlying docket is usually 100 to 1,000 times larger than this but it’s similar types of files and the appendix shows the various types of headers and footers that can exist.
  - kolinko 4d ago
    Here’s the oneshot, not sure if it’s slop or not though :) https://kolinko.eu/pdf-reading-order/ https://kolinko.eu/pdf-reading-order/ But I wonder about your opinion.