7 ms·

What's so hard about PDF text extraction?

11 points by devy 4y ago

dirtbag__dad 4y ago
This is a humbling read. Rn my company is trying to figure this out for PDFs in our domain. Any recommendations for saas solutions or libraries that we ought to explore? (( if your answer is no because this is part of your secret sauce, I totally understand ))
- wizofaus 4y ago
  I've done a fair bit of it in the past - the issues with missing spaces or substitute characters used for spaces certainly rings a bell. I know quite a few libraries in the .NET space for processing PDFs, including at least one open-source one (PDFSharp from memory) which we had in our source code repo, and made up something like 75% of the total LOC for our product, despite PDF processing being a relatively minor feature). We ended up abandoning that though, can't quite remember why. The real question is how did such a torturous (once proprietary) format manage to become the only ubiquitous standard for paginated, self-contained, formatted text suitable for printing. In principle HTML could actually do the job, but arguably it's not the ideal tool either, and I've not seen an example of an HTML document that just prints as expected across multiple pages either. Odd that this got posted almost the same time as https://news.ycombinator.com/item?id=33145498 https://news.ycombinator.com/item?id=33145498 btw...
  - devy 4y ago
    Yes, I saw the other submission and post this (as I read it a few months back). It's a hard enough domain area that warrants a startup to solve the pain point.
  - nuc1e0n 4y ago
    Pdf is meant for document reproducability. The fact it's glyph soup impedes tampering, which could be considered a feature by those who don't want their documents tampered with. i.e. to be presented to others exactly how they were when created. If you want an editable document, use a word document instead. HTML does have page-break-before and page-break-after css properties, but css could do with a "scale element to fit page", "only this element on a page" and "center on page" css properties. Even the print specific css properties that do exist currently are not well supported by chrome. Google really needs to fix that situation IMHO.
    - wizofaus 4y ago
      It's a pretty terrible way to prevent tampering though - unless it has a digital signature (and all the necessary proof of signature authenticity via certificate authorities etc. etc.). Yet your average punter is far more likely to believe a PDF hasn't been tampered with than even a webpage hosted on a TLS-protected website. Word documents are not just proprietary* but for a high percentage of users by default open up in an editor designed precisely for "tampering", and certainly not for reading. There's definitely a need for a document format that serves the purpose PDF does now, I just wish PDF wasn't that format. * sure, docx may be a form of OpenXML but it's still largely based around MS Word's proprietary behaviour - very few other tools can reliably display all docx files well.
      - nuc1e0n 4y ago
        pdfs should certainly have more robust means to prevent tampering I agree. pdf integrity assurances aren't much better than ROT13 is for cryptography.
        gettalong 4y ago
        This is what PDF digital signatures are for. They allow you to make sure that the PDF hasn't been tempered with. However, that is not there original intention. From what I remember the PDF standards groups are currently investigating adding something just for temper detection.
        wizofaus 4y ago
        Being able to place a digital signature on PDF as proof that you've read it and, for contracts, e.g. as a way of indicating your acceptance of its terms etc. would be far better than the rather absurd method used now where you can just type in your name in some fancy cursive looking font and send it back as supposed proof of your acknowledgement.
        gettalong 4y ago
  - gettalong 4y ago
    If the PDF creation software supports tagged PDF, text extraction gets much easier because all the text and structural information is preserved. This also allows "reflowing" of contents, similar to how a HTML page reflows on smaller screens.
trinovantes 4y ago
Stuffing keywords outside the page's bounding box seem like a good way to bypass (dumb) resume scanners
simonblack 4y ago
I find that most PDFs extract paragraphs out of sequence. It's so annoying. My most effective (so far) extractive program is 'pdftotext' and using the '-layout' option: pdftotext -layout my_file.pdf
nuc1e0n 4y ago
Extracting text from a pdf is akin to using text-to-speech software to produce a .wav file then expecting to still be able to edit the original text when only having the .wav file. Anyone wanting to extract text from a pdf would do better to hand it to a pool of typists for them to retype into a new document. Or better yet, edit the original document that the pdf was generated from.

11 points by devy 4y ago

This is a humbling read. Rn my company is trying to figure this out for PDFs in our domain. Any recommendations for saas solutions or libraries that we ought to explore? (( if your answer is no because this is part of your secret sauce, I totally understand ))

wizofaus 4y ago
I've done a fair bit of it in the past - the issues with missing spaces or substitute characters used for spaces certainly rings a bell. I know quite a few libraries in the .NET space for processing PDFs, including at least one open-source one (PDFSharp from memory) which we had in our source code repo, and made up something like 75% of the total LOC for our product, despite PDF processing being a relatively minor feature). We ended up abandoning that though, can't quite remember why. The real question is how did such a torturous (once proprietary) format manage to become the only ubiquitous standard for paginated, self-contained, formatted text suitable for printing. In principle HTML could actually do the job, but arguably it's not the ideal tool either, and I've not seen an example of an HTML document that just prints as expected across multiple pages either. Odd that this got posted almost the same time as https://news.ycombinator.com/item?id=33145498 https://news.ycombinator.com/item?id=33145498 btw...
- devy 4y ago
  Yes, I saw the other submission and post this (as I read it a few months back). It's a hard enough domain area that warrants a startup to solve the pain point.
- nuc1e0n 4y ago
  Pdf is meant for document reproducability. The fact it's glyph soup impedes tampering, which could be considered a feature by those who don't want their documents tampered with. i.e. to be presented to others exactly how they were when created. If you want an editable document, use a word document instead. HTML does have page-break-before and page-break-after css properties, but css could do with a "scale element to fit page", "only this element on a page" and "center on page" css properties. Even the print specific css properties that do exist currently are not well supported by chrome. Google really needs to fix that situation IMHO.
  - wizofaus 4y ago
    It's a pretty terrible way to prevent tampering though - unless it has a digital signature (and all the necessary proof of signature authenticity via certificate authorities etc. etc.). Yet your average punter is far more likely to believe a PDF hasn't been tampered with than even a webpage hosted on a TLS-protected website. Word documents are not just proprietary* but for a high percentage of users by default open up in an editor designed precisely for "tampering", and certainly not for reading. There's definitely a need for a document format that serves the purpose PDF does now, I just wish PDF wasn't that format. * sure, docx may be a form of OpenXML but it's still largely based around MS Word's proprietary behaviour - very few other tools can reliably display all docx files well.
    - nuc1e0n 4y ago
      pdfs should certainly have more robust means to prevent tampering I agree. pdf integrity assurances aren't much better than ROT13 is for cryptography.
      - gettalong 4y ago
        This is what PDF digital signatures are for. They allow you to make sure that the PDF hasn't been tempered with. However, that is not there original intention. From what I remember the PDF standards groups are currently investigating adding something just for temper detection.
        wizofaus 4y ago
        Being able to place a digital signature on PDF as proof that you've read it and, for contracts, e.g. as a way of indicating your acceptance of its terms etc. would be far better than the rather absurd method used now where you can just type in your name in some fancy cursive looking font and send it back as supposed proof of your acknowledgement.
        gettalong 4y ago
- gettalong 4y ago
  If the PDF creation software supports tagged PDF, text extraction gets much easier because all the text and structural information is preserved. This also allows "reflowing" of contents, similar to how a HTML page reflows on smaller screens.

trinovantes 4y ago

Stuffing keywords outside the page's bounding box seem like a good way to bypass (dumb) resume scanners

simonblack 4y ago

I find that most PDFs extract paragraphs out of sequence. It's so annoying. My most effective (so far) extractive program is 'pdftotext' and using the '-layout' option: pdftotext -layout my_file.pdf

nuc1e0n 4y ago

Extracting text from a pdf is akin to using text-to-speech software to produce a .wav file then expecting to still be able to edit the original text when only having the .wav file. Anyone wanting to extract text from a pdf would do better to hand it to a pool of typists for them to retype into a new document. Or better yet, edit the original document that the pdf was generated from.