5 ms·

Show HN: Parsing horse racing charts with Apache PDFBox

115 points by robinhowlett 9y ago

vbuwivbiu 9y ago
what I would love is an app that would reformat portrait PDFs as 2-column landscape for reading on my screen
- mpweiher 9y ago
  Most PDF viewers I am aware of (including my own, PostView) have 2 up modes, are those not sufficient?
  - vbuwivbiu 9y ago
    that's still presenting the pages in portrait orientation. I want the pages to be landscape and for the text to flow in at least 2 columns.
    - sk5t 9y ago
      PDF is used primarily for pre-paginated media and does not reflow text; if the PDF author wanted the pages in landscape orientation, or using some other paper dimensions, he would have specified that. Same goes for margins and the like.
      - grogenaut 9y ago
        Doesn't mean the author isn't wrong for my reading situation... Or isn't doing it just cause everyone else is.
        userbinator 9y ago
        I think what the parent is trying to say is that PDF is not like HTML or text or other formats where the viewer is primarily responsible for a lot of the formatting --- in fact, a PDF page contains not much more than primitive instructions of the form "move to X, Y"; "set font to F"; "draw text 'Some text here'" (some pathological cases issue individual moves and draws for each character) --- so expecting all PDF viewers to be able to somehow "reverse-engineer" or "decompile" that set of low-level drawing instructions into more semantic entities like lines of text or even words in order to reformat the text is a little too much. Anyone who has tried selecting text from a two-column PDF page will also quickly realise the nature of the problem.
        grogenaut 9y ago
        I totally get that; however since these days a standard sheet of papers is no longer the main reading mechanism... I'm not sure it's the best layout for reading. My brother is a PhD and reads papers all day. He hates the two column format and paid for a reflowing reader. PDF is terrible on screens with different ratios than paper... Eg good computers and mobile. As I get older I use my plethora of giant screens to crank the fonts way up and sit back relaxed. PDF is terrible in that situation. Just a thought... Maybe highly formatted PDFs for paper print shouldn't be the standard anymore. Eg my original point.
      - vbuwivbiu 9y ago
        ok, that's what the author wanted, but they're wrong. We are reading our text on computers in 2017 - we should be able to choose the format that reads best for us.
JabavuAdams 9y ago
Crazy! I was just looking in to this topic a few weeks ago, for a friend. Thanks!
Cyph0n 9y ago
Firstly, what an interesting library. Secondly, this is among the best TLDR readmes I've ever seen! I lack exposure to this area, so I'm actually quite impressed with the complexity of it. Keep up the great work.
joosters 9y ago
Very interesting! I had never heard of Apache PDFBox before, I must give it a try. I have a similar program that parses horse racing PDFs from sites such as www.racehorserunner.com - which are of a much simpler format, but cause endless problems for me when the PDFs have layout problems. For example, issues like one column being too long and overlapping with another, e.g the last race on http://www.racehorserunner.com/Archives/ELP/ELP170702.pdf http://www.racehorserunner.com/Archives/ELP/ELP170702.pdf All PDF parsers that I have tried cope very badly with these kind of situations, and often try to be 'too clever' in that they value the final layout of the text over and above the individual strings. Have you experienced similar problems with PDFBox, or does it handle formatting and layout fairly reliably?
- bpicolo 9y ago
  Huh, interesting. I was looking around for PDF libs previously and PDFBox didn't show up in google results. pdftk was the only one that showed up in Google results anywhere useful. Edit: Looks like it's on the second page of results and I never made it that far, heh. Goes to show how biasing the first page of results is.
- robinhowlett 9y ago
  Yes I encountered similar issues but many of them were able to be solved. With PDFBox I was able to deal with the content at a very low level (on a per-character basis), so that when for instance building a String, I would insert a pipe character when the distance between adjacent characters was greater than the width of the space character and then detect that when translating to a certain field. See the convertToText() method for an example: https://github.com/robinhowlett/chart-parser/blob/master/src/main/java/com/robinhowlett/chartparser/charts/pdf/Chart.java#L35 https://github.com/robinhowlett/chart-parser/blob/master/src... and https://github.com/robinhowlett/chart-parser/blob/f8d651e9a19d1f2e58522047c8754d3f27165aab/src/main/java/com/robinhowlett/chartparser/charts/pdf/wagering/WagerPayoffPools.java#L522 https://github.com/robinhowlett/chart-parser/blob/f8d651e9a1... for when I used this technique
  - tcho 9y ago
    Very cool, good to see the level of control this package allows.
maxxxxx 9y ago
I still don't understand how PDF could become one of the standards for publishing documents. Well structured content gets converted into PDF which loses most of that structure. And then a lot of work is done to guess that structure from PDF and convert it back to a better file format. It just shows that successful solutions don't have to be technically good.
- cpach 9y ago
  Sadly I think that often the publishers actually want it that way, i.e. the they do not want the data to be easily parsable...
  - beager 9y ago
    I think it's more that they want consistency in rendering across devices and media.
    - maxxxxx 9y ago
      For legal documents (where PDF was used first as far as I know) this may make sense but for manuals and other documents it makes no sense at all.
- userbinator 9y ago
  The keyword is "publishing" --- as in, producing human-readable physical copies, not electronic ones. It just so happens that the format was relatively suitable for the latter too (because it actually looks like a printed document rendered on the screen --- unlike HTML or other formats around at the time), which is why that use-case became popular. PDF is basically a descendant of PostScript, which was designed to control printers. (Its PostScript origins may also explain the bizarre mix of text and binary that constitute the file format. For example, page contents are in a relatively free-form PostScript-ish RPN-like textual language, but are found in "content streams" which may be compressed or encoded into a binary format. Data "object" structures include things like '<<'-delimited dictionaries, '[' arrays ']', textual "/Names", and even provisions for comments(!?). Then there are things like the cross-reference table of all objects in the file, which is an array of fixed-width textual numbers representing file offsets, e.g. "0000001056 00000 n" refers to something 1056 bytes from the start of the file. Reactions of WTF!? from those working with the format for the first time are not uncommon.)
beager 9y ago
Very neat, and gets me curious about PDFBox, but every time I see something that converts a consistent-layout PDF back to structured data, I just bemoan the fact that this would all be trivial with an API for these kinds of things.
hbcondo714 9y ago
Impressive! Seems like you can't just use PDFBox out of the box (no pun intended) and need to write some custom code specific to the PDF itself per the chart-parser commits[1] [1] https://github.com/robinhowlett/chart-parser/tree/master/src/main/java/com/robinhowlett/chartparser https://github.com/robinhowlett/chart-parser/tree/master/src...
- robinhowlett 9y ago
  Author here; well, PDFBox is good for simple text stripping. If I wanted to print all the text on the PDF, that would be very straightforward and not much code. However, the PDF chart here is in essence a representation of structured data. I wanted to get the content in that format so that I could both serialize to JSON plus have an SDK to boot.
richiverse 9y ago
As a python programmer, I found R's pdftools to be indispensable for messy text based PDFs. I couldn't find a python lib that worked as consistently across variously different formats.
- tunaoftheland 9y ago
  I came across https://github.com/pdfminer/pdfminer.six https://github.com/pdfminer/pdfminer.six recently and was impressed with what it could get done. The documentation can be challenging to parse, so I relied on a code sample from a StackOverflow answer. Have you had a chance to try it out? Curious about how/if it works well across platforms.
0x445442 9y ago
Great job! I was just looking at collecting race information and historical results data a month or two ago and was struck by the lack of available structured data. Heck, I couldn't easily find any for pay options either.
ocrimgproc 9y ago
Can it be used for invoices?

115 points by robinhowlett 9y ago

vbuwivbiu 9y ago

what I would love is an app that would reformat portrait PDFs as 2-column landscape for reading on my screen

mpweiher 9y ago
Most PDF viewers I am aware of (including my own, PostView) have 2 up modes, are those not sufficient?
- vbuwivbiu 9y ago
  that's still presenting the pages in portrait orientation. I want the pages to be landscape and for the text to flow in at least 2 columns.
  - sk5t 9y ago
    PDF is used primarily for pre-paginated media and does not reflow text; if the PDF author wanted the pages in landscape orientation, or using some other paper dimensions, he would have specified that. Same goes for margins and the like.
    - grogenaut 9y ago
      Doesn't mean the author isn't wrong for my reading situation... Or isn't doing it just cause everyone else is.
      - userbinator 9y ago
        I think what the parent is trying to say is that PDF is not like HTML or text or other formats where the viewer is primarily responsible for a lot of the formatting --- in fact, a PDF page contains not much more than primitive instructions of the form "move to X, Y"; "set font to F"; "draw text 'Some text here'" (some pathological cases issue individual moves and draws for each character) --- so expecting all PDF viewers to be able to somehow "reverse-engineer" or "decompile" that set of low-level drawing instructions into more semantic entities like lines of text or even words in order to reformat the text is a little too much. Anyone who has tried selecting text from a two-column PDF page will also quickly realise the nature of the problem.
        grogenaut 9y ago
        I totally get that; however since these days a standard sheet of papers is no longer the main reading mechanism... I'm not sure it's the best layout for reading. My brother is a PhD and reads papers all day. He hates the two column format and paid for a reflowing reader. PDF is terrible on screens with different ratios than paper... Eg good computers and mobile. As I get older I use my plethora of giant screens to crank the fonts way up and sit back relaxed. PDF is terrible in that situation. Just a thought... Maybe highly formatted PDFs for paper print shouldn't be the standard anymore. Eg my original point.
    - vbuwivbiu 9y ago
      ok, that's what the author wanted, but they're wrong. We are reading our text on computers in 2017 - we should be able to choose the format that reads best for us.

JabavuAdams 9y ago

Crazy! I was just looking in to this topic a few weeks ago, for a friend. Thanks!

Cyph0n 9y ago

Firstly, what an interesting library. Secondly, this is among the best TLDR readmes I've ever seen! I lack exposure to this area, so I'm actually quite impressed with the complexity of it. Keep up the great work.

joosters 9y ago

Very interesting! I had never heard of Apache PDFBox before, I must give it a try. I have a similar program that parses horse racing PDFs from sites such as www.racehorserunner.com - which are of a much simpler format, but cause endless problems for me when the PDFs have layout problems. For example, issues like one column being too long and overlapping with another, e.g the last race on http://www.racehorserunner.com/Archives/ELP/ELP170702.pdf http://www.racehorserunner.com/Archives/ELP/ELP170702.pdf All PDF parsers that I have tried cope very badly with these kind of situations, and often try to be 'too clever' in that they value the final layout of the text over and above the individual strings. Have you experienced similar problems with PDFBox, or does it handle formatting and layout fairly reliably?

bpicolo 9y ago
Huh, interesting. I was looking around for PDF libs previously and PDFBox didn't show up in google results. pdftk was the only one that showed up in Google results anywhere useful. Edit: Looks like it's on the second page of results and I never made it that far, heh. Goes to show how biasing the first page of results is.
robinhowlett 9y ago
Yes I encountered similar issues but many of them were able to be solved. With PDFBox I was able to deal with the content at a very low level (on a per-character basis), so that when for instance building a String, I would insert a pipe character when the distance between adjacent characters was greater than the width of the space character and then detect that when translating to a certain field. See the convertToText() method for an example: https://github.com/robinhowlett/chart-parser/blob/master/src/main/java/com/robinhowlett/chartparser/charts/pdf/Chart.java#L35 https://github.com/robinhowlett/chart-parser/blob/master/src... and https://github.com/robinhowlett/chart-parser/blob/f8d651e9a19d1f2e58522047c8754d3f27165aab/src/main/java/com/robinhowlett/chartparser/charts/pdf/wagering/WagerPayoffPools.java#L522 https://github.com/robinhowlett/chart-parser/blob/f8d651e9a1... for when I used this technique
- tcho 9y ago
  Very cool, good to see the level of control this package allows.

maxxxxx 9y ago

I still don't understand how PDF could become one of the standards for publishing documents. Well structured content gets converted into PDF which loses most of that structure. And then a lot of work is done to guess that structure from PDF and convert it back to a better file format. It just shows that successful solutions don't have to be technically good.

cpach 9y ago
Sadly I think that often the publishers actually want it that way, i.e. the they do not want the data to be easily parsable...
- beager 9y ago
  I think it's more that they want consistency in rendering across devices and media.
  - maxxxxx 9y ago
    For legal documents (where PDF was used first as far as I know) this may make sense but for manuals and other documents it makes no sense at all.
userbinator 9y ago
The keyword is "publishing" --- as in, producing human-readable physical copies, not electronic ones. It just so happens that the format was relatively suitable for the latter too (because it actually looks like a printed document rendered on the screen --- unlike HTML or other formats around at the time), which is why that use-case became popular. PDF is basically a descendant of PostScript, which was designed to control printers. (Its PostScript origins may also explain the bizarre mix of text and binary that constitute the file format. For example, page contents are in a relatively free-form PostScript-ish RPN-like textual language, but are found in "content streams" which may be compressed or encoded into a binary format. Data "object" structures include things like '<<'-delimited dictionaries, '[' arrays ']', textual "/Names", and even provisions for comments(!?). Then there are things like the cross-reference table of all objects in the file, which is an array of fixed-width textual numbers representing file offsets, e.g. "0000001056 00000 n" refers to something 1056 bytes from the start of the file. Reactions of WTF!? from those working with the format for the first time are not uncommon.)

beager 9y ago

Very neat, and gets me curious about PDFBox, but every time I see something that converts a consistent-layout PDF back to structured data, I just bemoan the fact that this would all be trivial with an API for these kinds of things.

hbcondo714 9y ago

Impressive! Seems like you can't just use PDFBox out of the box (no pun intended) and need to write some custom code specific to the PDF itself per the chart-parser commits[1] [1] https://github.com/robinhowlett/chart-parser/tree/master/src/main/java/com/robinhowlett/chartparser https://github.com/robinhowlett/chart-parser/tree/master/src...

robinhowlett 9y ago
Author here; well, PDFBox is good for simple text stripping. If I wanted to print all the text on the PDF, that would be very straightforward and not much code. However, the PDF chart here is in essence a representation of structured data. I wanted to get the content in that format so that I could both serialize to JSON plus have an SDK to boot.

richiverse 9y ago

As a python programmer, I found R's pdftools to be indispensable for messy text based PDFs. I couldn't find a python lib that worked as consistently across variously different formats.

tunaoftheland 9y ago
I came across https://github.com/pdfminer/pdfminer.six https://github.com/pdfminer/pdfminer.six recently and was impressed with what it could get done. The documentation can be challenging to parse, so I relied on a code sample from a StackOverflow answer. Have you had a chance to try it out? Curious about how/if it works well across platforms.

0x445442 9y ago

Great job! I was just looking at collecting race information and historical results data a month or two ago and was struck by the lack of available structured data. Heck, I couldn't easily find any for pay options either.

ocrimgproc 9y ago

Can it be used for invoices?