8 ms·

Have you considered XML. TEI, for example, is very robust and mature for marking up documents.

by mgr86 1y ago

esafak 1y ago
First I heard of it. https://en.wikipedia.org/wiki/Text_Encoding_Initiative https://en.wikipedia.org/wiki/Text_Encoding_Initiative
- mgr86 1y ago
  Understandable. I work in academic publishing, and while the XML is everywhere crowd is graying, retiring, or even dying :( it still remains an excellent option for document markup. Additionally, a lot of government data produced in the US and EU make heavy use of XML technologies. I imagine they could be an interested consumer of Nanonets-OCR. TEI could be a good choice as well tested and developed conversions exist to other popular, less structured, formats.
  - jxramos 1y ago
    maybe even epub, which is xhtml
  - agoose77 1y ago
    Do check out MyST Markdown (https://mystmd.org https://mystmd.org)! Academic publishing is a space that MyST is being used, such as https://www.elementalmicroscopy.com/ https://www.elementalmicroscopy.com/ via Curvenote. (I'm a MyST contributor)
    - viraptor 1y ago
      Do you know why myst got traction, instead of RST which seems to have all the custom tagging and extensibility build in from the beginning?
      - agoose77 1y ago
        MyST Markdown (the MD flavour, not the same-named Document Engine) was inspired by ReST. It was created to address the main pain-point of ReST for incoming users (it's not Markdown!). As a project, the tooling to parse MyST Markdown was built on top of Sphinx, which primarily expects ReST as input. Now, I would not be surprised if most _new_ Sphinx users are using MyST Markdown (but I have no data there!) Subsequently, the Jupyter Book project that built those tools has pivoted to building a new document engine that's better focused on the use-cases of our audience and leaning into modern tooling.
lukev 1y ago
Yeah this really hurts. If your goal is to precisely mark up a document with some structural elements, XML is strictly superior to Markdown. The fact that someone would go to all the work to build a model to extract the structure of documents, then choose an output format strictly less expressive than XML, speaks poorly of the state of cross-generational knowledge sharing within the industry.
- prats226 1y ago
  I think the choice mainly stems from how you want to use the output. If the output is going to get fed to another LLM, then you want to select markup language where 1) the grammer would not cause too many issues with tokenization 2) which LLM has seen a lot in past 3) generates minimal number of tokens. I think markdown fits it much better compared to other markup languages. If goal is to parse this output programmatically, then I agree a more structured markup language is better choice.

Have you considered XML. TEI, for example, is very robust and mature for marking up documents.

by mgr86 1y ago

Have you considered XML. TEI, for example, is very robust and mature for marking up documents.

First I heard of it. https://en.wikipedia.org/wiki/Text_Encoding_Initiative https://en.wikipedia.org/wiki/Text_Encoding_Initiative

mgr86 1y ago
Understandable. I work in academic publishing, and while the XML is everywhere crowd is graying, retiring, or even dying :( it still remains an excellent option for document markup. Additionally, a lot of government data produced in the US and EU make heavy use of XML technologies. I imagine they could be an interested consumer of Nanonets-OCR. TEI could be a good choice as well tested and developed conversions exist to other popular, less structured, formats.
- jxramos 1y ago
  maybe even epub, which is xhtml
- agoose77 1y ago
  Do check out MyST Markdown (https://mystmd.org https://mystmd.org)! Academic publishing is a space that MyST is being used, such as https://www.elementalmicroscopy.com/ https://www.elementalmicroscopy.com/ via Curvenote. (I'm a MyST contributor)
  - viraptor 1y ago
    Do you know why myst got traction, instead of RST which seems to have all the custom tagging and extensibility build in from the beginning?
    - agoose77 1y ago
      MyST Markdown (the MD flavour, not the same-named Document Engine) was inspired by ReST. It was created to address the main pain-point of ReST for incoming users (it's not Markdown!). As a project, the tooling to parse MyST Markdown was built on top of Sphinx, which primarily expects ReST as input. Now, I would not be surprised if most _new_ Sphinx users are using MyST Markdown (but I have no data there!) Subsequently, the Jupyter Book project that built those tools has pivoted to building a new document engine that's better focused on the use-cases of our audience and leaning into modern tooling.

lukev 1y ago

Yeah this really hurts. If your goal is to precisely mark up a document with some structural elements, XML is strictly superior to Markdown. The fact that someone would go to all the work to build a model to extract the structure of documents, then choose an output format strictly less expressive than XML, speaks poorly of the state of cross-generational knowledge sharing within the industry.

prats226 1y ago
I think the choice mainly stems from how you want to use the output. If the output is going to get fed to another LLM, then you want to select markup language where 1) the grammer would not cause too many issues with tokenization 2) which LLM has seen a lot in past 3) generates minimal number of tokens. I think markdown fits it much better compared to other markup languages. If goal is to parse this output programmatically, then I agree a more structured markup language is better choice.