8 ms·

Tree-sitter grammar for org-mode

146 points by happy-dude 4y ago

happy-dude 4y ago
From the readme: > Org grammar for tree-sitter. It is not meant to implement emacs' orgmode parser, but to implement a grammar that can usefully parse org files to be used in neovim and any library that uses tree-sitter parsers. This grammar is in active development and is being used by nvim-orgmode/orgmode [1], a org-mode neovim plugin. Some additional resources some might find useful: * Org Syntax - https://orgmode.org/worg/dev/org-syntax.html https://orgmode.org/worg/dev/org-syntax.html * EBNF grammar - https://github.com/200ok-ch/org-parser/blob/master/resources/org.ebnf https://github.com/200ok-ch/org-parser/blob/master/resources... * Tree-sitter - https://tree-sitter.github.io/tree-sitter/ https://tree-sitter.github.io/tree-sitter/ [1] https://github.com/nvim-orgmode/orgmode https://github.com/nvim-orgmode/orgmode
- tgbugs 4y ago
  We're in the middle of updating the org syntax document. I've been preoccupied and haven't gotten a chance to take another pass, but there should be a new version out in the next month or so. See also [0]. 0. https://github.com/tgbugs/laundry/tree/master/laundry/grammar https://github.com/tgbugs/laundry/tree/master/laundry/gramma...
- milisims 4y ago
  I'd had the 1.0 commit sitting on my machine for a while, this post gave me the activation energy to bump it. The grammar is fairly stable now.
nonrandomstring 4y ago
Great to have a generalised parser to help Org-up (org as a more widespread markup language) to florish. I'd really like to see Gemini adopt instead/(alongside) its markup.
spicybright 4y ago
I don't care for org mode but this could be huge for people that do. I find being able to open + edit notes quickly helps my thought process more, so hopefully this can help with that.
- lazyier 4y ago
  > I find being able to open + edit notes quickly helps my thought process more, I use org with roam in Doom Emacs. The org roam notes directory is backed by git. https://www.orgroam.com/ https://www.orgroam.com/ It allows for using a Zettelkasten style approach to note taking inside of Emacs. The basic idea is small quick notes that are cross referenced, indexed, and searchable. Doom Emacs is like Spacemacs were access to commands are activated through hitting the space bar while in Evil's command mode. This way if you stumble across a link or something notable or get an idea you want to hit later it takes about 15-20 seconds to generate a new note card, link it to other ones, and then get back to whatever you were doing before you got distracted. And then later you can go back and refine and expand on the idea with more note cards and have something you can easily use for a future reference. Nvim has it's own Zettelkasten-style note taking plugins, of course. But what sets Org apart, IMO, is just how flexible and FAST it is as a text-based UI for writing documents. Makes using MarkDown seem plodding in comparison. My favorite is, though, reStructuredText. RST was a work of genius. But the clincher for Org, when compared to RST, ends up being Org-babel. The ability to embed multiple formats into a single document, be able to quickly render those other documents to separate files, and being able to execute source code and maintain state between multiple src segments is crazy effective. Completely awesome if you want to get into literate programming. This turns your personal text-based notebook into having capabilities akin to Wolfwam or Jupyter Notebooks. Can even turn your indexed notes into an interface for Jupyter or Wolfram Mathmatica if you really want to. It's pretty nuts. Having something like that for NeoVim would be extremely nice. With new versions of Emacs having Wayland and Native Compilation it has improved performance significantly, but I don't think it will ever approach to just how fast NeoVim can be.
  - bananamerica 4y ago
    > Having something like that for NeoVim would be extremely nice. With new versions of Emacs having Wayland and Native Compilation it has improved performance significantly, but I don't think it will ever approach to just how fast NeoVim can be. Always good to remind that emacsclient makes starting an Emacs frame/window instantaneous.
tconfrey 4y ago
This is very exciting. IMO the Tools For Thought (#TFT) ecosystem is crying out for a standard interchange format to break down the silos. Org-mode could be that format. It's not just an outliner format, it's the best markup[1] and supports pretty much everything you need for journaling/task-management/project-management etc. What's been missing thus far is a single canonical definition that everyone can build on (beyond the org-mode source code). I'd love to see this be the reference implementation of Karl Voits Orgdown proposal [2]. (BTW there was a recent relevant discussion here [3]) [1] https://karl-voit.at/2017/09/23/orgmode-as-markup-only https://karl-voit.at/2017/09/23/orgmode-as-markup-only [2] https://gitlab.com/publicvoit/orgdown https://gitlab.com/publicvoit/orgdown [3] https://news.ycombinator.com/item?id=30879327 https://news.ycombinator.com/item?id=30879327
- preek 4y ago
  All good points. As a daily user of Org mode, I couldn’t agree more. Hence, we are working on building a parser that also acts as a canonical definition here: https://github.com/200ok-ch/org-parser https://github.com/200ok-ch/org-parser
  - tconfrey 4y ago
    Thanks @preek. I mentioned you guys in [3] above. BTW I'm actually using this parser: https://github.com/orgapp/orgajs https://github.com/orgapp/orgajs for my product (https://braintool.org https://braintool.org), so there are other choices. I guess the key thing is a single well defined grammar. Is GDrive syncing working in Organice these days? I've wanted to demonstrate interop with BrainTool (which syncs to GDrive files) but last I checked there was some bug.
- scottrblock 4y ago
  I agree! I'm the creator of phonetonote [1], which tries to integrate with different tools for thought. I just started a `tft-interop` repo [2] with similar goals. We're just getting started thinking through how to make the TFT structures (edn mostly) more interoperable. I'm not personally an org-mode user, but agree it could be a good format to parse these trees in and out of. I've mostly been thinking about OPML so far. Feel free to hop into the discussions where we're sorting through these ideas — https://github.com/phonetonote/tft-interop/discussions/2 https://github.com/phonetonote/tft-interop/discussions/2 [1] https://phonetonote.com https://phonetonote.com [2] https://github.com/phonetonote/tft-interop/ https://github.com/phonetonote/tft-interop/
sfink 4y ago
Offtopic, but I use org mode for my notes files (both general and technical), and I've never really figured out how to use it effectively. My pre-org notes file was just a giant text file with entries delimited by manually-entered '--------------' lines. I stuck a topic in each entry for searchability and then the rest was freeform text. It worked pretty well. org mode gives me more power, but it's getting a little unmanageable. I find myself expanding and collapsing sections a lot, especially when just trying to find the right place to add a new entry. (I'm using a hierarchy rather than the flat append-only list.) The file is pretty huge at this point, and I split it up once but that broke my most common usage of just isearching through the entire file, and it was annoying to have to load multiple files to do a complete search. I had links between files, but that didn't help for searching. I imagine there are tips and tricks to resolve all of my issues and nuisances, but I've never taken the time to figure them out. Is there a good workflow description somewhere that I should be looking at? It's hard to beat a basic text file, but after a decade or so it has gotten pretty big and I'd like to eg be able to cluster related things together so I can purge obsolete stuff. Maybe my problem is that org mode is intended for organizing things, and my use case is too free-form?
- tconfrey 4y ago
  Scaling your system is always going to be a problem, no matter what tools you use. I hear you on isearching. To support multiple files you could look at ag.el for fast cross-file search (although not incremental like isearch). Alternately you can use :tags: to add a different dimension of information by which things can be grouped. org-agenda can then be used to see items by tag (also since they are just text you can search them in a buffer). I use org-agenda in 'F'ollow mode whereby a second buffer shows the selected agenda item in its original context. It helps visualize where things are while navigating. The new hotness, ala org-roam, is bidirectional linking with tiny files. That would probably be a pain to migrate to, and isn't really aligned with your free-form model. But it might be the way to scale over the long term.
  - sfink 4y ago
    That sounds promising. You understood perfectly where I am and what I'm looking for, thanks. I'll check out ag.el. The incremental search is not that big a deal, I'm usually searching for a keyword anyway. I looked briefly at org-roam, and I agree with your assessment. It looks like it adds friction to the part that needs to be as smooth as possible for me: adding notes. I could probably dummy something up for searching by making a command that shells out to grep or whatever, but that feels like fighting against the data model.

146 points by happy-dude 4y ago

From the readme: > Org grammar for tree-sitter. It is not meant to implement emacs' orgmode parser, but to implement a grammar that can usefully parse org files to be used in neovim and any library that uses tree-sitter parsers. This grammar is in active development and is being used by nvim-orgmode/orgmode [1], a org-mode neovim plugin. Some additional resources some might find useful: * Org Syntax - https://orgmode.org/worg/dev/org-syntax.html https://orgmode.org/worg/dev/org-syntax.html * EBNF grammar - https://github.com/200ok-ch/org-parser/blob/master/resources/org.ebnf https://github.com/200ok-ch/org-parser/blob/master/resources... * Tree-sitter - https://tree-sitter.github.io/tree-sitter/ https://tree-sitter.github.io/tree-sitter/ [1] https://github.com/nvim-orgmode/orgmode https://github.com/nvim-orgmode/orgmode

tgbugs 4y ago
We're in the middle of updating the org syntax document. I've been preoccupied and haven't gotten a chance to take another pass, but there should be a new version out in the next month or so. See also [0]. 0. https://github.com/tgbugs/laundry/tree/master/laundry/grammar https://github.com/tgbugs/laundry/tree/master/laundry/gramma...
milisims 4y ago
I'd had the 1.0 commit sitting on my machine for a while, this post gave me the activation energy to bump it. The grammar is fairly stable now.

nonrandomstring 4y ago

Great to have a generalised parser to help Org-up (org as a more widespread markup language) to florish. I'd really like to see Gemini adopt instead/(alongside) its markup.

spicybright 4y ago

I don't care for org mode but this could be huge for people that do. I find being able to open + edit notes quickly helps my thought process more, so hopefully this can help with that.

lazyier 4y ago
> I find being able to open + edit notes quickly helps my thought process more, I use org with roam in Doom Emacs. The org roam notes directory is backed by git. https://www.orgroam.com/ https://www.orgroam.com/ It allows for using a Zettelkasten style approach to note taking inside of Emacs. The basic idea is small quick notes that are cross referenced, indexed, and searchable. Doom Emacs is like Spacemacs were access to commands are activated through hitting the space bar while in Evil's command mode. This way if you stumble across a link or something notable or get an idea you want to hit later it takes about 15-20 seconds to generate a new note card, link it to other ones, and then get back to whatever you were doing before you got distracted. And then later you can go back and refine and expand on the idea with more note cards and have something you can easily use for a future reference. Nvim has it's own Zettelkasten-style note taking plugins, of course. But what sets Org apart, IMO, is just how flexible and FAST it is as a text-based UI for writing documents. Makes using MarkDown seem plodding in comparison. My favorite is, though, reStructuredText. RST was a work of genius. But the clincher for Org, when compared to RST, ends up being Org-babel. The ability to embed multiple formats into a single document, be able to quickly render those other documents to separate files, and being able to execute source code and maintain state between multiple src segments is crazy effective. Completely awesome if you want to get into literate programming. This turns your personal text-based notebook into having capabilities akin to Wolfwam or Jupyter Notebooks. Can even turn your indexed notes into an interface for Jupyter or Wolfram Mathmatica if you really want to. It's pretty nuts. Having something like that for NeoVim would be extremely nice. With new versions of Emacs having Wayland and Native Compilation it has improved performance significantly, but I don't think it will ever approach to just how fast NeoVim can be.
- bananamerica 4y ago
  > Having something like that for NeoVim would be extremely nice. With new versions of Emacs having Wayland and Native Compilation it has improved performance significantly, but I don't think it will ever approach to just how fast NeoVim can be. Always good to remind that emacsclient makes starting an Emacs frame/window instantaneous.

tconfrey 4y ago

This is very exciting. IMO the Tools For Thought (#TFT) ecosystem is crying out for a standard interchange format to break down the silos. Org-mode could be that format. It's not just an outliner format, it's the best markup[1] and supports pretty much everything you need for journaling/task-management/project-management etc. What's been missing thus far is a single canonical definition that everyone can build on (beyond the org-mode source code). I'd love to see this be the reference implementation of Karl Voits Orgdown proposal [2]. (BTW there was a recent relevant discussion here [3]) [1] https://karl-voit.at/2017/09/23/orgmode-as-markup-only https://karl-voit.at/2017/09/23/orgmode-as-markup-only [2] https://gitlab.com/publicvoit/orgdown https://gitlab.com/publicvoit/orgdown [3] https://news.ycombinator.com/item?id=30879327 https://news.ycombinator.com/item?id=30879327

preek 4y ago
All good points. As a daily user of Org mode, I couldn’t agree more. Hence, we are working on building a parser that also acts as a canonical definition here: https://github.com/200ok-ch/org-parser https://github.com/200ok-ch/org-parser
- tconfrey 4y ago
  Thanks @preek. I mentioned you guys in [3] above. BTW I'm actually using this parser: https://github.com/orgapp/orgajs https://github.com/orgapp/orgajs for my product (https://braintool.org https://braintool.org), so there are other choices. I guess the key thing is a single well defined grammar. Is GDrive syncing working in Organice these days? I've wanted to demonstrate interop with BrainTool (which syncs to GDrive files) but last I checked there was some bug.
scottrblock 4y ago
I agree! I'm the creator of phonetonote [1], which tries to integrate with different tools for thought. I just started a `tft-interop` repo [2] with similar goals. We're just getting started thinking through how to make the TFT structures (edn mostly) more interoperable. I'm not personally an org-mode user, but agree it could be a good format to parse these trees in and out of. I've mostly been thinking about OPML so far. Feel free to hop into the discussions where we're sorting through these ideas — https://github.com/phonetonote/tft-interop/discussions/2 https://github.com/phonetonote/tft-interop/discussions/2 [1] https://phonetonote.com https://phonetonote.com [2] https://github.com/phonetonote/tft-interop/ https://github.com/phonetonote/tft-interop/

sfink 4y ago

Offtopic, but I use org mode for my notes files (both general and technical), and I've never really figured out how to use it effectively. My pre-org notes file was just a giant text file with entries delimited by manually-entered '--------------' lines. I stuck a topic in each entry for searchability and then the rest was freeform text. It worked pretty well. org mode gives me more power, but it's getting a little unmanageable. I find myself expanding and collapsing sections a lot, especially when just trying to find the right place to add a new entry. (I'm using a hierarchy rather than the flat append-only list.) The file is pretty huge at this point, and I split it up once but that broke my most common usage of just isearching through the entire file, and it was annoying to have to load multiple files to do a complete search. I had links between files, but that didn't help for searching. I imagine there are tips and tricks to resolve all of my issues and nuisances, but I've never taken the time to figure them out. Is there a good workflow description somewhere that I should be looking at? It's hard to beat a basic text file, but after a decade or so it has gotten pretty big and I'd like to eg be able to cluster related things together so I can purge obsolete stuff. Maybe my problem is that org mode is intended for organizing things, and my use case is too free-form?

tconfrey 4y ago
Scaling your system is always going to be a problem, no matter what tools you use. I hear you on isearching. To support multiple files you could look at ag.el for fast cross-file search (although not incremental like isearch). Alternately you can use :tags: to add a different dimension of information by which things can be grouped. org-agenda can then be used to see items by tag (also since they are just text you can search them in a buffer). I use org-agenda in 'F'ollow mode whereby a second buffer shows the selected agenda item in its original context. It helps visualize where things are while navigating. The new hotness, ala org-roam, is bidirectional linking with tiny files. That would probably be a pain to migrate to, and isn't really aligned with your free-form model. But it might be the way to scale over the long term.
- sfink 4y ago
  That sounds promising. You understood perfectly where I am and what I'm looking for, thanks. I'll check out ag.el. The incremental search is not that big a deal, I'm usually searching for a keyword anyway. I looked briefly at org-roam, and I agree with your assessment. It looks like it adds friction to the part that needs to be as smooth as possible for me: adding notes. I could probably dummy something up for searching by making a command that shells out to grep or whatever, but that feels like fighting against the data model.