17 ms·

A Scholarly Markdown

177 points by droque 12y ago

2mur 12y ago
Why not asciidoc?
0942v8653 12y ago
Looks like an interesting project. I may be missing something, but I think in terms of math especially it would be hard to use Markdown instead of something more robust like LaTeX which supports proper mathematical notation. Edit: yes, I was missing something. Scholdoc supports MathJax (which has browser, Office, and LaTeX support). Small issue: The title isn't showing up in Firefox (or Chrome) because there are two title elements and the first one is blank.
JetSpiegel 12y ago
Why? This really feels like cramming a square peg into a round hole. Anytime anyone wants something more complex than simply lightly formatted text they will have to use other tool (LaTeX), so why not just use LaTeX in the first place?
- droque 12y ago
  The way I see it, the main feature is being able to plug math formulas in, rather than formatting documents. It allows you to focus on your content (one feature of Markdown) while still being able to write mathematics.
  - mturmon 12y ago
    "It allows you to focus on your content (one feature of Markdown) while still being able to write mathematics." That's what LaTeX does too. Its markup, generally, is semantic.
    - sjy 12y ago
      In my experience LaTeX is quite poor at separating content from presentation, at least when you're unwilling to rely on the default styles. All the layout commands are context-sensitive, so I often end up writing presentation code in the middle of my content, or accidentally breaking the layout by moving content around.
      - mturmon 12y ago
        You should be putting your alternate style in a .cls file and not inline. If you are super picky about format, then none of these tools are for you.
- Dewie 12y ago
  Alternatively you can use org-mode and extend the document with latex where appropriate.
_-_-_-_ 12y ago
I'm still skimming the documentation so I apologize if the answer is obvious, but does anyone know offhand why this is implemented as a fork of Pandoc? Pandoc already has extended markdown features and the creator of pandoc is very much an academic (http://johnmacfarlane.net/ http://johnmacfarlane.net/), so is there are a reason why these contributions aren't part of pandoc proper?
- kazagistar 12y ago
  This was my first thought as well. Pandoc has a rather excellent infrastructure in place to let you extend and build on it, so if you instead choose to fork, there better be a good reason...
  - aaren 12y ago
    Pandoc filters let you do a lot of things provided you stick to the defined types. If you want to add new types of element, like this does, then the only practical way is a fork. (except for really simple cases)
    - zzleeper 12y ago
      However, in most cases you could just RawBlock (or was it BlockQuote?) with specific attributes, and then intercept those in a filter and convert to Latex.
      - timtylin 12y ago
        This was basically my original approach. However, after working out the math syntax I realized that some things, like the double-backtick inline math, just can't be accomplished without a pre-filter. At that point I decided to just start playing with the parser code. I also became super-convinced that some level of AST change was necessary to keep things sane, and since I wasn't able to use the existing Math and Image types anyways (they're not attributed), I ultimately just started a new AST type package namespace called "Scholdoc". Everything just evolved from there.
      - 12y ago
stared 12y ago
This thing is certainly needed (vide this discussion: https://hackpad.com/New-scientific-markup-language-utAjFcYuvvB https://hackpad.com/New-scientific-markup-language-utAjFcYuv...), but what would be really convincing is examples (or I am missing them?). As a side note, a lot of Markdown + LaTeX + Code can be done in IPython Notebook. (Though, there are some things absent, like referencing citations or other equations).
- lorddoig 12y ago
  I read your link and, per my main comment, I'm pretty sure AsciiDoc answers 99% of your needs. http://www.methods.co.nz/asciidoc/#_overview_and_examples http://www.methods.co.nz/asciidoc/#_overview_and_examples
  - stared 12y ago
    Cool! Any examples of using AsciiDoc for math or scientific notes? (I.e. with formulae, references...)
    - lorddoig 12y ago
      Well here's some formulae: http://www.noteshare.io/section/the-fundamental-class-of-projective-space http://asciidoctor.org/docs/user-manual/#using-multiple-stem-interpreters http://www.methods.co.nz/asciidoc/latex-filter.html http://dblatex.sourceforge.net/example/dblatex/example_mathml.pdf As for references, it has it's own lightweight bibliography system out of the box, but there's a plugin[0] for BibTeX too, and DocBook has full-on support for BibTeX so it's just a matter of tooling. AsciiDoc gives you DocBook, and DocBook gives you pretty much everything.[1] The whole thing is completely extensible at multiple levels (macros, XSL stylesheets), so adding any essential features it doesn't already have is certainly much simpler than starting from scratch! [0]: https://github.com/petercrlane/asciidoc-bib [1]: http://pub.hdcrd.com/kb/Dev/Documention/LaTeX/Tool/Dblatex%20%28DocBook%20to%20LaTeX%20Publishing%29/0.3/manual.pdf
- onalark 12y ago
  Yes, but it's not ideal. Since everything is done through Javascript, we have to first identify all the math, then hide it from the Markdown parser, then parse the Markdown, then restore the math, then call MathJax on the now-HTML. Equation references/numbering exist, but are turned off by default. It's a minor change to your MathJax configuration to enable.
smilekzs 12y ago
Around 2 years ago I wrote a patch for chjj/marked that supports MathJax. Despite heated discussion, author never noticed and it wasn't merged. Just a glance at how fragmented this markdown world has become.
ciroduran 12y ago
It seems that Markdown is starting to get a lot of disgregated reference implementations. Some months ago some important Markdown users were behind CommonMark (http://commonmark.org/ http://commonmark.org/). I'd think that it would be best to join this conversation rather than splintering Markdown even more.
- rylee 12y ago
  Wow, I've never seen CommonMark -- hope that comes into common use over everything else!
- netheril96 12y ago
  I'm in fundamental disagreement with the principle behind CommonMark. They prioritize standardization over practical usability, and for that reason people will keep splintering markdown. For example, they do not plan to add syntax highlighting blocks (the ```some code``` on GitHub) to their implementation, because they believe that it is outside the scope of markdown. Then, because a lot of people actually need this feature, they still have to patch or extend or plug-in the functionality into any implementation of CommonMark, leading to fragmentation again. And frankly, the reason that I start writing more markdown is precisely because of the syntax highlighting ability. Oh, and to write scientific articles, math formulas are a deal-breaker. They want to have a standard, unambiguous syntax specification, a suite of comprehensive tests and a cleanly implemented parser. They want to unify the community of markdown users and developers. All of those are commendable goals. But at the end of the day, it doesn't satisfy my needs, so I'd rather use a messy, poorly specified markdown flavor or even just render the markdown with GitHub's service.
  - fiddlosopher 12y ago
    "For example, they do not plan to add syntax highlighting blocks (the ```some code``` on GitHub) to their implementation, because they believe that it is outside the scope of markdown." To correct the record, fenced code blocks have been there from the beginning: http://spec.commonmark.org/0.18/#fenced-code-blocks http://spec.commonmark.org/0.18/#fenced-code-blocks.
ivan_ah 12y ago
A similar effort of introducing math/figures/refs into markdown is the `softcover markdown` syntax. It's basically markdown, with latex commands allowed: https://raw.githubusercontent.com/softcover/softcover_book/master/chapters/softcover_markdown.md https://raw.githubusercontent.com/softcover/softcover_book/m... The beautiful idea in markdown is that it allows you to mix (non-container) HTML tags in with the .md and it just works. Softcover markdown is in the same vein, allowing the more readable markdown for main copy, and intermix LaTeX tags as required. Beautiful if you ask me. Or at least beautifuler than ```math ... ```math. The "backward compatibility" of ScholarlyMarkdown with basic markdown is a cool feature as many tools/plarforms exist that "support" .md now, but to preview you'll still need something that renders the equations, so strictly speaking ScholarlyMarkdown is a new markup langauge.
dllu 12y ago
I created a similar thing for my personal website: http://www.dllu.net/programming/dllup/ http://www.dllu.net/programming/dllup/ which handles math using svgtex instead of clientside MathJax for faster rendering. It also compiles to both html5 and LaTeX. However, dllup is overall less polished, missing some features (labelling equations, sections).
samatman 12y ago
Interesting to see this come up. I've been migrating a couple projects to babel, the emacs org-mode literate programming style. Starting with the hello-world of babel, my emacs config file. Babel and Org have clunky syntax but the mode takes care of that. The combination isn't perfect but it's very powerful, in particular the ability to chain several languages together in flexible ways. Coming up with more ways to combine pretty formatting with syntax highlighting is polishing a pretty smooth surface. I'm dreaming of a really slick syntax and operating environment, like babel if it wasn't tacked onto org. In the meantime, using what we have.
rhythmvs 12y ago
John MacFarlane (Pandoc’s author) is heavily committed to the Common Mark standardization effort of Markdown (in fact he’s the principal designer of the Common Mark spec). Both Common Mark and Pandoc serve different purposes: the first is an initiative to counter fragmentation/balkanization of the Markdown ecosystem and in being so has to reckon with backward compatibility, consensus and adoption. The latter is a document conversion library, which, by design, needs to reckon with interchangeability between formats and may hence be hampered by the lowest common denominator as regards feature support. Internally, Pandoc keeps an Abstract Syntax Tree (accessible in json format), and defaults on its own flavour of Pandoc Markdown (featuring a Markdown superset of content element types). Both the Common Mark community and John MacFarlane have made it clear their first and foremost focus is on standardization, not so much on extending the feature set. Yet, scholars and technical writers are in dire need for something more heavy-weight than the rather small set of features offered by Common Mark implementations or Pandoc Markdown. Hence the Scholarly Markdown initiative and the scholdoc reference implementation (Pandoc fork). More on how Scholarly Markdown came about, can be read on the blog of one of it’s pacemakers, Martin Fenner; e.g. http://blog.martinfenner.org/2013/11/17/the-grammar-of-scholarly-communication/ http://blog.martinfenner.org/2013/11/17/the-grammar-of-schol..., http://blogs.plos.org/mfenner/2012/12/18/additional-markdown-we-need-in-scholarly-texts/ http://blogs.plos.org/mfenner/2012/12/18/additional-markdown...
- fiddlosopher 12y ago
  Although I'm involved in both these enterprises, let's not confuse their goals. CommonMark is currently focused on the task of giving a decent spec for the core syntax and robust, efficient implementations; extensions will wait til that project is done, but certainly aren't ruled out. Pandoc has always been in the game of extending the feature set. Here are just some of the Markdown extensions pandoc supports: LaTeX math (which can be rendered in a variety of formats, including native Word and MathML), LaTeX macros, inline LaTeX, automatically numbered examples and cross-references to these, automatically generated citations (using CSL styles), super and subscripts, strikeout, figures, YAML metadata, definition lists, several styles of tables, fenced code blocks with syntax highlighting, header identifiers, and footnotes. scholdoc just adds a few things on top of all this (and many of them could be implemented in pandoc filters). As noted in one of the other comments on this thread, most of the features scholdoc adds are under active discussion in pandoc as well. So it's not that pandoc and scholdoc have different aims; pandoc just moves more slowly, because it has to worry about how features are implemented in many more output formats, and it operates under some other constraints that scholdoc rejects (e.g. trying to avoid the use of English words like "Figure" for syntax cues).
xiaq 12y ago
If you want to write scientific documents with markdown, beware that pandoc already supports inline TeX (math with $$ and TeX commands starting with \) when doing md -> latex conversion. That has been my setup for quite some time.
bshimmin 12y ago
I wonder how long it'll be before Gruber complains about the name.
samuell 12y ago
My spontaneous reaction is: Why don't use the existing MediaWiki syntax? It has gone through the test of years (decades?) of real-life needs, has support for (latex) formulae, everything you might need around images and references etc, one of the most powerful template systems I've ever seen, and the list goes on and on. Additionally there is a very stable, performant and flexible reference implementation implemented as a web app (mediawiki) with excellent import/export in XML format (and the list goes on ...), versioning with syntax-highlighted diff, etc etc etc. What have I missed? :)
- duskwuff 12y ago
  MediaWiki syntax is incredibly grody ('''bold''' / ''italic'', tables, template logic, etc), and has exactly one implementation in common usage (MediaWiki itself -- written in PHP, and horribly convoluted), and has no formal specification. The diff feature you're referencing as a "syntax-highlighted diff" is a simple diff of the source -- it's entirely ignorant of formatting, and the XML export is similarly just an export of the raw source. Nobody likes MediaWiki syntax, not even its own users. It's awful. The only reason it's still used at Wikipedia is because there's too much content to reasonably convert.
lorddoig 12y ago
And for everything else, there's AsciiDoc.[0] For an extra five minutes learning you get a boatload (think container ship) more features[1] - it compiles to DocBook: a mature, actually standardised, highly structured format, and from that you get HTML, EPUB, PDF, slideshows, and man pages for free.[0] For math you get MathML, ASCIIMath, and LaTeX (along with a number of ways to render them.) It has a super nice syntax, is equally good at little docs and huge books, and you could theoretically write a proper academic paper in it with the LaTeX backend. And you always know what's going to happen when you try to mix bold and italic... Also endorsed by Linus.[2] [0]: http://www.methods.co.nz/asciidoc/#_overview_and_examples [1]: Well, five minutes to be able to do everything Markdown can do; everything else will take a bit longer [2]: https://plus.google.com/+LinusTorvalds/posts/X2XVf9Q7MfV (comments)
- treerex 12y ago
  I have yet to see a good stylesheet for AsciiDoc's DocBook output for creating a decent PDF. Do you have recommendations? I love AsciiDoc, but am only using it to generate HTML right now.
  - lorddoig 12y ago
    Unfortunately not, XSL isn't really my thing. I'd probably consider trying to go via LaTeX, but that's because I like the way LaTeX looks out of the box and probably isn't appropriate for most things.
  - jauco 12y ago
    Try asciidoctor and its stylesheets.
  - boundlessdreamz 12y ago
    As @jauco said try Asciidoctor - http://asciidoctor.org/ http://asciidoctor.org/ In addition to default asciidoctor style, there are other themes available - http://themes.asciidoctor.org/preview/ http://themes.asciidoctor.org/preview/ (See bottom right for theme switcher). You can also take a look at Pro Git book styles - http://git-scm.com/book/en/v2 http://git-scm.com/book/en/v2 which was written in asciidoc - https://github.com/progit/progit2 https://github.com/progit/progit2
nikdaheratik 12y ago
Alot of interesting work, judging from the site, but this still feels like reinventing the wheel to me. In addition to AsciiDoc, there's MultiMarkdown (http://fletcherpenney.net/multimarkdown/features/ http://fletcherpenney.net/multimarkdown/features/ feature list is nearly identical), and probably another 3-4 that I haven't come across. Of course, having spent the past few months writing an app that basically reinvents this same wheel (http://www.eqeditor.com/writer/ http://www.eqeditor.com/writer/) I suppose I'm in good company.
- nikdaheratik 12y ago
  Another thing I'm curious about is how the ScholarlyMarkdown authors would go about trying to get papers formatted in this way to go into a database like PubMed but that's pretty far down the track, and not addressed by any of the other solutions either.
  - timtylin 12y ago
    The goal is eventually to be able to, like what Authorea is trying to do now, just "slap on" various LaTeX style templates during final rendering (similar to how static blog engines format plain md posts using templates). It's going to be PAINFUL but I think it is eventually doable with enough elbow grease. Markdown is a nice starting point because the features are so spartan that you actually have some hope of finding the "greatest common denominator" document model that can pretty much map injectively to all the major journal-specific LaTeX document-classes. I'm imagining some kind of online component like ShareLaTeX that will become a clearinghouse for a number of tested and proven conversion paths, and can handle compiling a ScholarlyMarkdown document to different formats. This project won't go anywhere if this can't at least be done for major houses like Elsevier, PNAS, Phys Rev, etc.
kbd 12y ago
I'm annoyed that there's still no way to underline in markdown.
j2kun 12y ago
If you want me (as an academic) to use ScholarlyMarkdown, then you can't force my collaborators to use ScholarlyMarkdown as a consequence. There should be a nice ScholarlyMarkdown -> LaTeX cross compiler for starting simple documents in TeX and then sharing with collaborators (pick some obvious defaults or allow a config file to get fancy). But more importantly, if I am joining a project that already has a bunch of LaTeX wizardry going on, I should be able to seamlessly and implicitly edit the text parts in ScholarlyMarkdown without my collaborators knowing. Can ScholarlyMarkdown do this? If not then I'm not really interested.
- zzleeper 12y ago
  My guess is that he is building the PDFs through latex/xetex. I built something similar to that, but instead of forking Pandoc (and investing the time to learn Haskell), I just added filters through Python. It's not at fast, but you can intercept mostly anything. Thus, the flow ends up as: paper.md -> Pandoc -> paper.json + metadata -> paper.tex -> paper.pdf
  - j2kun 12y ago
    I want the flow to be paper.tex -> paper.md + metadata ->(merge) paper.tex -> paper.pdf
- arnsholt 12y ago
  That's probably never going to happen (for any Markdown-based project). As you probably know, TeX is a full programming language (albeit an odd one), which means that to compile (La)TeX to Markdown, Markdown needs to support all the features of TeX, which means that it has to be Turing-complete. Not to mention the fact that parsing TeX is kind of nuts too.
  - chriswarbo 12y ago
    There's no need to convert all TeX back to markdown; the converter only needs to recognise the sub-set of TeX that it produces. If anyone edits the TeX to fall outside that sub-set, it's reasonable to just embed those parts verbatim in the resulting Markdown.
zzleeper 12y ago
This looks very interesting but I have a problem with this. I think MOST of these features could have been build just with clever uses of Pandoc filters. This means that i) it's fully compatible ii) it's very easy to extend
mitchi 12y ago
I actually like the HTML/CSS solution too. http://thomaspark.me/2015/01/pubcss-formatting-academic-publications-in-html-css/ http://thomaspark.me/2015/01/pubcss-formatting-academic-publ... It's more difficult but I feel like there's a lot more potential.
- timtylin 12y ago
  Yep, that's the idea. Published work today should be able to take advantage of modern screens and its ability to resize, reflow, and animate.
bambax 12y ago
> It is a fork of Pandoc and is build upon the same parsing engine. => built
Skywing 12y ago
Part of the beauty of markdown, imo, is its simplicity. If I ever have a question about markdown syntax, I can glance at the original docs and have the answer in less than 30 seconds.

177 points by droque 12y ago

2mur 12y ago

Why not asciidoc?

Looks like an interesting project. I may be missing something, but I think in terms of math especially it would be hard to use Markdown instead of something more robust like LaTeX which supports proper mathematical notation. Edit: yes, I was missing something. Scholdoc supports MathJax (which has browser, Office, and LaTeX support). Small issue: The title isn't showing up in Firefox (or Chrome) because there are two title elements and the first one is blank.

JetSpiegel 12y ago

Why? This really feels like cramming a square peg into a round hole. Anytime anyone wants something more complex than simply lightly formatted text they will have to use other tool (LaTeX), so why not just use LaTeX in the first place?

droque 12y ago
The way I see it, the main feature is being able to plug math formulas in, rather than formatting documents. It allows you to focus on your content (one feature of Markdown) while still being able to write mathematics.
- mturmon 12y ago
  "It allows you to focus on your content (one feature of Markdown) while still being able to write mathematics." That's what LaTeX does too. Its markup, generally, is semantic.
  - sjy 12y ago
    In my experience LaTeX is quite poor at separating content from presentation, at least when you're unwilling to rely on the default styles. All the layout commands are context-sensitive, so I often end up writing presentation code in the middle of my content, or accidentally breaking the layout by moving content around.
    - mturmon 12y ago
      You should be putting your alternate style in a .cls file and not inline. If you are super picky about format, then none of these tools are for you.
Dewie 12y ago
Alternatively you can use org-mode and extend the document with latex where appropriate.

_-_-_-_ 12y ago

I'm still skimming the documentation so I apologize if the answer is obvious, but does anyone know offhand why this is implemented as a fork of Pandoc? Pandoc already has extended markdown features and the creator of pandoc is very much an academic (http://johnmacfarlane.net/ http://johnmacfarlane.net/), so is there are a reason why these contributions aren't part of pandoc proper?

kazagistar 12y ago
This was my first thought as well. Pandoc has a rather excellent infrastructure in place to let you extend and build on it, so if you instead choose to fork, there better be a good reason...
- aaren 12y ago
  Pandoc filters let you do a lot of things provided you stick to the defined types. If you want to add new types of element, like this does, then the only practical way is a fork. (except for really simple cases)
  - zzleeper 12y ago
    However, in most cases you could just RawBlock (or was it BlockQuote?) with specific attributes, and then intercept those in a filter and convert to Latex.
    - timtylin 12y ago
      This was basically my original approach. However, after working out the math syntax I realized that some things, like the double-backtick inline math, just can't be accomplished without a pre-filter. At that point I decided to just start playing with the parser code. I also became super-convinced that some level of AST change was necessary to keep things sane, and since I wasn't able to use the existing Math and Image types anyways (they're not attributed), I ultimately just started a new AST type package namespace called "Scholdoc". Everything just evolved from there.
    - 12y ago

stared 12y ago

This thing is certainly needed (vide this discussion: https://hackpad.com/New-scientific-markup-language-utAjFcYuvvB https://hackpad.com/New-scientific-markup-language-utAjFcYuv...), but what would be really convincing is examples (or I am missing them?). As a side note, a lot of Markdown + LaTeX + Code can be done in IPython Notebook. (Though, there are some things absent, like referencing citations or other equations).

lorddoig 12y ago
I read your link and, per my main comment, I'm pretty sure AsciiDoc answers 99% of your needs. http://www.methods.co.nz/asciidoc/#_overview_and_examples http://www.methods.co.nz/asciidoc/#_overview_and_examples
- stared 12y ago
  Cool! Any examples of using AsciiDoc for math or scientific notes? (I.e. with formulae, references...)
  - lorddoig 12y ago
    Well here's some formulae: http://www.noteshare.io/section/the-fundamental-class-of-projective-space http://asciidoctor.org/docs/user-manual/#using-multiple-stem-interpreters http://www.methods.co.nz/asciidoc/latex-filter.html http://dblatex.sourceforge.net/example/dblatex/example_mathml.pdf As for references, it has it's own lightweight bibliography system out of the box, but there's a plugin[0] for BibTeX too, and DocBook has full-on support for BibTeX so it's just a matter of tooling. AsciiDoc gives you DocBook, and DocBook gives you pretty much everything.[1] The whole thing is completely extensible at multiple levels (macros, XSL stylesheets), so adding any essential features it doesn't already have is certainly much simpler than starting from scratch! [0]: https://github.com/petercrlane/asciidoc-bib [1]: http://pub.hdcrd.com/kb/Dev/Documention/LaTeX/Tool/Dblatex%20%28DocBook%20to%20LaTeX%20Publishing%29/0.3/manual.pdf
onalark 12y ago
Yes, but it's not ideal. Since everything is done through Javascript, we have to first identify all the math, then hide it from the Markdown parser, then parse the Markdown, then restore the math, then call MathJax on the now-HTML. Equation references/numbering exist, but are turned off by default. It's a minor change to your MathJax configuration to enable.

smilekzs 12y ago

Around 2 years ago I wrote a patch for chjj/marked that supports MathJax. Despite heated discussion, author never noticed and it wasn't merged. Just a glance at how fragmented this markdown world has become.

ciroduran 12y ago

It seems that Markdown is starting to get a lot of disgregated reference implementations. Some months ago some important Markdown users were behind CommonMark (http://commonmark.org/ http://commonmark.org/). I'd think that it would be best to join this conversation rather than splintering Markdown even more.

rylee 12y ago
Wow, I've never seen CommonMark -- hope that comes into common use over everything else!
netheril96 12y ago
I'm in fundamental disagreement with the principle behind CommonMark. They prioritize standardization over practical usability, and for that reason people will keep splintering markdown. For example, they do not plan to add syntax highlighting blocks (the ```some code``` on GitHub) to their implementation, because they believe that it is outside the scope of markdown. Then, because a lot of people actually need this feature, they still have to patch or extend or plug-in the functionality into any implementation of CommonMark, leading to fragmentation again. And frankly, the reason that I start writing more markdown is precisely because of the syntax highlighting ability. Oh, and to write scientific articles, math formulas are a deal-breaker. They want to have a standard, unambiguous syntax specification, a suite of comprehensive tests and a cleanly implemented parser. They want to unify the community of markdown users and developers. All of those are commendable goals. But at the end of the day, it doesn't satisfy my needs, so I'd rather use a messy, poorly specified markdown flavor or even just render the markdown with GitHub's service.
- fiddlosopher 12y ago
  "For example, they do not plan to add syntax highlighting blocks (the ```some code``` on GitHub) to their implementation, because they believe that it is outside the scope of markdown." To correct the record, fenced code blocks have been there from the beginning: http://spec.commonmark.org/0.18/#fenced-code-blocks http://spec.commonmark.org/0.18/#fenced-code-blocks.

ivan_ah 12y ago

A similar effort of introducing math/figures/refs into markdown is the `softcover markdown` syntax. It's basically markdown, with latex commands allowed: https://raw.githubusercontent.com/softcover/softcover_book/master/chapters/softcover_markdown.md https://raw.githubusercontent.com/softcover/softcover_book/m... The beautiful idea in markdown is that it allows you to mix (non-container) HTML tags in with the .md and it just works. Softcover markdown is in the same vein, allowing the more readable markdown for main copy, and intermix LaTeX tags as required. Beautiful if you ask me. Or at least beautifuler than ```math ... ```math. The "backward compatibility" of ScholarlyMarkdown with basic markdown is a cool feature as many tools/plarforms exist that "support" .md now, but to preview you'll still need something that renders the equations, so strictly speaking ScholarlyMarkdown is a new markup langauge.

dllu 12y ago

I created a similar thing for my personal website: http://www.dllu.net/programming/dllup/ http://www.dllu.net/programming/dllup/ which handles math using svgtex instead of clientside MathJax for faster rendering. It also compiles to both html5 and LaTeX. However, dllup is overall less polished, missing some features (labelling equations, sections).

samatman 12y ago

Interesting to see this come up. I've been migrating a couple projects to babel, the emacs org-mode literate programming style. Starting with the hello-world of babel, my emacs config file. Babel and Org have clunky syntax but the mode takes care of that. The combination isn't perfect but it's very powerful, in particular the ability to chain several languages together in flexible ways. Coming up with more ways to combine pretty formatting with syntax highlighting is polishing a pretty smooth surface. I'm dreaming of a really slick syntax and operating environment, like babel if it wasn't tacked onto org. In the meantime, using what we have.

rhythmvs 12y ago

John MacFarlane (Pandoc’s author) is heavily committed to the Common Mark standardization effort of Markdown (in fact he’s the principal designer of the Common Mark spec). Both Common Mark and Pandoc serve different purposes: the first is an initiative to counter fragmentation/balkanization of the Markdown ecosystem and in being so has to reckon with backward compatibility, consensus and adoption. The latter is a document conversion library, which, by design, needs to reckon with interchangeability between formats and may hence be hampered by the lowest common denominator as regards feature support. Internally, Pandoc keeps an Abstract Syntax Tree (accessible in json format), and defaults on its own flavour of Pandoc Markdown (featuring a Markdown superset of content element types). Both the Common Mark community and John MacFarlane have made it clear their first and foremost focus is on standardization, not so much on extending the feature set. Yet, scholars and technical writers are in dire need for something more heavy-weight than the rather small set of features offered by Common Mark implementations or Pandoc Markdown. Hence the Scholarly Markdown initiative and the scholdoc reference implementation (Pandoc fork). More on how Scholarly Markdown came about, can be read on the blog of one of it’s pacemakers, Martin Fenner; e.g. http://blog.martinfenner.org/2013/11/17/the-grammar-of-scholarly-communication/ http://blog.martinfenner.org/2013/11/17/the-grammar-of-schol..., http://blogs.plos.org/mfenner/2012/12/18/additional-markdown-we-need-in-scholarly-texts/ http://blogs.plos.org/mfenner/2012/12/18/additional-markdown...

fiddlosopher 12y ago
Although I'm involved in both these enterprises, let's not confuse their goals. CommonMark is currently focused on the task of giving a decent spec for the core syntax and robust, efficient implementations; extensions will wait til that project is done, but certainly aren't ruled out. Pandoc has always been in the game of extending the feature set. Here are just some of the Markdown extensions pandoc supports: LaTeX math (which can be rendered in a variety of formats, including native Word and MathML), LaTeX macros, inline LaTeX, automatically numbered examples and cross-references to these, automatically generated citations (using CSL styles), super and subscripts, strikeout, figures, YAML metadata, definition lists, several styles of tables, fenced code blocks with syntax highlighting, header identifiers, and footnotes. scholdoc just adds a few things on top of all this (and many of them could be implemented in pandoc filters). As noted in one of the other comments on this thread, most of the features scholdoc adds are under active discussion in pandoc as well. So it's not that pandoc and scholdoc have different aims; pandoc just moves more slowly, because it has to worry about how features are implemented in many more output formats, and it operates under some other constraints that scholdoc rejects (e.g. trying to avoid the use of English words like "Figure" for syntax cues).

xiaq 12y ago

If you want to write scientific documents with markdown, beware that pandoc already supports inline TeX (math with $$ and TeX commands starting with \) when doing md -> latex conversion. That has been my setup for quite some time.

bshimmin 12y ago

I wonder how long it'll be before Gruber complains about the name.

samuell 12y ago

My spontaneous reaction is: Why don't use the existing MediaWiki syntax? It has gone through the test of years (decades?) of real-life needs, has support for (latex) formulae, everything you might need around images and references etc, one of the most powerful template systems I've ever seen, and the list goes on and on. Additionally there is a very stable, performant and flexible reference implementation implemented as a web app (mediawiki) with excellent import/export in XML format (and the list goes on ...), versioning with syntax-highlighted diff, etc etc etc. What have I missed? :)

duskwuff 12y ago
MediaWiki syntax is incredibly grody ('''bold''' / ''italic'', tables, template logic, etc), and has exactly one implementation in common usage (MediaWiki itself -- written in PHP, and horribly convoluted), and has no formal specification. The diff feature you're referencing as a "syntax-highlighted diff" is a simple diff of the source -- it's entirely ignorant of formatting, and the XML export is similarly just an export of the raw source. Nobody likes MediaWiki syntax, not even its own users. It's awful. The only reason it's still used at Wikipedia is because there's too much content to reasonably convert.

lorddoig 12y ago

And for everything else, there's AsciiDoc.[0] For an extra five minutes learning you get a boatload (think container ship) more features[1] - it compiles to DocBook: a mature, actually standardised, highly structured format, and from that you get HTML, EPUB, PDF, slideshows, and man pages for free.[0] For math you get MathML, ASCIIMath, and LaTeX (along with a number of ways to render them.) It has a super nice syntax, is equally good at little docs and huge books, and you could theoretically write a proper academic paper in it with the LaTeX backend. And you always know what's going to happen when you try to mix bold and italic... Also endorsed by Linus.[2] [0]: http://www.methods.co.nz/asciidoc/#_overview_and_examples [1]: Well, five minutes to be able to do everything Markdown can do; everything else will take a bit longer [2]: https://plus.google.com/+LinusTorvalds/posts/X2XVf9Q7MfV (comments)

treerex 12y ago
I have yet to see a good stylesheet for AsciiDoc's DocBook output for creating a decent PDF. Do you have recommendations? I love AsciiDoc, but am only using it to generate HTML right now.
- lorddoig 12y ago
  Unfortunately not, XSL isn't really my thing. I'd probably consider trying to go via LaTeX, but that's because I like the way LaTeX looks out of the box and probably isn't appropriate for most things.
- jauco 12y ago
  Try asciidoctor and its stylesheets.
- boundlessdreamz 12y ago
  As @jauco said try Asciidoctor - http://asciidoctor.org/ http://asciidoctor.org/ In addition to default asciidoctor style, there are other themes available - http://themes.asciidoctor.org/preview/ http://themes.asciidoctor.org/preview/ (See bottom right for theme switcher). You can also take a look at Pro Git book styles - http://git-scm.com/book/en/v2 http://git-scm.com/book/en/v2 which was written in asciidoc - https://github.com/progit/progit2 https://github.com/progit/progit2

nikdaheratik 12y ago

Alot of interesting work, judging from the site, but this still feels like reinventing the wheel to me. In addition to AsciiDoc, there's MultiMarkdown (http://fletcherpenney.net/multimarkdown/features/ http://fletcherpenney.net/multimarkdown/features/ feature list is nearly identical), and probably another 3-4 that I haven't come across. Of course, having spent the past few months writing an app that basically reinvents this same wheel (http://www.eqeditor.com/writer/ http://www.eqeditor.com/writer/) I suppose I'm in good company.

nikdaheratik 12y ago
Another thing I'm curious about is how the ScholarlyMarkdown authors would go about trying to get papers formatted in this way to go into a database like PubMed but that's pretty far down the track, and not addressed by any of the other solutions either.
- timtylin 12y ago
  The goal is eventually to be able to, like what Authorea is trying to do now, just "slap on" various LaTeX style templates during final rendering (similar to how static blog engines format plain md posts using templates). It's going to be PAINFUL but I think it is eventually doable with enough elbow grease. Markdown is a nice starting point because the features are so spartan that you actually have some hope of finding the "greatest common denominator" document model that can pretty much map injectively to all the major journal-specific LaTeX document-classes. I'm imagining some kind of online component like ShareLaTeX that will become a clearinghouse for a number of tested and proven conversion paths, and can handle compiling a ScholarlyMarkdown document to different formats. This project won't go anywhere if this can't at least be done for major houses like Elsevier, PNAS, Phys Rev, etc.

kbd 12y ago

I'm annoyed that there's still no way to underline in markdown.

j2kun 12y ago

If you want me (as an academic) to use ScholarlyMarkdown, then you can't force my collaborators to use ScholarlyMarkdown as a consequence. There should be a nice ScholarlyMarkdown -> LaTeX cross compiler for starting simple documents in TeX and then sharing with collaborators (pick some obvious defaults or allow a config file to get fancy). But more importantly, if I am joining a project that already has a bunch of LaTeX wizardry going on, I should be able to seamlessly and implicitly edit the text parts in ScholarlyMarkdown without my collaborators knowing. Can ScholarlyMarkdown do this? If not then I'm not really interested.

zzleeper 12y ago
My guess is that he is building the PDFs through latex/xetex. I built something similar to that, but instead of forking Pandoc (and investing the time to learn Haskell), I just added filters through Python. It's not at fast, but you can intercept mostly anything. Thus, the flow ends up as: paper.md -> Pandoc -> paper.json + metadata -> paper.tex -> paper.pdf
- j2kun 12y ago
  I want the flow to be paper.tex -> paper.md + metadata ->(merge) paper.tex -> paper.pdf
arnsholt 12y ago
That's probably never going to happen (for any Markdown-based project). As you probably know, TeX is a full programming language (albeit an odd one), which means that to compile (La)TeX to Markdown, Markdown needs to support all the features of TeX, which means that it has to be Turing-complete. Not to mention the fact that parsing TeX is kind of nuts too.
- chriswarbo 12y ago
  There's no need to convert all TeX back to markdown; the converter only needs to recognise the sub-set of TeX that it produces. If anyone edits the TeX to fall outside that sub-set, it's reasonable to just embed those parts verbatim in the resulting Markdown.

zzleeper 12y ago

This looks very interesting but I have a problem with this. I think MOST of these features could have been build just with clever uses of Pandoc filters. This means that i) it's fully compatible ii) it's very easy to extend

mitchi 12y ago

I actually like the HTML/CSS solution too. http://thomaspark.me/2015/01/pubcss-formatting-academic-publications-in-html-css/ http://thomaspark.me/2015/01/pubcss-formatting-academic-publ... It's more difficult but I feel like there's a lot more potential.

timtylin 12y ago
Yep, that's the idea. Published work today should be able to take advantage of modern screens and its ability to resize, reflow, and animate.

bambax 12y ago

> It is a fork of Pandoc and is build upon the same parsing engine. => built

Skywing 12y ago

Part of the beauty of markdown, imo, is its simplicity. If I ever have a question about markdown syntax, I can glance at the original docs and have the answer in less than 30 seconds.