10 ms·

EDIT: Release notes: https://tug.org/texlive/doc/texlive-en/texlive-en.html# https://tug.org/texlive/doc/texlive-en/texlive-en.html# One technology that has st

by fxfan 7y ago

EDIT: Release notes: https://tug.org/texlive/doc/texlive-en/texlive-en.html# https://tug.org/texlive/doc/texlive-en/texlive-en.html# One technology that has stood the test of time- amazingly well designed TeX by Donald Knuth and LaTeX by Leslie Lamport. Plain TeX can be a bit beginner unfriendly because most beginners use LaTeX and experienced people have their own 'coding style' but it's amazingly powerful. I only wish somebody would add the minimum required primitives to plain TeX to render reflowable text in browsers and ebook readers TeX was designed for "beautiful" typesetting but it gives you much more control than that IMHO. I'd urge everyone to try it out once, I use it offline but I think overleaf.com allows for plain TeX too. (XeTeX may work best for unicode, it's plain tex + minimal additions for Unicode) On that note- I am not fond of PDFs because of their awfully poor unicode search support- does anybody knowledgeable know of a good target format I should use (and the appropriate drivers?)

bfirsh 7y ago
You might like this LaTeX to HTML converter I'm working on: https://github.com/arxiv-vanity/engrafo https://github.com/arxiv-vanity/engrafo What primitives do you think need adding to TeX? LaTeXML, which powers Engrafo, does a pretty good job of converting plain TeX, as well as LaTeX.
- fxfan 7y ago
  This is super cool, thank you for sharing!
- tadfisher 7y ago
  Ironically, Computer Modern is terrible to read on computer displays. It was designed for ink and toner bleed, and looks great when printed on 1970s-era Xerox printers [1], but it's far too thin for digital rendering. I might suggest Bitstream Charter for a well-designed and readable analogue to CMR for digital use. [1] https://www.typografie.info/3/topic/22238-ist-die-computer-modern-wirklich-zu-d%C3%BCnn/ https://www.typografie.info/3/topic/22238-ist-die-computer-m... (German language)
  - timClicks 7y ago
    A very good font family for screens that's openly licensed is Inter[2]. It's well designed and also quite versatile. [2] https://rsms.me/inter/ https://rsms.me/inter/
  - khaled 7y ago
    That is the Type 1 realization of it, the original Metafont outputs bitmaps tailored for the target device. Unfortunately no widely font rendering library supports Metafont directly, so they are often converted to Type 1 or OpenType and loose that capability.
- nanna 7y ago
  Out of interest, what are the advantages of LaTeXML over Pandoc's LaTeX to XML/HTML conversion? Why did you choose LaTeXML?
  - pletnes 7y ago
    Last time I checked, pandoc’s LaTeX parser did not support too much of the TeX syntax. It only works for some subset of LaTeX, basically
_emacsomancer_ 7y ago
Doesn't Xe(La)TeX/Lua(La)TeX produce PDFs with decent unicode search support? What issues do you have?
- fxfan 7y ago
  XeTeX in particular does but my problem is with the uncertainty around the whole process and the frustration when a search fails on a 300 page document
JulianMorrison 7y ago
TeX is fundamentally not compatible with reflowable text because it's at the lowest level about putting glyphs in positions.
- sytelus 7y ago
  I fail to see any fundamental incompatibility. Every text renderer of any kind is about putting glyphs in positions. The layout would declare few things not moveable and algorithms needs to decide how to fit text around those constraints. The algorithms for TeX are computationally expensive but I don't see why with faster computers, you can't reflow the text. About 15 years ago it used to take dozen second to compile my typical PDF, now overleaf.com does that in just a second in browser.
  - macleginn 7y ago
    Compilation seems to take place on the server.
  - chongli 7y ago
    It's still way too slow. Lots of the documents I've worked on recently have taken over ten seconds to compile on my 2017 MacBook Pro (touch bar). That's 3 orders of magnitude too slow for reflowing at 60hz when you resize a window. I doubt a laptop will ever be 3 orders of magnitude faster (in sequential execution) than the one I have now, radical post-semiconductor computers notwithstanding.
  - JulianMorrison 7y ago
    If you are outputting a layout for a reflowable medium like HTML or EPUB, you are not putting glyphs in positions. You are constructing graphical objects and defining their relationships (and how those relationships change based on form factor), and you are permitting the output device to render glyphs and put them in positions. This is why we don't use PDF for web pages.
    - jhpriestley 7y ago
      The web browser also "puts glyphs into positions." Neither HTML nor TeX source specifies positioning in the source format though. The same TeX document can be re-rendered at different page sizes, font sizes etc. Really not sure what fundamental difference you are seeing.
mwfunk 7y ago
Unicode search in PDF works fine, there are just a few issues that can cripple it for specific files, or specific files using specific PDF implementations. The following is advice for anyone working with the format, it should not be construed as making excuses for how PDF does things. It's a ginormous format with a ginormous spec that's been around for a very long time and has accumulated a lot of baroque qualities over the years. So I don't mean to condemn it, but I'm not exonerating it either. It is what it is, and if you have to work with it this might be useful. Anyhoo: Regardless of what one thinks of their other qualities, if a PDF can be searched in Adobe Reader or Acrobat, then the file is probably OK but the PDF reader that you were trying to search it with has a bug on its end (or more likely, some unimplemented dark corner of the PDF spec). On the other hand, if the file isn't searchable via Reader/Acrobat, then the problem is most likely with the authoring of the file itself. The most common thing that breaks searching is when instead of embedding all fonts used, a PDF refers to fonts by name from the local system. This can cause unpredictable issues when reading the PDF from another OS that can't resolve those font names. Another common breaker of search that seems to be much more common with TeX are workflows which somehow produce PDFs that have embedded Type 3 fonts. Type 3 fonts represent glyphs using PDF drawing instructions. It's less a file format than something that only exists as an embedded font within a PDF, but I've only seen them in the wild when authored by pdflatex or similar. It seems like TrueType or OpenType are the most reliable formats to embed across the most commonly used PDF implementations, but that's an educated guess. Type 3 font support is spotty in non-Adobe implementations. Finally, font subsetting might screw up search. Most software that knows how to produce PDFs can produce PDFs with embedded but subsetted fonts. This means the software that created the file embedded TrueType fonts (for example), but created a special version of the font for embedding that only contained the glyphs used in the PDF. Depending on the quality of the software used to do the font subsetting, the output PDF might not retain the mapping of glyphs back to Unicode characters. If that happens, text using that font in the PDF becomes unsearchable, but the PDF remains renderable. None of this is meant to excuse the shortcomings of the format, it's clearly overcomplicated and fragile. But when I've seen problems with searchability of PDFs, it's most often been with the software that was used to create the files, or how that software was configured by the author. And when it is a problem with the authored file itself, it's almost always because of fonts. Either the fonts aren't embedded, or they're embedded in a slightly oddball format that's in spec for PDF but not perfectly, universally supported by all of the various non-Adobe PDF implementations.
Liskni_si 7y ago
I haven't tried this myself but I think ConTeXt can output PDFs with embedded XML markup and even epub files. You need to use \startsection and \stopsection instead of just \section, and maybe there are other limitations, but it's a small price to pay, isn't it? https://wiki.contextgarden.net/Epub https://wiki.contextgarden.net/Epub https://wiki.contextgarden.net/Export https://wiki.contextgarden.net/Export
Veen 7y ago
The problem is that browsers choose not to implement high-quality justification algorithms like the Knuth-Plass algorithm that Tex uses because it is computationally intensive. That’s why justified text looks like garbage on the web. There are some experimental JavaScript implementations, but without browser support reflowing high-quality justified text is a non-starter on the web.
- benschulz 7y ago
  My assumption was also that performance is the reason we aren't getting more esthetically pleasing line breaking. Until I read a comment[1] by Philip Walton, who works on WebRender at Mozilla, that is. [1]: https://news.ycombinator.com/item?id=19473277 https://news.ycombinator.com/item?id=19473277
  - Veen 7y ago
    I'm not a web developer, so take this with a huge pinch of salt, but, if floats are the problem, does that imply that with layouts that use CSS Grid or Flexbox, we could have a decent justification algorithm?
  - jimhefferon 7y ago
    Oh, thanks for this. Very intertesting.
  - jfk13 7y ago
    He's Patrick, not Philip.
    - benschulz 7y ago
      Woups. Thanks for the correction. I think it's too late to edit my original post. :/
  - wahern 7y ago
    See also the 8-year-old Mozilla bug for a more detailed discussion: https://bugzilla.mozilla.org/show_bug.cgi?id=630181 https://bugzilla.mozilla.org/show_bug.cgi?id=630181
pjmlp 7y ago
Kind of. I was deep into LaTeX during the 90's while at the university, my Lamport's book copy state reflects how much I used to refer back to it. Nowadays I rather prefer the convenience of something like FrameMaker.
jimjimjim 7y ago
? PDF works fine for searching. It depends on the document, a pdf page could be: 1) Text 2) An image of text 3) Lines/Curves that happen to be in the shape of letters/text If it's text it perfectly searchable. If it's one of the others the creator has to also OCR it and add the text behind the image or in front but invisible (no stroke or fill).
- syockit 7y ago
  It could also be mostly text, but mixed with ligatures having their own codepoints (sometimes non-Unicode).
  - jfk13 7y ago
    If you set the \XeTeXgenerateactualtext=1 option in a Xe(La)TeX document, the resulting PDF will include ActualText annotations to support searching even in complex scripts with ligatures, character reordering, etc.
gammarator 7y ago
Thought about updating, but the actual changelog is pretty underwhelming for a 5+GB download... https://tug.org/texlive/doc/texlive-en/texlive-en.html#news https://tug.org/texlive/doc/texlive-en/texlive-en.html#news

by fxfan 7y ago

bfirsh 7y ago

You might like this LaTeX to HTML converter I'm working on: https://github.com/arxiv-vanity/engrafo https://github.com/arxiv-vanity/engrafo What primitives do you think need adding to TeX? LaTeXML, which powers Engrafo, does a pretty good job of converting plain TeX, as well as LaTeX.

fxfan 7y ago
This is super cool, thank you for sharing!
tadfisher 7y ago
Ironically, Computer Modern is terrible to read on computer displays. It was designed for ink and toner bleed, and looks great when printed on 1970s-era Xerox printers [1], but it's far too thin for digital rendering. I might suggest Bitstream Charter for a well-designed and readable analogue to CMR for digital use. [1] https://www.typografie.info/3/topic/22238-ist-die-computer-modern-wirklich-zu-d%C3%BCnn/ https://www.typografie.info/3/topic/22238-ist-die-computer-m... (German language)
- timClicks 7y ago
  A very good font family for screens that's openly licensed is Inter[2]. It's well designed and also quite versatile. [2] https://rsms.me/inter/ https://rsms.me/inter/
- khaled 7y ago
  That is the Type 1 realization of it, the original Metafont outputs bitmaps tailored for the target device. Unfortunately no widely font rendering library supports Metafont directly, so they are often converted to Type 1 or OpenType and loose that capability.
nanna 7y ago
Out of interest, what are the advantages of LaTeXML over Pandoc's LaTeX to XML/HTML conversion? Why did you choose LaTeXML?
- pletnes 7y ago
  Last time I checked, pandoc’s LaTeX parser did not support too much of the TeX syntax. It only works for some subset of LaTeX, basically

_emacsomancer_ 7y ago

Doesn't Xe(La)TeX/Lua(La)TeX produce PDFs with decent unicode search support? What issues do you have?

fxfan 7y ago
XeTeX in particular does but my problem is with the uncertainty around the whole process and the frustration when a search fails on a 300 page document

JulianMorrison 7y ago

TeX is fundamentally not compatible with reflowable text because it's at the lowest level about putting glyphs in positions.

sytelus 7y ago
I fail to see any fundamental incompatibility. Every text renderer of any kind is about putting glyphs in positions. The layout would declare few things not moveable and algorithms needs to decide how to fit text around those constraints. The algorithms for TeX are computationally expensive but I don't see why with faster computers, you can't reflow the text. About 15 years ago it used to take dozen second to compile my typical PDF, now overleaf.com does that in just a second in browser.
- macleginn 7y ago
  Compilation seems to take place on the server.
- chongli 7y ago
  It's still way too slow. Lots of the documents I've worked on recently have taken over ten seconds to compile on my 2017 MacBook Pro (touch bar). That's 3 orders of magnitude too slow for reflowing at 60hz when you resize a window. I doubt a laptop will ever be 3 orders of magnitude faster (in sequential execution) than the one I have now, radical post-semiconductor computers notwithstanding.
- JulianMorrison 7y ago
  If you are outputting a layout for a reflowable medium like HTML or EPUB, you are not putting glyphs in positions. You are constructing graphical objects and defining their relationships (and how those relationships change based on form factor), and you are permitting the output device to render glyphs and put them in positions. This is why we don't use PDF for web pages.
  - jhpriestley 7y ago
    The web browser also "puts glyphs into positions." Neither HTML nor TeX source specifies positioning in the source format though. The same TeX document can be re-rendered at different page sizes, font sizes etc. Really not sure what fundamental difference you are seeing.

mwfunk 7y ago

Unicode search in PDF works fine, there are just a few issues that can cripple it for specific files, or specific files using specific PDF implementations. The following is advice for anyone working with the format, it should not be construed as making excuses for how PDF does things. It's a ginormous format with a ginormous spec that's been around for a very long time and has accumulated a lot of baroque qualities over the years. So I don't mean to condemn it, but I'm not exonerating it either. It is what it is, and if you have to work with it this might be useful. Anyhoo: Regardless of what one thinks of their other qualities, if a PDF can be searched in Adobe Reader or Acrobat, then the file is probably OK but the PDF reader that you were trying to search it with has a bug on its end (or more likely, some unimplemented dark corner of the PDF spec). On the other hand, if the file isn't searchable via Reader/Acrobat, then the problem is most likely with the authoring of the file itself. The most common thing that breaks searching is when instead of embedding all fonts used, a PDF refers to fonts by name from the local system. This can cause unpredictable issues when reading the PDF from another OS that can't resolve those font names. Another common breaker of search that seems to be much more common with TeX are workflows which somehow produce PDFs that have embedded Type 3 fonts. Type 3 fonts represent glyphs using PDF drawing instructions. It's less a file format than something that only exists as an embedded font within a PDF, but I've only seen them in the wild when authored by pdflatex or similar. It seems like TrueType or OpenType are the most reliable formats to embed across the most commonly used PDF implementations, but that's an educated guess. Type 3 font support is spotty in non-Adobe implementations. Finally, font subsetting might screw up search. Most software that knows how to produce PDFs can produce PDFs with embedded but subsetted fonts. This means the software that created the file embedded TrueType fonts (for example), but created a special version of the font for embedding that only contained the glyphs used in the PDF. Depending on the quality of the software used to do the font subsetting, the output PDF might not retain the mapping of glyphs back to Unicode characters. If that happens, text using that font in the PDF becomes unsearchable, but the PDF remains renderable. None of this is meant to excuse the shortcomings of the format, it's clearly overcomplicated and fragile. But when I've seen problems with searchability of PDFs, it's most often been with the software that was used to create the files, or how that software was configured by the author. And when it is a problem with the authored file itself, it's almost always because of fonts. Either the fonts aren't embedded, or they're embedded in a slightly oddball format that's in spec for PDF but not perfectly, universally supported by all of the various non-Adobe PDF implementations.

Liskni_si 7y ago

I haven't tried this myself but I think ConTeXt can output PDFs with embedded XML markup and even epub files. You need to use \startsection and \stopsection instead of just \section, and maybe there are other limitations, but it's a small price to pay, isn't it? https://wiki.contextgarden.net/Epub https://wiki.contextgarden.net/Epub https://wiki.contextgarden.net/Export https://wiki.contextgarden.net/Export

Veen 7y ago

The problem is that browsers choose not to implement high-quality justification algorithms like the Knuth-Plass algorithm that Tex uses because it is computationally intensive. That’s why justified text looks like garbage on the web. There are some experimental JavaScript implementations, but without browser support reflowing high-quality justified text is a non-starter on the web.

benschulz 7y ago
My assumption was also that performance is the reason we aren't getting more esthetically pleasing line breaking. Until I read a comment[1] by Philip Walton, who works on WebRender at Mozilla, that is. [1]: https://news.ycombinator.com/item?id=19473277 https://news.ycombinator.com/item?id=19473277
- Veen 7y ago
  I'm not a web developer, so take this with a huge pinch of salt, but, if floats are the problem, does that imply that with layouts that use CSS Grid or Flexbox, we could have a decent justification algorithm?
- jimhefferon 7y ago
  Oh, thanks for this. Very intertesting.
- jfk13 7y ago
  He's Patrick, not Philip.
  - benschulz 7y ago
    Woups. Thanks for the correction. I think it's too late to edit my original post. :/
- wahern 7y ago
  See also the 8-year-old Mozilla bug for a more detailed discussion: https://bugzilla.mozilla.org/show_bug.cgi?id=630181 https://bugzilla.mozilla.org/show_bug.cgi?id=630181

pjmlp 7y ago

Kind of. I was deep into LaTeX during the 90's while at the university, my Lamport's book copy state reflects how much I used to refer back to it. Nowadays I rather prefer the convenience of something like FrameMaker.

jimjimjim 7y ago

? PDF works fine for searching. It depends on the document, a pdf page could be: 1) Text 2) An image of text 3) Lines/Curves that happen to be in the shape of letters/text If it's text it perfectly searchable. If it's one of the others the creator has to also OCR it and add the text behind the image or in front but invisible (no stroke or fill).

syockit 7y ago
It could also be mostly text, but mixed with ligatures having their own codepoints (sometimes non-Unicode).
- jfk13 7y ago
  If you set the \XeTeXgenerateactualtext=1 option in a Xe(La)TeX document, the resulting PDF will include ActualText annotations to support searching even in complex scripts with ligatures, character reordering, etc.

gammarator 7y ago

Thought about updating, but the actual changelog is pretty underwhelming for a 5+GB download... https://tug.org/texlive/doc/texlive-en/texlive-en.html#news https://tug.org/texlive/doc/texlive-en/texlive-en.html#news