6 ms·

This is interesting. Can you show which Japanese names cannot be encoded in UTF-8 please?

kstrauser 2y ago
They cannot.
lmm 2y ago
Any of the well-known han unification examples. People claim this can be solved with some kind of out-of-band font selection vaporware, but the kind of person who thinks it's fine to demand UTF-8 everywhere never actually stoops to implement this imaginary font selection functionality.
- mnau 2y ago
  > implement this imaginary font selection functionality We are already doing GSUB/GDEF tables and a lot of horrible stuff to display pretty glyphs on the screen. Hell, there are literal instructions for VM in TTF files to help with pretty rasterization on low-res screens. Making font rendering library is hard, Nintendo hard. That's just a nature of the beast. Fonts are messy and if we want one standard to deal with it once and for all, that means some compromises must be done. CJK is not some obscure alphabet. In this case, perfect is enemy of good. I am just glad there really is only one (non-niche) standard, obligatory https://xkcd.com/927/ https://xkcd.com/927/
  - lmm 2y ago
    > CJK is not some obscure alphabet. Right. One would think that misrendering a major world language used by over 100 million people would be an issue that warrants some attention. But too many of the HN crowd don't care. > In this case, perfect is enemy of good. It's not though. Unicode-only is not just imperfect, it's an outright regression for Japanese. Meanwhile traditional encoding-aware programs render Japanese just fine.
    - mnau 2y ago
      > It's not though. Unicode-only is not just imperfect, it's an outright regression for Japanese. Unicode has variant selectors which can deal with region variants (e.g. https://imgur.com/a/syMcWNO https://imgur.com/a/syMcWNO) to deal with the issue. Granted, it's not very widely used, but Unicode provided a solution. The onus is now on application developers and font providers. Choosing to limit number of characters to 2 bytes was a good technical choice back in 1992, non-unified CJK wouldn't fit (that was a time of 2-4MB RAM and 100-200MB HDD). A solution to problem was later provided. Using Shift-JIS is like doing on-IP, non-UDP protocol on the internet. Sure, you can do it, but it's just non-perspective choice. A better invested time is on improving standard protocol, which is Unicode (e.g. conversion software).
      - lmm 2y ago
        > Unicode has variant selectors which can deal with region variants (e.g. https://imgur.com/a/syMcWNO https://imgur.com/a/syMcWNO) to deal with the issue. They're officially deprecated and cause issues with breaking search etc.. So they're still below feature parity with using a traditional encoding. Traditional encoding support is also easier to test since documents with traditional encodings are more widespread and exist for many world languages, not just Japanese.
- ogurechny 2y ago
  Well, I must nitpick: Shift JIS is actually just one of those out-of-band vaporware methods. Sure, it's better supported because of legacy, but new code that doesn't care about handling some lang metadata is not going to care about Shift JIS either. Of course, there is no correct solution (people in other comments seem to believe it exists a bit too quickly). Dynamic HTML page can load pieces of text written by people all across the world, it can't rely on any “main” language. Those people can't be automatically classified (based on some browser settings or location). Then there are always people who know both Chinese and Japanese (while using some other system locale, to make things more complex). It is wrong to assume that they should not be able to use Chinese forms and Japanese forms at once, even inside a single paragraph or phrase. I wonder why Unicode has not simply introduced some “combining language form character” to make user choice stick. After all, there's a whole subsystem for emoji modification, and those things were once “weird Japanese texting customs”. As for complexity of handling Unicode text, it asymptotically reaches its maximum anyway. Wait a second, there are “variation selectors” and some “Ideographic Variation Database”. Is it the solution? Can IMEs and converters simply stamp each variable character with invisible mark based on current input language? I suppose there's some catch…
  - lmm 2y ago
    > Shift JIS is actually just one of those out-of-band vaporware methods. It's the opposite of vaporware; there's a whole bunch of well-known software that handles encodings correctly. > new code that doesn't care about handling some lang metadata is not going to care about Shift JIS either. If you support encodings the way almost every programming language tells you too, you'll handle Shift-JIS just fine. If you use the "legacy" encodings approach and test even one non-English language you'll handle Japanese fine. > Can IMEs and converters simply stamp each variable character with invisible mark based on current input language? I suppose there's some catch… They're officially deprecated, IMEs and converters don't use them, naïve search implementations (which is to say most search implementations, because people do not test for an obscure edge case) break, ....
    - ogurechny 2y ago
      But there is no “naive” implementation of Unicode. It won't handle emoji, normalization, and thousands other things. People use The Library anyway. Adding another mutagen to it should be no different from the rest. As for official stance, maybe it's time for some group to agree on certain non-conflicting sequences, and implement them in some library/stack. A particularly evil solution would choose arbitrary existing combining characters that can't be declared “illegal” retroactively. By the way, is it possible to make proper quotes from other languages as described above when using Shift JIS?
TheDong 2y ago
The parent said "displayed correctly", not "encoded". For example, if I want to talk about the fairly rare japanese name '刃一' (jinichi), there's a chance your computer displays it correctly, but there's also a chance your computer displays the chinese variant of the first character, making it look wrong. It's basically up to font-choice of your computer. The "correct" way to fix that would be for me to be able to tag it with 'lang=ja', but Hacker News doesn't let me include html tags or some other 'language selector' in my comment, so I'm unable to indicate whether that's supposed to be the chinese or japanese variant of the character. Most unicode text files don't have extra metadata indicating if a certain bit of text is japanese or chinese, so displaying it correctly by adding the correct 'lang' tag is impossible, especially since it's perfectly possible for one utf-8 text to mix both chinese and japanese. See https://heistak.github.io/your-code-displays-japanese-wrong/ https://heistak.github.io/your-code-displays-japanese-wrong/
- int_19h 2y ago
  How does Shift-JIS fix this?
  - lmm 2y ago
    The Shift-JIS codepoints for the characters of that name are understood to refer to Japanese characters, so fonts render them correctly. Encoding-aware programs have different representations (such as GB 18030 codepoints) for the similar-but-different Chinese characters that unicode-only programs tend to display these characters as, and so will render them differently.
    - int_19h 2y ago
      If I understand you correctly, it means that the encoding itself serves as the metadata that indicates Chinese/Japanese. In which case, why is it unreasonable to ask for the same for UTF-8, except using some more clearly specified way to indicate this (like lang="ja" etc), rather than encoding it all into separate characters?
      - 2y ago

They cannot.

lmm 2y ago

Any of the well-known han unification examples. People claim this can be solved with some kind of out-of-band font selection vaporware, but the kind of person who thinks it's fine to demand UTF-8 everywhere never actually stoops to implement this imaginary font selection functionality.

mnau 2y ago
> implement this imaginary font selection functionality We are already doing GSUB/GDEF tables and a lot of horrible stuff to display pretty glyphs on the screen. Hell, there are literal instructions for VM in TTF files to help with pretty rasterization on low-res screens. Making font rendering library is hard, Nintendo hard. That's just a nature of the beast. Fonts are messy and if we want one standard to deal with it once and for all, that means some compromises must be done. CJK is not some obscure alphabet. In this case, perfect is enemy of good. I am just glad there really is only one (non-niche) standard, obligatory https://xkcd.com/927/ https://xkcd.com/927/
- lmm 2y ago
  > CJK is not some obscure alphabet. Right. One would think that misrendering a major world language used by over 100 million people would be an issue that warrants some attention. But too many of the HN crowd don't care. > In this case, perfect is enemy of good. It's not though. Unicode-only is not just imperfect, it's an outright regression for Japanese. Meanwhile traditional encoding-aware programs render Japanese just fine.
  - mnau 2y ago
    > It's not though. Unicode-only is not just imperfect, it's an outright regression for Japanese. Unicode has variant selectors which can deal with region variants (e.g. https://imgur.com/a/syMcWNO https://imgur.com/a/syMcWNO) to deal with the issue. Granted, it's not very widely used, but Unicode provided a solution. The onus is now on application developers and font providers. Choosing to limit number of characters to 2 bytes was a good technical choice back in 1992, non-unified CJK wouldn't fit (that was a time of 2-4MB RAM and 100-200MB HDD). A solution to problem was later provided. Using Shift-JIS is like doing on-IP, non-UDP protocol on the internet. Sure, you can do it, but it's just non-perspective choice. A better invested time is on improving standard protocol, which is Unicode (e.g. conversion software).
    - lmm 2y ago
      > Unicode has variant selectors which can deal with region variants (e.g. https://imgur.com/a/syMcWNO https://imgur.com/a/syMcWNO) to deal with the issue. They're officially deprecated and cause issues with breaking search etc.. So they're still below feature parity with using a traditional encoding. Traditional encoding support is also easier to test since documents with traditional encodings are more widespread and exist for many world languages, not just Japanese.
ogurechny 2y ago
Well, I must nitpick: Shift JIS is actually just one of those out-of-band vaporware methods. Sure, it's better supported because of legacy, but new code that doesn't care about handling some lang metadata is not going to care about Shift JIS either. Of course, there is no correct solution (people in other comments seem to believe it exists a bit too quickly). Dynamic HTML page can load pieces of text written by people all across the world, it can't rely on any “main” language. Those people can't be automatically classified (based on some browser settings or location). Then there are always people who know both Chinese and Japanese (while using some other system locale, to make things more complex). It is wrong to assume that they should not be able to use Chinese forms and Japanese forms at once, even inside a single paragraph or phrase. I wonder why Unicode has not simply introduced some “combining language form character” to make user choice stick. After all, there's a whole subsystem for emoji modification, and those things were once “weird Japanese texting customs”. As for complexity of handling Unicode text, it asymptotically reaches its maximum anyway. Wait a second, there are “variation selectors” and some “Ideographic Variation Database”. Is it the solution? Can IMEs and converters simply stamp each variable character with invisible mark based on current input language? I suppose there's some catch…
- lmm 2y ago
  > Shift JIS is actually just one of those out-of-band vaporware methods. It's the opposite of vaporware; there's a whole bunch of well-known software that handles encodings correctly. > new code that doesn't care about handling some lang metadata is not going to care about Shift JIS either. If you support encodings the way almost every programming language tells you too, you'll handle Shift-JIS just fine. If you use the "legacy" encodings approach and test even one non-English language you'll handle Japanese fine. > Can IMEs and converters simply stamp each variable character with invisible mark based on current input language? I suppose there's some catch… They're officially deprecated, IMEs and converters don't use them, naïve search implementations (which is to say most search implementations, because people do not test for an obscure edge case) break, ....
  - ogurechny 2y ago
    But there is no “naive” implementation of Unicode. It won't handle emoji, normalization, and thousands other things. People use The Library anyway. Adding another mutagen to it should be no different from the rest. As for official stance, maybe it's time for some group to agree on certain non-conflicting sequences, and implement them in some library/stack. A particularly evil solution would choose arbitrary existing combining characters that can't be declared “illegal” retroactively. By the way, is it possible to make proper quotes from other languages as described above when using Shift JIS?

TheDong 2y ago

The parent said "displayed correctly", not "encoded". For example, if I want to talk about the fairly rare japanese name '刃一' (jinichi), there's a chance your computer displays it correctly, but there's also a chance your computer displays the chinese variant of the first character, making it look wrong. It's basically up to font-choice of your computer. The "correct" way to fix that would be for me to be able to tag it with 'lang=ja', but Hacker News doesn't let me include html tags or some other 'language selector' in my comment, so I'm unable to indicate whether that's supposed to be the chinese or japanese variant of the character. Most unicode text files don't have extra metadata indicating if a certain bit of text is japanese or chinese, so displaying it correctly by adding the correct 'lang' tag is impossible, especially since it's perfectly possible for one utf-8 text to mix both chinese and japanese. See https://heistak.github.io/your-code-displays-japanese-wrong/ https://heistak.github.io/your-code-displays-japanese-wrong/

int_19h 2y ago
How does Shift-JIS fix this?
- lmm 2y ago
  The Shift-JIS codepoints for the characters of that name are understood to refer to Japanese characters, so fonts render them correctly. Encoding-aware programs have different representations (such as GB 18030 codepoints) for the similar-but-different Chinese characters that unicode-only programs tend to display these characters as, and so will render them differently.
  - int_19h 2y ago
    If I understand you correctly, it means that the encoding itself serves as the metadata that indicates Chinese/Japanese. In which case, why is it unreasonable to ask for the same for UTF-8, except using some more clearly specified way to indicate this (like lang="ja" etc), rather than encoding it all into separate characters?
    - 2y ago