9 ms·
Why we can't process Emoji anymore
- gkoberger 14y agoTook me a bit to realize that this is talking about the Voxer iOS app (http://voxer.com/ http://voxer.com/), not Github (https://github.com/blog/816-emoji https://github.com/blog/816-emoji).
- whit537 14y agoYeah, I was worried there for a sec. :^)
- sneak 14y agoTLDR: node sucks
- akie 14y agoTLDR: The V8 engine can't (supposedly) encode Unicode codepoints that are over 16-bits in length, because it uses the UCS-2 encoding.
- throwaway54-762 14y agoTLDR: v8 "sucks" (and doesn't support Unicode code points outside of the lowest ~64k characters). Edit: v8 in general is pretty cool, but not supporting Unicode outside UCS-2 is pretty bad.
- deleted 14y ago[deleted]
- marshray 14y agoMost apps seem to "support" surrogate pairs by simply not being aware of them at all. Good on the V8 developers for recognizing these conditions that their code didn't fully handle and refusing to muddle on through with broken processing.
- tptacek 14y agoIt's v8's fault, and v8 does not suck.
- prodigal_erik 14y agoUnicode 2.0 added surrogate pairs in 1996. Unfortunately, the first versions of both Java and JavaScript predated this and got strings horribly wrong, and now any conforming implementation of either is required to suck. The Right Thing would be for almost everyone to work with only combining character sequences, except for a rare few who need to know how to dissect one into its codepoints and reassemble them correctly (just as people don't normally need to extract high or low bits from an ASCII character).
- jrabone 14y agoNo. Combining characters and NF(K)C/D normalisation rules are a different problem entirely - consider the "heavy metal umlaut" (ie. Spın̈al Tap) where there is no lossless conversion possible - only “n" followed by U+0308
- prodigal_erik 14y agoThey're facets of the same problem. I shouldn't routinely be dealing with either surrogates or combining marks; unless I have a specific reason, it's only an opportunity to make a mistake that hardly anyone knows how to troubleshoot. "n̈" should be an indivisible string of length one until I need to ask how it would actually be encoded in UTF-16 or whatever.
- jrabone 14y agoBut that's the point - there is no such character. Given the Unicode consortium have added codepoints for every other bloody thing under the sun, I'm amazed that there isn't one for n-diaresis but there you are. Add a small number of people who for artistic reasons decide that they want to make life hard (Rinôçérôse I'm looking at you) and you just have to accept that the length of your string might not equal the number of codepoints contained therein...
- rymith 14y agoNot even a little bit accurate.
- driverdan 14y agoIf you search for V8 UCS-2 you'll find a lot of discussion on this issue dating back at least a few years. There are ways to work around V8's lack of support for surrogate pairs. See this V8 issue for ideas: https://code.google.com/p/v8/issues/detail?id=761 https://code.google.com/p/v8/issues/detail?id=761 My question is why does V8 (or anything else) still use UCS-2?
- est 14y agobecause counting 2 bytes is much faster for computers than counting vary 1, 2, 3 or even 4 bytes.
- speleding 14y agoThis is not a real issue because counting code points in an UTF8 string is easy too: the encoding is cleverly defined such that you just need to check the number of bytes that have the top bit cleared. Since UTF8 strings are generally shorter it can even be faster than counting UTF-16 if you don't know the length in advance.
- gsnedders 14y agoThe ES5 spec defines a string as being a series of UTF-16 code-units, which inherently means surrogates show through. APIs like that tend to be low priority because they aren't used by browsers (which pass everything through as UTF-16 code-units, typically treating them as possibly-valid UTF-16 strings).
- masklinn 14y ago> My question is why does V8 (or anything else) still use UCS-2? Because the ES spec defines a string as a sequence of UTF-16 code units (aka UCS-2-with-visible-surrogates), because as many others (e.g. Java) the language's strings were created during/inherited from Unicode 1.0 which fit in 16 bits (UTF-16 is a retrofitting of Unicode 1.0 fixed-width to accomodate the full range of later unicode version by adding surrogate pairs)
- pjscott 14y agoThe quick summary, for people who don't like ignoring all those = signs, is that V8 uses UCS-2 internally to represent strings, and therefore can't handle Unicode characters which lie outside the Basic Multilingual Plane -- including Emoji.
- adrianpike 14y agoHere's the thread in the v8 bug tracker about this issue: http://code.google.com/p/v8/issues/detail?id=761 http://code.google.com/p/v8/issues/detail?id=761 Is there a reason that the workaround in comment 8 won't address some of these issues?
- dgl 14y agoI don't think it's needed anymore. If you read closely you'll see the original linked message is from January and there's an update on that issue from March when a fix was made in V8.
- praptak 14y agoSometimes you need to know about encodings, even if you're just a consumer. Putting just one non 7-bit character in your SMS message will silently change its encoding from 7-bit (160 chars) to 8-bit (140 chars) or even 16 bit (70 chars) which might make the phone split it into many chunks. The resulting chunks are billed as separate messages.
- jrabone 14y agoThe GSM 03.38 charset specified for SMS is not straight 7-bit ASCII. See eg. http://www.dreamfabric.com/sms/default_alphabet.html http://www.dreamfabric.com/sms/default_alphabet.html
- fwr 14y agoOn iOS, using any non-basic Latin character in SMS makes it switch to 16 bit, even when there is no reason for that to happen. It's a thing that most foreign language speakers must live with. By doing this full of excuses write-up, this guy wasted a substantial amount of time that he could have spent better researching the issue. Your consumer doesn't care that Emoji is this much or that much bits, it doesn't matter for him that you're running your infrastructure on poorly chosen software - there is absolutely no excuse for not supporting this in a native iOS app, especially now that Emoji is so widely used and deeply integrated in iOS. How is that a problem they are focusing on, anyway, when their landing page features awful, out of date mockups of the app? (not even actual screenshots - notice the positions of menu bar items) They are also featuring Emoji in every screenshot - ending support might be a fresh development, but I still find that ironic.
- jrabone 14y agoAbsolutely right. The customer does not care that you made a shortsighted decision to pick a language for a TEXT based system that cannot correctly support none-BMP Unicode. There are no excuses, surrogates have been out there for years (Windows was using UTF16-LE from NT 3.51 / Unicode 2) as have 4 byte UTF8 encodings. JavaScript is a joke in this respect, and is keeping horrors like Shift-JIS alive long after they should have been retired.
- 14y ago
- deleted 14y ago[deleted]
- freedrull 14y agoWhy on earth would the people who wrote V8 use UCS-2? What about alternative JS runtimes?
- marshray 14y agoBecause Unicode was sold to the world's software developers as a fixed-width encoding claiming 16 bits would be all we'd ever need.
- thristian 14y agoYes it was, in 1991 when NT and Java and Cocoa were new or under development. In 1996, Unicode 2.0 came out with surrogate pairs and astral planes, and Unicode was no longer a 16 bit encoding. I'm pretty sure work on V8 started after 1996. More likely is the idea that the authors of V8 felt that UCS-2 was an acceptable speed/correctness trade-off.
- dmethvin 14y agoYes, and several C/C++ conventions and types seemed to make that a safe choice, for example wchar_t. Let's face it, collectively we really screwed this one up. It's the biggest mistake since Microsoft chose the backslash as a path separator in DOS 2.0.
- magic_haze 14y agoIt was actually IBM's fault: they used '/' to denote CLI args in the apps they wrote for DOS 1.0, which didn't have any concept of directories. From Larry Osterman's blog [1]: > Here's a little known secret about MS-DOS. The DOS developers weren't particularly happy about this state of affairs - heck, they all used Xenix machines for email and stuff, so they were familiar with the *nix command semantics. So they coded the OS to accept either "/" or "\" character as the path character (this continues today, btw - try typing "notepad c:/boot.ini" on an XP machine (if you're an admin)). And they went one step further. They added an undocumented system call to change the switch character. And updated the utilities to respect this flag. [1] http://blogs.msdn.com/b/larryosterman/archive/2005/06/24/432386.aspx http://blogs.msdn.com/b/larryosterman/archive/2005/06/24/432...
- languagehacker 14y agoWe seem to be seeing this more and more with Node-based applications. It's a symptom of the platform being too immature. This is why you shouldn't adopt these sorts of stacks unless there's some feature they provide that none of the more mature stacks support yet. And even then, you should probably ask yourself if you really need that feature.
- fusiongyro 14y agoAccording to Cogito, this was fixed in March: http://news.ycombinator.com/item?id=4834731 http://news.ycombinator.com/item?id=4834731 I want to agree with you simply because I don't like Node, but it's hardly fair to damn something over a bug that was fixed 9 months ago.
- oofabz 14y agoThis is why UTF-8 is great. If it works for any Unicode character it will work for them all. Surrogate pairs are rare enough that they are poorly tested. With UTF-8, if there are issues with multi-byte characters, they are obvious enough to get fixed. UTF-16 is not a very good encoding. It only exists for legacy reasons. It has the same major drawback as UTF-8 (variable-length encoding) but none of the benefits (ASCII compatibility, size efficient).
- pixelcort 14y agoThe problem with UTF-8 is that lots of tools have 3 byte limits, and characters like Emoji take up 4 bytes in UTF-8.
- derleth 14y agoAnd the problem with UTF-16 is that a lot of applications can't handle surrogate pairs, except a lot of Emoji are above the BMP, aren't they? So why is this a bigger deal for UTF-8 than UTF-16?
- masklinn 14y ago> except a lot of Emoji are above the BMP, aren't they? All of the Unicode 6.0 emoji are.
- eps 14y agoWhich tools? Honest question, as the three byte limit seems rather arbitrary and no more logical than, say, a four byte one.
- jrabone 14y agoIt is totally arbitrary - there's no reason you can't have degenerate 6-byte encodings, and compliant decoders should cope with them. See Marcus Kuhn's excellent UTF-8 decoder torture test page linked elsewhere in this thread.
- shocks 14y agoVery informative, great read. Thanks!
- ender7 14y agoApropos: http://mathiasbynens.be/notes/javascript-encoding http://mathiasbynens.be/notes/javascript-encoding TL;DR: - Javascript engines are free to internally represent strings as either UCS-2 or UTF-16. Engines that choose to go USC-2 tend to replace all glyphs outside of the BMP with the replacement char (U+FFFD). Firefox, IE, Opera, and Safari all do this (with some inconsistencies). - However, from the point of view of the actual JS code that gets executed, strings are always UCS-2 (sort of). In UTF-16, code points outside the BMP are encoded as surrogate pairs (4 bytes). But -- if you have a Javascript string that contains such a character, it will be treated as two consecutive 2-byte characters. var x = '𝌆'; x.length; // 2 x[0]; // \uD834 x[1]; // \uDF06 Note that if you insert said string into the DOM, it will still render correctly (you'll see a single character instead of two ?s).
- othermaciej 14y agoSafari uses UTF-16, not UCS-2. I believe this is true of other browsers as well. Otherwise this would render the replacement char, but it doesn't, it renders correctly: javascript:var x = '𝌆';document.write(x);
- gsnedders 14y agoWell, a JS string is just a series of UTF-16 code-units (per ES5, there is no impl choice here), so there isn't really any encoding pre-se (and isn't necessarily a UTF-16 string, per the spec's definition thereof, as lone surrogates are valid). The fact that that works is more a testament to the the DOM being UTF-16 than JS. (On the other hand, I'm sure you knew that. But probably there are people reading your comment who didn't. :))
- othermaciej 14y agoYou are technically correct, the best kind of correct! But I think we both agree there is absolutely no sense in which anything in browser engines is UCS-2, and that browsers will not in fact replace characters beyond the BMP with the replacement glyph, as the top-level comment claimed. It is kind of embarassing that the top rated comment (as of writing) but says completely false things.
- ricardobeat 14y agoPlease, if you're going to post text to a Gist at least use the .md extension: https://gist.github.com/4151124 https://gist.github.com/4151124
- ctrlaltesc 14y agoWhich enables an even more readable layout with gist.io http://gist.io/4151124 http://gist.io/4151124
- FredericJ 14y agoHow about this npm module : https://npmjs.org/package/emoji https://npmjs.org/package/emoji ?
- hkmurakami 14y ago>Wow, you read though all of that? You rock. I'm humbled that you gave me so much of your attention. That was actually really fun to read, even as a now non-technical guy. I can't put a finger on it, but there was something about his style that gave off a really friendly vibe even through all the technical jargon. That's a definite skill!
- jgeorge 14y agoDeSalvo's source comments have always been an entertaining read. :)
- eps 14y agoThey control their clients, so they could've just re-encoded emojies with custom 16bit escaping scheme, make the backend transparently relay it over in escaped form and decode it back to 17bits at the other end. Or am I missing something obviuos here?
- cjensen 14y agoUCS-16 is only used by programs which jumped the gun and implemented Unicode before it was all done. (It was 16 bits for awhile with Asian languages sharing code points so that the font in use determined whether the text was displayed as Chinese vs Japanese vs. etc). What Century was V8 written in that they thought UCS-16 was an acceptable thing to implement? Good rule of thumb for implementers: get over it and use 32 bits internally. Always use UTF-8 when encoding into a byte stream. Add UTF-16 encoding if you must interface with archaic libraries.
- masklinn 14y ago> UCS-16 is only used by programs which jumped the gun and implemented Unicode before it was all done. There's no such thing as "all done", Unicode 1.0 was 16 bit, Unicode 6 was released recently.
- clebio 14y agoSomewhat meta, but this would be one where showing subdomain on HN submissions would be nice. The title is vague enough that I assumed it was something to do with _Github_ not processing Emoji (which would be sort of a strange state of affairs...).
- ladon86 14y agoNot that strange, Github implements much of the Emoji set using different shortcuts, see the reference here: http://www.emoji-cheat-sheet.com/ http://www.emoji-cheat-sheet.com/ Before I read the article I guessed that maybe the icon set had some licensing issues for Github. Luckily, not so! (:smiley:)
- clebio 14y agoThat was basically my point. It would be strange if they _stopped_ processing it.
- beaumartinez 14y agoThis is dated January 2012. By the looks of things, this was fixed in March 2012[1] [1] https://code.google.com/p/v8/issues/detail?id=761#c33 https://code.google.com/p/v8/issues/detail?id=761#c33
- Cogito 14y agoI wonder if this has been rolled into Node yet. [edit] Node currently uses V8 version 3.11.10.25, which was released after this fix was made, but not sure if the fix was merged to trunk [edit2] actually, looks like it has, though I can't identify the merge commit
- pla3rhat3r 14y agoI love this article. So often it has been difficult to explain to people why one set of characters can work while others will not. This lays out some great historical info that will be helpful going forward.
- kstenerud 14y agoSmall nitpick, but Objective-C does not require a particular string encoding internally. In Mac OS and iOS, NSString uses one of the cfinfo flags to specify whether the internal representation is UTF-16 or ASCII (as a space-saving mechanism).
- evincarofautumn 14y agoFailures in Unicode support seem usually to result from the standard’s persistently shortsighted design—well intentioned and carefully considered though it undoubtedly is. It’s a “good enough” solution to a very difficult problem, but I wonder if we won’t see Unicode supplanted in the next decade. All that aside: emoji should not be in Unicode. Fullstop.
- deleted 14y ago[deleted]
- alexbosworth 14y agoFixed a good while ago for node.js
- dgreensp 14y agoThe specific problems the author describes don't seem to be present today; perhaps they were fixed. That's not to say this conversions aren't a source of issues, just that I don't see any show-stopper problems currently in Node, V8, or JavaScript. In JavaScript, a string is a series of UTF-16 code units, so the smiley face is written '\ud83d\ude04'. This string has length 2, not 1, and behaves like a length-2 string as far as regexes, etc., which is too bad. But even though you don't get the character-counting APIs you might want, the JavaScript engine knows this is a surrogate pair and represents a single code point (character). (It just doesn't do much with this knowledge.) You can assign '\ud83d\ude04' to document.body.innerHTML in modern Chrome, Firefox, or Safari. In Safari you get a nice Emoji; in stock Chrome and Firefox, you don't, but the empty space is selectable and even copy-and-pastable as a smiley! So the character is actually there, it just doesn't render as a smiley. The bug that may have been present in V8 or Node is: what happens if you take this length-2 string and write it to a UTF8 buffer, does it get translated correctly? Today, it does. What if you put the smiley directly into a string literal in JS source code, not \u-escaped? Does that work? Yes, in Chrome, Firefox, and Safari.
- jruderman 14y agoThe invisible smiley was a font system problem, fixed in Firefox 19 Aurora (assuming you're on Mac). https://bugzilla.mozilla.org/show_bug.cgi?id=715798 https://bugzilla.mozilla.org/show_bug.cgi?id=715798
- pbiggar 14y agoA couple of reasons why it makes sense for V8 and other vendors to use UCS2: - The spec says UCS2 or UTF16. Those are the only options. - UCS2 allows random access to characters, UTF-16 does not. - Remember how the JS engines were fighting for speed on arbitrary benchmarks, and nobody cared about anything else for 5 years? UCS2 helps string benchmarks be fast! - Changing from UCS2 to UTF-16 might "break the web", something browser vendors hate (and so do web developers) - Java was UCS2. Then Java 5 changed to UTF-16. Why didn't JS change to UTF-16? Because a Java VM only has to run one program at once! In JS, you can't specify a version, an encoding, and one engine has to run everything on the web. No migration path to other encodings!
- cmccabe 14y agoUCS2 allows random access to characters, UTF-16 does not. I'm not sure if that's really true. On IBM's site, they define 3 levels of UCS-2, only one of which excludes "combining characters" (really code points). http://pic.dhe.ibm.com/infocenter/aix/v6r1/index.jsp?topic=%2Fcom.ibm.aix.nls%2Fdoc%2Fnlsgdrf%2Fiso10646_ucs-2.htm http://pic.dhe.ibm.com/infocenter/aix/v6r1/index.jsp?topic=%... If you have combining characters, then you can't simply take the number of bytes and divide by 2 to get the number of letters. If you don't have combining characters, then you have something which isn't terribly useful except for European languages (I think?) Maybe someone more familiar with the implementation can describe which path they actually went down for this... given what I've heard so far, I'm not optimistic.
- pbiggar 14y agoOK, I cracked into the V8 source to take a look at what actually happens. It looks like the implementation does use random access for two-byte strings. However, it also uses multiple multiple string implementations (ASCII, 2 byte strings, "consString" (I presume some kind of Rope), "Sliced Strings" (sounds like a rope again, but might be shared storage of the string contents for immutable strings)), so they could likely use other implementations with whatever properties they choose. See https://github.com/v8/v8/blob/3ff861bbbb62a6c0078e042d8077b27f6ae2fa8f/src/objects-inl.h#L2469 https://github.com/v8/v8/blob/3ff861bbbb62a6c0078e042d8077b2... and https://github.com/v8/v8/blob/3ff861bbbb62a6c0078e042d8077b27f6ae2fa8f/src/objects-inl.h#L2555 https://github.com/v8/v8/blob/3ff861bbbb62a6c0078e042d8077b2....
- csense 14y agoA two-character sequence for a smiley face that should be compatible with everything in existence: :) Problem solved. Why is this front page material (#6 as of this writing)?
- xn 14y agoHere's the message decoded from quoted-printable: https://gist.github.com/4151707#file_emoji_sad_decoded.txt https://gist.github.com/4151707#file_emoji_sad_decoded.txt
- eloisant 14y agoMaybe nickpicking but I don't think Softbank came up with the Emoji. Emoji existed way before Softbank bought the Japanese Vodaphone, and even before Vodaphone bought J-Phone. So emoji were probably invented by J-Phone, while Softbank was mostly taking care of Yahoo Japan.
- dale-cooper 14y agoThe UCS-2 heritage is kind of annoying. In java for example, chars (the primitive type, which the Character class just wraps) are 16 bits. So one instance of a Character may not be a full "character" but rather a part of a surrogate pair. This creates a small gotcha where the length of a string might not be the same as the amount of characters it has. And that you just cant split/splice a Character array naively (because you might split it at a surrogate pair).
- masklinn 14y agoWhich, at the end of the day, doesn't really matter since a code point is not a "character" in the sense of "the smallest unit of writing" (as interpreted by an end-user): many "characters" may (depending on the normalization form) or will (jamo) span multiple codepoints. Splitting on a character array is always broken, regardless of surrogate pairs.
- dale-cooper 14y agoYes. What i'm saying is that it would feel less error prone if the character object was actually a codepoint. It's a leaky abstraction, you shouldn't need to handle something that is tied to the internal representation of strings in the jvm. Can one "character" span multiple codepoints? Do you have an example of this?
- masklinn 14y ago> It's a leaky abstraction, you shouldn't need to handle something that is tied to the internal representation of strings in the jvm. And I'm saying it doesn't really matter, because unicode codepoints are already a form of "leaky abstraction" which you'll have to handle (in that a read/written "character" does not correspond 1:1 to a codepoint anyway). Unicode is a tentative standardization of historical human production, and if you expect that to end up clean and simple you're going to have a hard time. > Can one "character" span multiple codepoints? Yes. > Do you have an example of this? Devanagari (the script used for e.g. Sanskrit) is full of them. For instance, "sanskrit" is written "संस्कृतम्" [sə̃skɹ̩t̪əm]. If you try to select "characters" in your browser you might get 4 (सं, स्कृ, त and म्) or 5 (सं, स्, कृ, त and म्) or maybe yet another different count, but this is a sequence of 9 codepoints (regardless of the normalization, it's the same in all of NFC, NFD, NFKC and NFKD as far as I can tell): स: DEVANAGARI LETTER SA ं: DEVANAGARI SIGN ANUSVARA स: DEVANAGARI LETTER SA ्: DEVANAGARI SIGN VIRAMA क: DEVANAGARI LETTER KA ृ: DEVANAGARI VOWEL SIGN VOCALIC R त: DEVANAGARI LETTER TA म: DEVANAGARI LETTER MA ्: DEVANAGARI SIGN VIRAMA Note: I'm not a Sanskrit speaker and I don't actually know devanagari (beyond knowing that it's troublesome for computers, as are jamo) so I can't even tell you how many "symbols" a native reader would see there.
- masklinn 14y agoWow, the first half of the text is basically full of crap and claims which don't even remotely match reality, and now I'm reaching the technical section which can only get even more wrong.
- masklinn 14y agoTo whoever the downvoter was: no, seriously. For instance in the first few paragraphs: * emoji were invented by NTT DoCoMo, not Softbank * even if that had been right Softbank's copyrighting of their emoji representations has no bearing on NTT and KDDI/au using completely different implementations (and I do mean completely, KDDI/au essentially use <img> tags) * lack of cooperation is endemic to japanese markets (especially telecoms) and has nothing to do with "ganging up" * if NTT and au/KDDI wanted to gang up on Softbank you'd think they'd share the same emoji * you didn't have to run "adware apps" to unlock the emoji keyboard (there were numerous ways to do so from dedicated — and usually quickly nuked – apps to apps "easter eggs" to jailbreak to phone backup edit/restore) That's barely the first third.
- mranney 14y agoNote that this message is almost a year old now. The issue has been addressed by the node and V8 teams.