9 ms·

Emoji.length == 2

279 points by stanzheng 10y ago

phkahler 10y ago
Unicode is fucked. All these bullshit emojis remind me of the 1980s when ASCII was 7 bits but every computer manufacturer (Atari, Commodore, Apple, IBM, TI, etc...) made their own set of characters for the 128 values of a byte beyond ASCII. Of course Unicode is a global standard so your pile-of-poop emoji will still be a pile-of-poop on every device even if the amount of steam is different for some people. It's beyond me why this is happening. Who decides which bullshit symbols get into the standard anyway?
- raphlinus 10y ago
  Meet the shadowy overlords who approve emojis[0] [0] http://www.latimes.com/business/technology/la-fi-tn-emoji-q-and-a-20160125-htmlstory.html http://www.latimes.com/business/technology/la-fi-tn-emoji-q-...
- sdegutis 10y ago
  Language is inherently complex, there's no way to solve this in any "cleaner" way than what we already came up with. Unfortunately the best way forward is to build up what we already have and cover all the warts with wrapper functions/libraries.
  - phkahler 10y ago
    And where are the sex emoji? The dirtiest thing I've been able to text is a heart and a pair of handcuffs ;-)
    - masklinn 10y ago
      The Love Hotel (U+1F3E9) is rather obvious, maybe the kiss mark (U+1F48B) as well, though the raunchiest ones (in actual use) are a bit more… discreet?: the aubergine (U+1F346) and splashing "sweat" (U+1F4A6).
      - pbhjpbhj 10y ago
        On Ubuntu, and probably other OS/distro too, ctrl+alt+shift+U gives you a underline-u symbol, type in the Unicode and then press <enter>.
        masklinn 10y ago
        I'm not sure what use that would be? U+<hex> is a normal way to designate codepoints, and I can't put the actual emoji in the comment as they're stripped on submission.
        pbhjpbhj 10y ago
        Ah, sorry, that comment was for those playing along who might want to see the listed glyphs.
    - tveita 10y ago
      Available for over 2000 years now. https://mobile.twitter.com/unicode/status/722133439726505984 https://mobile.twitter.com/unicode/status/722133439726505984
  - PeterisP 10y ago
    Well, there is one way, we can simplify and standardize format of language. Unfortunately that requires generations of "reeducation", so it's not a viable solution in the short term - but it does seem possible that this is where languages are going in the next few centuries, as globalization, easier travel and more interrelated communities are likely to result in slow, gradual convergence to less languages as many of the current 6000+ languages cease to be used in practice.
    - Manishearth 10y ago
      I am always surprised when people think that the solution to "dealing with language in programming is complex" is "let's reeducate the world by changing their language" instead of "let's reeducate programmers".
- TAForObvReasons 10y ago
  There is a Unicode encoding "UTF-32" which has the advantage of being fixed width. This is not popular for the obvious reason that even ascii characters are expanded to 4 bytes. Additionally the windows APIs, among other interfaces, are not equipped to handle 4-byte codepages.
  - raphlinus 10y ago
    It's fixed width with respect to code points, but not with respect to any of the other things mentioned in the linked article. For example, the black heart with emoji variation selector (which makes it render red) is two code points.
  - Manishearth 10y ago
    Being fixed width is not an advantage. Code points aren't a very useful unit of text outside of the implementation of algorithms defined by unicode. All of these algorithms generally require iteration anyway. O(1) code point indexing is nearly useless. http://manishearth.github.io/blog/2017/01/14/stop-ascribing-meaning-to-unicode-code-points/ http://manishearth.github.io/blog/2017/01/14/stop-ascribing-...
  - marcosdumay 10y ago
    > "UTF-32" which has the advantage of being fixed width It's fixed width for now. It can not hold all the current available code-points, so it will probably have the same fate as UTF-16 (but it will probably take a long time). I'd stay away from it.
    - jcranmer 10y ago
      There are currently 17 × 65536 code points (U+0000..U+10FFFF) in Unicode. UTF-32 could theoretically encode up to a hypothetical U+FFFFFFFF and still be fixed-width. Note that, at present, only 4 of the 17 planes have defined characters (Planes 0, 1, 2, and 14), two are reserved for private use (15 and 16), and an additional is unused but is thought to be needed (Plane 3, the TIP for historic Chinese script predecessors). Four planes appear to be sufficient to support every script ever written on Earth, as it's doubtful there are unidentified scripts with an ideographic repertoire as massive as the Unified CJK ideographs database. We are very unlikely to ever fill up the current space of Unicode, let alone the plausible maximum space permissible by UTF-8, let alone the plausible maximum space permissible by UTF-32.
- XaspR8d 10y ago
  If anything, their adaptability gives me confidence. They have little power to stop vendors from creating new emojis that are morphologically distinct from existing ones, so they might as well wrangle them into a standard.
- masklinn 10y ago
  > Unicode is fucked. All these bullshit emojis Ah yes, all those bloody emoji taking the place of better worthier characters, those dastardly pictures taking up all of one half of one 16th of one Unicode plane (which has only 16 of those, and only 14 public). And the gall they have, actually being used and lighting up their section of plane 1 like a christmas tree while the rest of the plane lies in the darkness: http://reedbeta.com/blog/programmers-intro-to-unicode/heatmap-wiki+tweets.png http://reedbeta.com/blog/programmers-intro-to-unicode/heatma... what a disgrace, not only existing but being found useful, what has the world come to. And then of course there's the technical side of things: emoji actually forced western developers — and especially anglo ones — to stop fucking up non-ASCII let alone non-BMP codepoints. I don't think it's a coincidence that MySQL finally added support for astral characters once emoji started getting prominent. In fact, I have a pet theory that the rash of combining emoji in the latest revisions is in part a vehicle to teach developers to finally stop fucking up text segmentation and stop assuming every codepoint is a grapheme cluster.
- carapace 10y ago
  Unicode is a conflation of two ideas, one good and the other impossible. The good idea is to have a standard mapping from numbers to little pictures (glyphs, symbols, kanji, ideograms, cuneiform pokings in dried clay, scratches on a rock, whatever.) This is really all ASCII was. The impossible idea is to encode human languages into bits. This can't be done and will only continue to cause heartache in those who try. ASCII had English letters but wasn't an encoding for English, although you can and everyone did and does use it for that.
  - Manishearth 10y ago
    I hate this argument every time I see it because it's invariably used in the wrong place. Yes, the goal of encoding all human languages into bits is one that's near impossible. Unicode tries, and has broken half-solutions in many places. Lots of heartache everywhere. This is completely irrelevant to the discussion here. The issue of code points not always mapping to graphemes is only an issue because programmers ignore it. It's a completely solved problem, theoretically speaking. It's necessary to be able to handle many scripts, but it's not something that "breaks" unicode.
    - carapace 10y ago
      > It's a completely solved problem, theoretically speaking. lol. Unicode was ambitious for its time, but naive. Today we know better. It "jumped the shark" when the pizza slice showed up and has only been getting stupider since. Eventually it will go the way of XML (yes, I know XML hasn't gone anywhere, shut up) and we will be using some JSON hottness (forgive the labored metaphor please!) that probably consist of a wad of per-language standards and ML/AI/NLP stuff, etc.. blah blah hand-wave.) Unicode just sucks.
      - Manishearth 10y ago
        Again. Yes, "it jumped the shark when the pizza slice showed up". However, that doesn't imply that it did everything wrong. The notion of multi-codepoint characters is necessary to handle other languages. that is a solved problem, it's just that programmers mess up when dealing with it. Emoji may be a mistake, but the underlying "problems" caused by emoji existed anyway, and they're not really problems, just programmers being stupid. We had multiple per-language encodings. It sucked.
danbruc 10y ago
The Unicode standard describes in Annex 29 [1] how to properly split strings into grapheme clusters. And here [2] is a JavaScript implementation. This is a solved problem. [1] http://www.unicode.org/reports/tr29/ http://www.unicode.org/reports/tr29/ [2] https://github.com/orling/grapheme-splitter https://github.com/orling/grapheme-splitter
- newtang 10y ago
  I'm the author. Thank you for you sharing! I will check it out shortly.
- Joeri 10y ago
  This is most definitely not a solved problem, because graphemes (visual symbols) are a poor way to deal with unicode in the real world. Pretty much all systems either deal with the length in bytes (if they're old-style C), in code units / byte pairs (if they're UTF-16 based, like windows, java and javascript), or in unicode code points (if they're UTF-8 based, like every proper system should be). Dealing with the length in visual symbols is actually pretty much impossible in practice because databases won't let you define field lengths in graphemes. The way things compose: bytes combine into code points (unicode numbers), and code points combine into graphemes (visual symbols). In UTF-16 for legacy compatibility reasons with UCS-2, code points decompose into code units (byte pairs), and high code points, which need a lot of bits to represent their number, need two code units (4 bytes) instead of one. Java and JavaScript are UTF-16 based, so they measure length in code units and not code points. An emoji code point can be a low or high number depending on when it was added. Low numbers can be stored in two bytes, high numbers need four bytes. So an emoji can have length 1 or 2 in UTF-16. However, when moving to the database it will typically be stored in UTF-8, and the field length will be code points, not code units. So, that emoji will have a length of 1 regardless of whether it is low or high. You don't notice this as a problem because app-level field length checks will return a bigger number than what the database perceives, so no field length limits are exceeded. There isn't any such thing as "characters" in code. In documentation when they say "characters" usually they mean bytes, code units or code points. Almost never do they mean graphemes, which is intuitively what people think they mean. The bottom line is two-fold: (A) always understand what is meant in documentation by "length in characters", because it almost never means the intuitive thing, and (B) don't try to use graphemes as your unit of length, it won't work in practice.
wcummings 10y ago
> Sometimes, I think people come up with these names just to add excitement to their lives. Let's get outta here guys, we've been rumbled!
darkengine 10y ago
The thing that frustrates me the most about Unicode emoji is the astounding number of combining characters. For combining characters in written languages, you can do an NFC normalization and, with moderate success, get a 1 codepoint = 1 grapheme mapping, but "Emoji 2.0" introduced some ridiculous emoji compositions with the ZWJ character. To use the author's example: ‍woman - 1 codepoint black woman - 2 codepoints, woman + dark Fitzpatrick modifier ‍️‍‍woman kissing woman - 7 codepoints, woman + ZWJ + heart + ZWJ + kissy lips + ZWJ + woman It's like composing Mayan pictographs, except you have to include an invisible character in between each component. Here's another fun one: country flags. Unicode has special characters 🇱 🇮 🇰 🇪 🇹 🇭 🇮 🇸 that you can combine into country codes to create a flag. 🇰+🇷 = 🇰🇷 edit: looks like HN strips emoji? Changed the emoji in the example into English words. They are all supposed to render as a single "character".
- melloclello 10y ago
  Man, imagine if you could compose chinese characters out of radicals like this. I'm not sure if that would be a good thing or a bad thing.
  - darkengine 10y ago
    Somebody involved with Unicode must have had the same idea, because the ideographic description characters exist. However, I've never seen them used in practice because they don't actually render the character. You just get something like ⿰扌足, which corresponds to 捉. https://en.wikipedia.org/wiki/Ideographic_Description_Characters_(Unicode_block) https://en.wikipedia.org/wiki/Ideographic_Description_Charac...
    - Manishearth 10y ago
      They're not supposed to render, it's purely for describing text. As are the interlinear ruby annotations.
      - rspeer 10y ago
openasocket 10y ago
The issue doesn't really seem to be the emojis, but rather the variation sequences, which seem to be really awkward to work with, but I can sort of see why they're necessary. But the fact that we need special libraries to answer fairly basic queries about unicode text doesn't bode well.
- masklinn 10y ago
  > But the fact that we need special libraries to answer fairly basic queries about unicode text doesn't bode well. That's always been needed to actually properly work with unicode, what do you think ICU is? Few if any languages have complete native Unicode support. And it's hardly new, Unicode has an annex (#29) dedicated to text segmentation: http://www.unicode.org/reports/tr29/ http://www.unicode.org/reports/tr29/
sorenjan 10y ago
I think the article "A Programmer's Introduction to Unicode" that was shared here recently is a good read and explains Unicode well. https://news.ycombinator.com/item?id=13790575 https://news.ycombinator.com/item?id=13790575
Animats 10y ago
Before emoji, fonts and colors were independent. Combining the two creates a mess. Try using emoji in an editor with syntax coloring. We got into this because some people thought that single-color emoji were racist.[1] So now there are five skin tone options. The no-option case is usually rendered as bright yellow, which comes from the old AOL client. They got it from the happy-face icon of the 1970s. Here's the current list of valid emoji, including upcoming ones being added in the next revision.[2] A reasonable test for passwords is to run them through an IDNA checker, which checks whether a string is acceptable as a domain name component. This catches most weird stuff, such as mixed left-to-right and right-to-left symbols, zero-width markers, homoglyphs, and emoji. [1] https://www.washingtonpost.com/news/the-intersect/wp/2015/02/24/are-apples-new-yellow-face-emoji-racist/?utm_term=.ec65e2f8ef2d https://www.washingtonpost.com/news/the-intersect/wp/2015/02... [2] http://unicode.org/emoji/charts-beta/full-emoji-list.html http://unicode.org/emoji/charts-beta/full-emoji-list.html
- Exofunctor 10y ago
  It's interesting that in certain subcultures, including a large portion of the tech community, things that are non-racial are now considered racist. Not only do you have to be "racially aware", as they say, but you have to be racially aware in the right way. Being "colorblind" isn't enough anymore. Even the emoticons have to express race! Perhaps predictably, this has backfired and will continue to backfire quite spectacularly; it turns out that when you force people to start thinking along racial lines, they might not end up with the exact same ideas about race that you have. I suspect this may be a large contributing factor behind the recent resurgence of ethno-nationalism (see the Alt Right et al.).
- masklinn 10y ago
  > Before emoji, fonts and colors were independent. Fonts and colors are still, for the most part, independent. Color or the lack thereof is a property of your font and text rendering subsystems. For instance Noto Emoji provides B&W emoji, and Noto Color Emoji provides colored ones.
- Manishearth 10y ago
  Unicode does not require fonts to use color, and the spec does try to deal with the case where you don't want to use color; explicitly talking about black-and-white renderings in multiple places. This is no different for the skin tone modifiers, it's perfectly okay to fall back to a greyscale emoji (indeed, it might make sense to render all emoji in greyscale or B&W in a text editor).
jtymann 10y ago
Makes me wonder whether or not that should be considered a bug.
- Manishearth 10y ago
  I'm sure all browser designers out there would love it if we could switch JS over to UTF8, or in general have any system where JS uses a well formed encoding when it comes to unicode. We can't, because of backwards compatability.
mhils 10y ago
The Zero-Width-Joiner allows for some really strange things: https://blog.emojipedia.org/ninja-cat-the-windows-only-emoji/ https://blog.emojipedia.org/ninja-cat-the-windows-only-emoji.... One can basically achieve an unlimited number of emojis by concatenating the current ones.
kmill 10y ago
There are multiple ways of counting "length" of a string. Number of UTF-8 bytes, number of UTF-16 code units, number of codepoints, number of grapheme clusters. These are all distinct yet valid concepts of "length." For the purpose of allocating buffers, I can see the obvious use in knowing number of bytes, UTF-16 code units, or the number of codepoints. I also see the use in being able to iterate through grapheme clusters, for instance for rendering a fragment of text, or for parsing. Perhaps someone can shed light on a compelling use case for knowing the number of grapheme clusters in a particular string, because I haven't been able to think of one. I'm not sure about calculating password lengths: if the point is entropy, the number of bytes seems good enough to me! The password field bug is possibly compelling, but I don't think it's obvious what a password field should do. Should it represent keystrokes? Codepoints? Grapheme clusters? Ligatures? Replace all the glyphs with bullets during font rendering? (Similarly, perhaps someone could explain why they think reversing a string should be a sensible operation. That this is hard to do is something I occasionally hear echoing around the internet. The best I've heard is that you can reuse the default forward lexicographic ordering on reversed strings for a use I've forgotten.)
- deleted 10y ago
  [deleted]
- jfoutz 10y ago
  It's a lot like equality. Same pointer? Same value? p and q point to different nodes in a circular list. Does p equal q? Semantics matter a lot.
  - kmill 10y ago
    To expand on this point, one resolution to the Ship of Theseus problem is that the point at which the ship stops being the "same" ship depends on how you are going to define "same." "Same" could mean different things depending on what you are trying to do, so this isn't just an it's-just-semantics cop-out. In particular, to borrow something Ravi Vakil once said, a definition is worthless unless it has a use (which in his case, as a mathematician, if it can be used to uncover and prove a theorem). This is what I have in mind: I do not think it is worthwhile to worry about "the true length of a Unicode string" unless there is something you could do if only you could compute it, and I've been trying to think of something but have come up short. Speaking of equality: in a lecture about logic I once gave, I asked the students whether {1,2} and {1,2} were the same. In a very real sense, they are different because I drew them (or typed them) in different places and slightly differently -- I promise I typed the second {1,2} with different fingers. But, through the lens of same-means-same-elements, they are the same. That is a warmup for {1,2} vs {1,1,2}, and {1,2} vs {n : n is a natural number and 1 <= n <= 2}. (There's also kind of a joke about how my set of natural numbers might be red and your set of natural numbers might be blue, but the theory of sets doesn't care about the difference.)
hwc 10y ago
How can that entire article never mention the term UTF-16?
- Retr0spectrum 10y ago
  Why should it? Other than for explaining why the abomination of surrogate pairs came into existence.
ge0rg 10y ago
I've gone through exactly the same discovery process when implementing faux stamps (something between images and Emoji) in my xmpp app yesterday. My idea was to increase the font size of a message that only consists of Emoji, depending on the number of Emoji in the message, like this: https://xmpp.pix-art.de/imagehost/display/file/2017-03-09_09-36-09_r8m468so4vh7.jpg https://xmpp.pix-art.de/imagehost/display/file/2017-03-09_09... The code turned out more complex than first expected, mirroring the same problems OP encountered: https://github.com/ge0rg/yaxim/blob/gradle/src/org/yaxim/androidclient/util/XMPPHelper.java#L66-L93 https://github.com/ge0rg/yaxim/blob/gradle/src/org/yaxim/and...
- kalleboo 10y ago
  I'm working on a project that has to handle special rendering of emoji as well, and I simply ask the system "will this string render in the emoji font" and "how big of a rect do I need to render this string" to calculate the same thing, rather than trying to handle it myself and relying on assumptions about the sizing of the emoji. I figure this way I also future proof against whatever emoji they think up in the future.
remx 10y ago
Just going to leave this link here: https://mathiasbynens.be/notes/javascript-unicode https://mathiasbynens.be/notes/javascript-unicode
tantalor 10y ago
> I have no idea if there’s a good reason for the name “astral plane.” Sometimes, I think people come up with these names just to add excitement to their lives. https://en.wikipedia.org/wiki/Plane_(esotericism)#The_Planes https://en.wikipedia.org/wiki/Plane_(esotericism)#The_Planes
chungy 10y ago
> The current largest codepoint? Why that would be a cheese wedge at U+1F9C0. How did we ever communicate before this? Sounds cute, but inaccurate. If we count the last two planes that are reserved for private use (aka, applications/users can use them for whatever domain problems they like), that would be U+10FFFD. If we count the variation selector codepoints (used for things like changing skin tone, or the look of certain other characters), U+E01EF. If we count the last honestly-for-real-written-language character assigned, it would be 𪘀 U+2FA1D CJK COMPATIBILITY IDEOGRAPH-2FA. But I suppose none of that sounds as fun as an emoji (which are really a very small part of the Unicode standard).
- rspeer 10y ago
  I tried to look up what U+2FA1D, the highest-numbered printable character, means in context. It is a Traditional Chinese character. It's a variant of U+2F600, 𪘀, which is pronounced "pián". It apparently is used in zero words. It's in Unicode because it's listed in the 7th section of TCA-CNS 11643-1992, a Taiwanese computing standard. Searching for it gives lots of sites that acknowledge that it's a character that exists and then provide no definition for it. My guess: it occurred in someone's name at some point. Pretty strange that it ended up requiring a compatibility mapping, though, when nobody seems to use the character or the character it's mapped to!
- deleted 10y ago
  [deleted]
- newtang 10y ago
  You're right, thank you! I'll add an edit.
zach417 10y ago
Tom Scott did a nice YouTube video related to this: https://www.youtube.com/watch?v=sTzp76JXsoY https://www.youtube.com/watch?v=sTzp76JXsoY
TorKlingberg 10y ago
If you want to do Unicode correctly, you shouldn't ask for the "length" of a string. The is no true definition of length. If want to know how many bytes it uses in storage, ask for that. If you want to know how wide it will be on the screen, ask for that. Do not iterate over strings character by character.
- deathanatos 10y ago
  I'd avoid the term "character", but I'd argue there are valid reasons to consume a Unicode string grapheme by grapheme. For example, a regex engine trying to match "e + combining_acute_accent" wants to match both the pre-combined version and the version that uses combining characters. The main thrust of your point — that "length" without clarification of what measure of length is meaningless — I agree with.
- fryguy 10y ago
  How many dots/stars should one display for a password? That's a question that can't be answered by your two valid question. Are you suggesting that dots/stars shouldn't be displayed for passwords, since you can't ask how many "characters" it is?
  - toast0 10y ago
    If you have an entry indicator, it should probably be about the same width as the entered text; or if you're concerned about leaking precise length information for fields that aren't monospaced, you could add a dot each time the rendered text would increase in width.
  - slededit 10y ago
    You could divide the length of the string by the length the '*' character in a monospaced font. It doesn't really make sense for a combining or other invisible character to get its own asterisk.
nutbutter 10y ago
The golf course flag equals one obviously because at a hole-in-one. :)
codezero 10y ago
I see your 2 and raise you 2: "(this is a color-hued hand from Apple that doesn't render on HN)".length == 4 I ran into the length==2 bug when truncating some text, it led to errors trying to url encode a string :) The author's `fancyCount2` still returns a size of 2 for these kinds of emoji, but I'm not too surprised.
gtrubetskoy 10y ago
In Go: func main() { shit := "\U0001f4a9" fmt.Printf("len of %s is %d\n", shit, utf8.RuneCountInString(shit)) } $ len of � is 1 Though I can't say that this is all that intuitive either...
- geocar 10y ago
  Codepoints still aren't the same as characters. Consider the examples given about combining emoji; Consider two runes that make one character: e and ◌́
teknologist 10y ago
This appears to be a rehash of what Mathias Bynens was talking about a few years ago. http://vimeo.com/76597193 http://vimeo.com/76597193 https://mathiasbynens.be/notes/javascript-unicode https://mathiasbynens.be/notes/javascript-unicode
rsmets 10y ago
(U+200B), zero width space, should be outlawed... got me good a couple years ago! Had todo a hexdump to see what was going on.
TheRealPomax 10y ago
but the real question is why he needed password length constraints instead of password strength constraints...
beaugunderson 10y ago
lodash's toArray and split both support emoji, with good unit tests. I also wrote emoji-aware for this purpose: https://www.npmjs.com/package/emoji-aware https://www.npmjs.com/package/emoji-aware
deleted 10y ago
[deleted]
joeblau 10y ago
I ran into this 2 years ago on Swift when I was creating an emojified version of Twitter. I wanted to ensure that each message sent had at least 1 emoji and I quickly realized that validating a string with 1 emoji was not as simple as: if (lastString.characters.count == 2) { // pseudo code to allow string and activate send button } This was the app I was working on [1]; code is finished, but I'm not launching it (probably ever). The whole emoji length piece was quite frustrating because my assumption of character counting went right out of the window when I had people testing the app in Test Flight. [1] - https://joeblau.com/emo/ https://joeblau.com/emo/
- Manishearth 10y ago
  Actually, this is just due to Swift not implementing Unicode 9's version of UAX 29 (which had just come out at the time). Swift should handle it correctly, but it's lagging behind in unicode 9 support. In general a "character" in a string is a grapheme cluster, and most visually-single emoji are single grapheme clusters. The exception is stuff like ‍️[1]. That should render as a male judge (I don't think there's font support for it yet) according to the spec, and it should be a single grapheme cluster, but the spec has what I consider a mistake in it where it isn't considered to be one. I've filed a bug about this, since the emoji-zwj-sequences file lists it as a valid zwj sequence, but applying the spec to the sequence gives two grapheme clusters. There's active work now for Unicode 9 support in Swift. Since string handling is heavily dependent on this algorithm (they have a unicode trie and all for optimization!) it's trickier than just rewriting the algorithm. But, in general, you should be able to trust Swift to do the right thing here, barring bugs like "not up to date with the spec". Swift is great like that. [1]: https://r12a.github.io/uniview/?charlist=%F0%9F%91%A8%F0%9F%8F%BB%E2%80%8D%E2%9A%96%EF%B8%8F https://r12a.github.io/uniview/?charlist=%F0%9F%91%A8%F0%9F%...
xem 10y ago
Here are my 2 cents: you can decompose an Unicode string with the ES6 spread operator: [..."(insert 5 poo emoji here)"].length === 5 [..."(insert 5 poo emoji here)"][1] === "(poo emoji)"
Traubenfuchs 10y ago
If this interests you, read the source of Java's abstractStringBuilder.reverse(). It's interesting and very short. I am not sure it can deal with multi-emoji-emoji though.
lsv1 10y ago
As a developer dealing with the encoding of user input made in UTF-8 into a legacy systems which only support ASCII... I prefer this.
pc2g4d 10y ago
Just ran into this yesterday when I discovered that an emoji character wouldn't fit into Rust's `char` type. I just changed the type to `&'static str` but I still wish there was a single `grapheme` type or something like that.
marichards 10y ago
create table twitter(tweet varchar(? ... that's it, I give up, time to become an Uber driver

279 points by stanzheng 10y ago

Unicode is fucked. All these bullshit emojis remind me of the 1980s when ASCII was 7 bits but every computer manufacturer (Atari, Commodore, Apple, IBM, TI, etc...) made their own set of characters for the 128 values of a byte beyond ASCII. Of course Unicode is a global standard so your pile-of-poop emoji will still be a pile-of-poop on every device even if the amount of steam is different for some people. It's beyond me why this is happening. Who decides which bullshit symbols get into the standard anyway?

raphlinus 10y ago
Meet the shadowy overlords who approve emojis[0] [0] http://www.latimes.com/business/technology/la-fi-tn-emoji-q-and-a-20160125-htmlstory.html http://www.latimes.com/business/technology/la-fi-tn-emoji-q-...
sdegutis 10y ago
Language is inherently complex, there's no way to solve this in any "cleaner" way than what we already came up with. Unfortunately the best way forward is to build up what we already have and cover all the warts with wrapper functions/libraries.
- phkahler 10y ago
  And where are the sex emoji? The dirtiest thing I've been able to text is a heart and a pair of handcuffs ;-)
  - masklinn 10y ago
    The Love Hotel (U+1F3E9) is rather obvious, maybe the kiss mark (U+1F48B) as well, though the raunchiest ones (in actual use) are a bit more… discreet?: the aubergine (U+1F346) and splashing "sweat" (U+1F4A6).
    - pbhjpbhj 10y ago
      On Ubuntu, and probably other OS/distro too, ctrl+alt+shift+U gives you a underline-u symbol, type in the Unicode and then press <enter>.
      - masklinn 10y ago
        I'm not sure what use that would be? U+<hex> is a normal way to designate codepoints, and I can't put the actual emoji in the comment as they're stripped on submission.
        pbhjpbhj 10y ago
        Ah, sorry, that comment was for those playing along who might want to see the listed glyphs.
  - tveita 10y ago
    Available for over 2000 years now. https://mobile.twitter.com/unicode/status/722133439726505984 https://mobile.twitter.com/unicode/status/722133439726505984
- PeterisP 10y ago
  Well, there is one way, we can simplify and standardize format of language. Unfortunately that requires generations of "reeducation", so it's not a viable solution in the short term - but it does seem possible that this is where languages are going in the next few centuries, as globalization, easier travel and more interrelated communities are likely to result in slow, gradual convergence to less languages as many of the current 6000+ languages cease to be used in practice.
  - Manishearth 10y ago
    I am always surprised when people think that the solution to "dealing with language in programming is complex" is "let's reeducate the world by changing their language" instead of "let's reeducate programmers".
TAForObvReasons 10y ago
There is a Unicode encoding "UTF-32" which has the advantage of being fixed width. This is not popular for the obvious reason that even ascii characters are expanded to 4 bytes. Additionally the windows APIs, among other interfaces, are not equipped to handle 4-byte codepages.
- raphlinus 10y ago
  It's fixed width with respect to code points, but not with respect to any of the other things mentioned in the linked article. For example, the black heart with emoji variation selector (which makes it render red) is two code points.
- Manishearth 10y ago
  Being fixed width is not an advantage. Code points aren't a very useful unit of text outside of the implementation of algorithms defined by unicode. All of these algorithms generally require iteration anyway. O(1) code point indexing is nearly useless. http://manishearth.github.io/blog/2017/01/14/stop-ascribing-meaning-to-unicode-code-points/ http://manishearth.github.io/blog/2017/01/14/stop-ascribing-...
- marcosdumay 10y ago
  > "UTF-32" which has the advantage of being fixed width It's fixed width for now. It can not hold all the current available code-points, so it will probably have the same fate as UTF-16 (but it will probably take a long time). I'd stay away from it.
  - jcranmer 10y ago
    There are currently 17 × 65536 code points (U+0000..U+10FFFF) in Unicode. UTF-32 could theoretically encode up to a hypothetical U+FFFFFFFF and still be fixed-width. Note that, at present, only 4 of the 17 planes have defined characters (Planes 0, 1, 2, and 14), two are reserved for private use (15 and 16), and an additional is unused but is thought to be needed (Plane 3, the TIP for historic Chinese script predecessors). Four planes appear to be sufficient to support every script ever written on Earth, as it's doubtful there are unidentified scripts with an ideographic repertoire as massive as the Unified CJK ideographs database. We are very unlikely to ever fill up the current space of Unicode, let alone the plausible maximum space permissible by UTF-8, let alone the plausible maximum space permissible by UTF-32.
XaspR8d 10y ago
If anything, their adaptability gives me confidence. They have little power to stop vendors from creating new emojis that are morphologically distinct from existing ones, so they might as well wrangle them into a standard.
masklinn 10y ago
> Unicode is fucked. All these bullshit emojis Ah yes, all those bloody emoji taking the place of better worthier characters, those dastardly pictures taking up all of one half of one 16th of one Unicode plane (which has only 16 of those, and only 14 public). And the gall they have, actually being used and lighting up their section of plane 1 like a christmas tree while the rest of the plane lies in the darkness: http://reedbeta.com/blog/programmers-intro-to-unicode/heatmap-wiki+tweets.png http://reedbeta.com/blog/programmers-intro-to-unicode/heatma... what a disgrace, not only existing but being found useful, what has the world come to. And then of course there's the technical side of things: emoji actually forced western developers — and especially anglo ones — to stop fucking up non-ASCII let alone non-BMP codepoints. I don't think it's a coincidence that MySQL finally added support for astral characters once emoji started getting prominent. In fact, I have a pet theory that the rash of combining emoji in the latest revisions is in part a vehicle to teach developers to finally stop fucking up text segmentation and stop assuming every codepoint is a grapheme cluster.
carapace 10y ago
Unicode is a conflation of two ideas, one good and the other impossible. The good idea is to have a standard mapping from numbers to little pictures (glyphs, symbols, kanji, ideograms, cuneiform pokings in dried clay, scratches on a rock, whatever.) This is really all ASCII was. The impossible idea is to encode human languages into bits. This can't be done and will only continue to cause heartache in those who try. ASCII had English letters but wasn't an encoding for English, although you can and everyone did and does use it for that.
- Manishearth 10y ago
  I hate this argument every time I see it because it's invariably used in the wrong place. Yes, the goal of encoding all human languages into bits is one that's near impossible. Unicode tries, and has broken half-solutions in many places. Lots of heartache everywhere. This is completely irrelevant to the discussion here. The issue of code points not always mapping to graphemes is only an issue because programmers ignore it. It's a completely solved problem, theoretically speaking. It's necessary to be able to handle many scripts, but it's not something that "breaks" unicode.
  - carapace 10y ago
    > It's a completely solved problem, theoretically speaking. lol. Unicode was ambitious for its time, but naive. Today we know better. It "jumped the shark" when the pizza slice showed up and has only been getting stupider since. Eventually it will go the way of XML (yes, I know XML hasn't gone anywhere, shut up) and we will be using some JSON hottness (forgive the labored metaphor please!) that probably consist of a wad of per-language standards and ML/AI/NLP stuff, etc.. blah blah hand-wave.) Unicode just sucks.
    - Manishearth 10y ago
      Again. Yes, "it jumped the shark when the pizza slice showed up". However, that doesn't imply that it did everything wrong. The notion of multi-codepoint characters is necessary to handle other languages. that is a solved problem, it's just that programmers mess up when dealing with it. Emoji may be a mistake, but the underlying "problems" caused by emoji existed anyway, and they're not really problems, just programmers being stupid. We had multiple per-language encodings. It sucked.

danbruc 10y ago

The Unicode standard describes in Annex 29 [1] how to properly split strings into grapheme clusters. And here [2] is a JavaScript implementation. This is a solved problem. [1] http://www.unicode.org/reports/tr29/ http://www.unicode.org/reports/tr29/ [2] https://github.com/orling/grapheme-splitter https://github.com/orling/grapheme-splitter

newtang 10y ago
I'm the author. Thank you for you sharing! I will check it out shortly.
Joeri 10y ago
This is most definitely not a solved problem, because graphemes (visual symbols) are a poor way to deal with unicode in the real world. Pretty much all systems either deal with the length in bytes (if they're old-style C), in code units / byte pairs (if they're UTF-16 based, like windows, java and javascript), or in unicode code points (if they're UTF-8 based, like every proper system should be). Dealing with the length in visual symbols is actually pretty much impossible in practice because databases won't let you define field lengths in graphemes. The way things compose: bytes combine into code points (unicode numbers), and code points combine into graphemes (visual symbols). In UTF-16 for legacy compatibility reasons with UCS-2, code points decompose into code units (byte pairs), and high code points, which need a lot of bits to represent their number, need two code units (4 bytes) instead of one. Java and JavaScript are UTF-16 based, so they measure length in code units and not code points. An emoji code point can be a low or high number depending on when it was added. Low numbers can be stored in two bytes, high numbers need four bytes. So an emoji can have length 1 or 2 in UTF-16. However, when moving to the database it will typically be stored in UTF-8, and the field length will be code points, not code units. So, that emoji will have a length of 1 regardless of whether it is low or high. You don't notice this as a problem because app-level field length checks will return a bigger number than what the database perceives, so no field length limits are exceeded. There isn't any such thing as "characters" in code. In documentation when they say "characters" usually they mean bytes, code units or code points. Almost never do they mean graphemes, which is intuitively what people think they mean. The bottom line is two-fold: (A) always understand what is meant in documentation by "length in characters", because it almost never means the intuitive thing, and (B) don't try to use graphemes as your unit of length, it won't work in practice.

wcummings 10y ago

> Sometimes, I think people come up with these names just to add excitement to their lives. Let's get outta here guys, we've been rumbled!

darkengine 10y ago

The thing that frustrates me the most about Unicode emoji is the astounding number of combining characters. For combining characters in written languages, you can do an NFC normalization and, with moderate success, get a 1 codepoint = 1 grapheme mapping, but "Emoji 2.0" introduced some ridiculous emoji compositions with the ZWJ character. To use the author's example: ‍woman - 1 codepoint black woman - 2 codepoints, woman + dark Fitzpatrick modifier ‍️‍‍woman kissing woman - 7 codepoints, woman + ZWJ + heart + ZWJ + kissy lips + ZWJ + woman It's like composing Mayan pictographs, except you have to include an invisible character in between each component. Here's another fun one: country flags. Unicode has special characters 🇱 🇮 🇰 🇪 🇹 🇭 🇮 🇸 that you can combine into country codes to create a flag. 🇰+🇷 = 🇰🇷 edit: looks like HN strips emoji? Changed the emoji in the example into English words. They are all supposed to render as a single "character".

melloclello 10y ago
Man, imagine if you could compose chinese characters out of radicals like this. I'm not sure if that would be a good thing or a bad thing.
- darkengine 10y ago
  Somebody involved with Unicode must have had the same idea, because the ideographic description characters exist. However, I've never seen them used in practice because they don't actually render the character. You just get something like ⿰扌足, which corresponds to 捉. https://en.wikipedia.org/wiki/Ideographic_Description_Characters_(Unicode_block) https://en.wikipedia.org/wiki/Ideographic_Description_Charac...
  - Manishearth 10y ago
    They're not supposed to render, it's purely for describing text. As are the interlinear ruby annotations.
    - rspeer 10y ago

openasocket 10y ago

The issue doesn't really seem to be the emojis, but rather the variation sequences, which seem to be really awkward to work with, but I can sort of see why they're necessary. But the fact that we need special libraries to answer fairly basic queries about unicode text doesn't bode well.

masklinn 10y ago
> But the fact that we need special libraries to answer fairly basic queries about unicode text doesn't bode well. That's always been needed to actually properly work with unicode, what do you think ICU is? Few if any languages have complete native Unicode support. And it's hardly new, Unicode has an annex (#29) dedicated to text segmentation: http://www.unicode.org/reports/tr29/ http://www.unicode.org/reports/tr29/

sorenjan 10y ago

I think the article "A Programmer's Introduction to Unicode" that was shared here recently is a good read and explains Unicode well. https://news.ycombinator.com/item?id=13790575 https://news.ycombinator.com/item?id=13790575

Animats 10y ago

Before emoji, fonts and colors were independent. Combining the two creates a mess. Try using emoji in an editor with syntax coloring. We got into this because some people thought that single-color emoji were racist.[1] So now there are five skin tone options. The no-option case is usually rendered as bright yellow, which comes from the old AOL client. They got it from the happy-face icon of the 1970s. Here's the current list of valid emoji, including upcoming ones being added in the next revision.[2] A reasonable test for passwords is to run them through an IDNA checker, which checks whether a string is acceptable as a domain name component. This catches most weird stuff, such as mixed left-to-right and right-to-left symbols, zero-width markers, homoglyphs, and emoji. [1] https://www.washingtonpost.com/news/the-intersect/wp/2015/02/24/are-apples-new-yellow-face-emoji-racist/?utm_term=.ec65e2f8ef2d https://www.washingtonpost.com/news/the-intersect/wp/2015/02... [2] http://unicode.org/emoji/charts-beta/full-emoji-list.html http://unicode.org/emoji/charts-beta/full-emoji-list.html

Exofunctor 10y ago
It's interesting that in certain subcultures, including a large portion of the tech community, things that are non-racial are now considered racist. Not only do you have to be "racially aware", as they say, but you have to be racially aware in the right way. Being "colorblind" isn't enough anymore. Even the emoticons have to express race! Perhaps predictably, this has backfired and will continue to backfire quite spectacularly; it turns out that when you force people to start thinking along racial lines, they might not end up with the exact same ideas about race that you have. I suspect this may be a large contributing factor behind the recent resurgence of ethno-nationalism (see the Alt Right et al.).
masklinn 10y ago
> Before emoji, fonts and colors were independent. Fonts and colors are still, for the most part, independent. Color or the lack thereof is a property of your font and text rendering subsystems. For instance Noto Emoji provides B&W emoji, and Noto Color Emoji provides colored ones.
Manishearth 10y ago
Unicode does not require fonts to use color, and the spec does try to deal with the case where you don't want to use color; explicitly talking about black-and-white renderings in multiple places. This is no different for the skin tone modifiers, it's perfectly okay to fall back to a greyscale emoji (indeed, it might make sense to render all emoji in greyscale or B&W in a text editor).

jtymann 10y ago

Makes me wonder whether or not that should be considered a bug.

Manishearth 10y ago
I'm sure all browser designers out there would love it if we could switch JS over to UTF8, or in general have any system where JS uses a well formed encoding when it comes to unicode. We can't, because of backwards compatability.

mhils 10y ago

The Zero-Width-Joiner allows for some really strange things: https://blog.emojipedia.org/ninja-cat-the-windows-only-emoji/ https://blog.emojipedia.org/ninja-cat-the-windows-only-emoji.... One can basically achieve an unlimited number of emojis by concatenating the current ones.

kmill 10y ago

There are multiple ways of counting "length" of a string. Number of UTF-8 bytes, number of UTF-16 code units, number of codepoints, number of grapheme clusters. These are all distinct yet valid concepts of "length." For the purpose of allocating buffers, I can see the obvious use in knowing number of bytes, UTF-16 code units, or the number of codepoints. I also see the use in being able to iterate through grapheme clusters, for instance for rendering a fragment of text, or for parsing. Perhaps someone can shed light on a compelling use case for knowing the number of grapheme clusters in a particular string, because I haven't been able to think of one. I'm not sure about calculating password lengths: if the point is entropy, the number of bytes seems good enough to me! The password field bug is possibly compelling, but I don't think it's obvious what a password field should do. Should it represent keystrokes? Codepoints? Grapheme clusters? Ligatures? Replace all the glyphs with bullets during font rendering? (Similarly, perhaps someone could explain why they think reversing a string should be a sensible operation. That this is hard to do is something I occasionally hear echoing around the internet. The best I've heard is that you can reuse the default forward lexicographic ordering on reversed strings for a use I've forgotten.)

deleted 10y ago
[deleted]
jfoutz 10y ago
It's a lot like equality. Same pointer? Same value? p and q point to different nodes in a circular list. Does p equal q? Semantics matter a lot.
- kmill 10y ago
  To expand on this point, one resolution to the Ship of Theseus problem is that the point at which the ship stops being the "same" ship depends on how you are going to define "same." "Same" could mean different things depending on what you are trying to do, so this isn't just an it's-just-semantics cop-out. In particular, to borrow something Ravi Vakil once said, a definition is worthless unless it has a use (which in his case, as a mathematician, if it can be used to uncover and prove a theorem). This is what I have in mind: I do not think it is worthwhile to worry about "the true length of a Unicode string" unless there is something you could do if only you could compute it, and I've been trying to think of something but have come up short. Speaking of equality: in a lecture about logic I once gave, I asked the students whether {1,2} and {1,2} were the same. In a very real sense, they are different because I drew them (or typed them) in different places and slightly differently -- I promise I typed the second {1,2} with different fingers. But, through the lens of same-means-same-elements, they are the same. That is a warmup for {1,2} vs {1,1,2}, and {1,2} vs {n : n is a natural number and 1 <= n <= 2}. (There's also kind of a joke about how my set of natural numbers might be red and your set of natural numbers might be blue, but the theory of sets doesn't care about the difference.)

hwc 10y ago

How can that entire article never mention the term UTF-16?

Retr0spectrum 10y ago
Why should it? Other than for explaining why the abomination of surrogate pairs came into existence.

ge0rg 10y ago

I've gone through exactly the same discovery process when implementing faux stamps (something between images and Emoji) in my xmpp app yesterday. My idea was to increase the font size of a message that only consists of Emoji, depending on the number of Emoji in the message, like this: https://xmpp.pix-art.de/imagehost/display/file/2017-03-09_09-36-09_r8m468so4vh7.jpg https://xmpp.pix-art.de/imagehost/display/file/2017-03-09_09... The code turned out more complex than first expected, mirroring the same problems OP encountered: https://github.com/ge0rg/yaxim/blob/gradle/src/org/yaxim/androidclient/util/XMPPHelper.java#L66-L93 https://github.com/ge0rg/yaxim/blob/gradle/src/org/yaxim/and...

kalleboo 10y ago
I'm working on a project that has to handle special rendering of emoji as well, and I simply ask the system "will this string render in the emoji font" and "how big of a rect do I need to render this string" to calculate the same thing, rather than trying to handle it myself and relying on assumptions about the sizing of the emoji. I figure this way I also future proof against whatever emoji they think up in the future.

remx 10y ago

Just going to leave this link here: https://mathiasbynens.be/notes/javascript-unicode https://mathiasbynens.be/notes/javascript-unicode

tantalor 10y ago

> I have no idea if there’s a good reason for the name “astral plane.” Sometimes, I think people come up with these names just to add excitement to their lives. https://en.wikipedia.org/wiki/Plane_(esotericism)#The_Planes https://en.wikipedia.org/wiki/Plane_(esotericism)#The_Planes

chungy 10y ago

> The current largest codepoint? Why that would be a cheese wedge at U+1F9C0. How did we ever communicate before this? Sounds cute, but inaccurate. If we count the last two planes that are reserved for private use (aka, applications/users can use them for whatever domain problems they like), that would be U+10FFFD. If we count the variation selector codepoints (used for things like changing skin tone, or the look of certain other characters), U+E01EF. If we count the last honestly-for-real-written-language character assigned, it would be 𪘀 U+2FA1D CJK COMPATIBILITY IDEOGRAPH-2FA. But I suppose none of that sounds as fun as an emoji (which are really a very small part of the Unicode standard).

rspeer 10y ago
I tried to look up what U+2FA1D, the highest-numbered printable character, means in context. It is a Traditional Chinese character. It's a variant of U+2F600, 𪘀, which is pronounced "pián". It apparently is used in zero words. It's in Unicode because it's listed in the 7th section of TCA-CNS 11643-1992, a Taiwanese computing standard. Searching for it gives lots of sites that acknowledge that it's a character that exists and then provide no definition for it. My guess: it occurred in someone's name at some point. Pretty strange that it ended up requiring a compatibility mapping, though, when nobody seems to use the character or the character it's mapped to!
deleted 10y ago
[deleted]
newtang 10y ago
You're right, thank you! I'll add an edit.

zach417 10y ago

Tom Scott did a nice YouTube video related to this: https://www.youtube.com/watch?v=sTzp76JXsoY https://www.youtube.com/watch?v=sTzp76JXsoY

TorKlingberg 10y ago

If you want to do Unicode correctly, you shouldn't ask for the "length" of a string. The is no true definition of length. If want to know how many bytes it uses in storage, ask for that. If you want to know how wide it will be on the screen, ask for that. Do not iterate over strings character by character.

deathanatos 10y ago
I'd avoid the term "character", but I'd argue there are valid reasons to consume a Unicode string grapheme by grapheme. For example, a regex engine trying to match "e + combining_acute_accent" wants to match both the pre-combined version and the version that uses combining characters. The main thrust of your point — that "length" without clarification of what measure of length is meaningless — I agree with.
fryguy 10y ago
How many dots/stars should one display for a password? That's a question that can't be answered by your two valid question. Are you suggesting that dots/stars shouldn't be displayed for passwords, since you can't ask how many "characters" it is?
- toast0 10y ago
  If you have an entry indicator, it should probably be about the same width as the entered text; or if you're concerned about leaking precise length information for fields that aren't monospaced, you could add a dot each time the rendered text would increase in width.
- slededit 10y ago
  You could divide the length of the string by the length the '*' character in a monospaced font. It doesn't really make sense for a combining or other invisible character to get its own asterisk.

nutbutter 10y ago

The golf course flag equals one obviously because at a hole-in-one. :)

codezero 10y ago

I see your 2 and raise you 2: "(this is a color-hued hand from Apple that doesn't render on HN)".length == 4 I ran into the length==2 bug when truncating some text, it led to errors trying to url encode a string :) The author's `fancyCount2` still returns a size of 2 for these kinds of emoji, but I'm not too surprised.

gtrubetskoy 10y ago

In Go: func main() { shit := "\U0001f4a9" fmt.Printf("len of %s is %d\n", shit, utf8.RuneCountInString(shit)) } $ len of � is 1 Though I can't say that this is all that intuitive either...

geocar 10y ago
Codepoints still aren't the same as characters. Consider the examples given about combining emoji; Consider two runes that make one character: e and ◌́

teknologist 10y ago

This appears to be a rehash of what Mathias Bynens was talking about a few years ago. http://vimeo.com/76597193 http://vimeo.com/76597193 https://mathiasbynens.be/notes/javascript-unicode https://mathiasbynens.be/notes/javascript-unicode

rsmets 10y ago

(U+200B), zero width space, should be outlawed... got me good a couple years ago! Had todo a hexdump to see what was going on.

TheRealPomax 10y ago

but the real question is why he needed password length constraints instead of password strength constraints...

beaugunderson 10y ago

lodash's toArray and split both support emoji, with good unit tests. I also wrote emoji-aware for this purpose: https://www.npmjs.com/package/emoji-aware https://www.npmjs.com/package/emoji-aware

deleted 10y ago

[deleted]

joeblau 10y ago

I ran into this 2 years ago on Swift when I was creating an emojified version of Twitter. I wanted to ensure that each message sent had at least 1 emoji and I quickly realized that validating a string with 1 emoji was not as simple as: if (lastString.characters.count == 2) { // pseudo code to allow string and activate send button } This was the app I was working on [1]; code is finished, but I'm not launching it (probably ever). The whole emoji length piece was quite frustrating because my assumption of character counting went right out of the window when I had people testing the app in Test Flight. [1] - https://joeblau.com/emo/ https://joeblau.com/emo/

Manishearth 10y ago
Actually, this is just due to Swift not implementing Unicode 9's version of UAX 29 (which had just come out at the time). Swift should handle it correctly, but it's lagging behind in unicode 9 support. In general a "character" in a string is a grapheme cluster, and most visually-single emoji are single grapheme clusters. The exception is stuff like ‍️[1]. That should render as a male judge (I don't think there's font support for it yet) according to the spec, and it should be a single grapheme cluster, but the spec has what I consider a mistake in it where it isn't considered to be one. I've filed a bug about this, since the emoji-zwj-sequences file lists it as a valid zwj sequence, but applying the spec to the sequence gives two grapheme clusters. There's active work now for Unicode 9 support in Swift. Since string handling is heavily dependent on this algorithm (they have a unicode trie and all for optimization!) it's trickier than just rewriting the algorithm. But, in general, you should be able to trust Swift to do the right thing here, barring bugs like "not up to date with the spec". Swift is great like that. [1]: https://r12a.github.io/uniview/?charlist=%F0%9F%91%A8%F0%9F%8F%BB%E2%80%8D%E2%9A%96%EF%B8%8F https://r12a.github.io/uniview/?charlist=%F0%9F%91%A8%F0%9F%...

xem 10y ago

Here are my 2 cents: you can decompose an Unicode string with the ES6 spread operator: [..."(insert 5 poo emoji here)"].length === 5 [..."(insert 5 poo emoji here)"][1] === "(poo emoji)"

Traubenfuchs 10y ago

If this interests you, read the source of Java's abstractStringBuilder.reverse(). It's interesting and very short. I am not sure it can deal with multi-emoji-emoji though.

lsv1 10y ago

As a developer dealing with the encoding of user input made in UTF-8 into a legacy systems which only support ASCII... I prefer this.

pc2g4d 10y ago

Just ran into this yesterday when I discovered that an emoji character wouldn't fit into Rust's `char` type. I just changed the type to `&'static str` but I still wish there was a single `grapheme` type or something like that.

marichards 10y ago

create table twitter(tweet varchar(? ... that's it, I give up, time to become an Uber driver