11 ms·

Why can't you reverse a string with a flag emoji?

189 points by da12 5y ago

WA9ACE 5y ago
I feel like I'm obligated to share this almost 20 year old Spolsky post that gave me my understanding of characters. https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
- xmprt 5y ago
  In that same vein, here's my introduction to Unicode about 10 years ago from Tom Scott. https://www.youtube.com/watch?v=MijmeoH9LT4 https://www.youtube.com/watch?v=MijmeoH9LT4
  - zerox7felf 5y ago
    poor man gave me and many others something like half of our introduction to computer science, but has gotten far more fame as the "emoji guy" for his repeated bouts with this particular part of unicode :)
  - ciupicri 5y ago
    That's more about the UTF-8 encoding than Unicode itself.
codingkev 5y ago
Yes, this allows for easy building of flag emojis as long as you know the ISO 3166 two-letter country code. Example: https://github.com/kennell/flagz/blob/master/flagz.py https://github.com/kennell/flagz/blob/master/flagz.py
exdsq 5y ago
Am I missing something or is this Day 1 of a programming course in C?
Mesopropithecus 5y ago
Unfortunately the HN text input won't let me do this, but a funny starter for the article would have been this: '(Spanish flag)'[::-1] basically ''.join([chr(127466), chr(127480)]) vs. ''.join([chr(127466), chr(127480)])[::-1] I'll add this to my collection of party tricks and show myself out. Cool article!
deleted 5y ago
[deleted]
architectdrone 5y ago
humorously, on my local machine, I only see the string "us", and was rather confused when he was asserting that it was a single character :D
- kingcharles 5y ago
  You're on Windows? Windows doesn't render flag emojis as flags.
hffftz 5y ago
jupp0r 5y ago
With all the criticism I normally have for Rust, I must say that its type safe handling of UTF-8 and its unambiguous distinction between byte strings and UTF-8 strings are extremely helpful in handling situations mentioned in the article correctly (and also efficiently). Yes it's a pain, but the way the standard library designed its types force you to handle conversions correctly, for example when byte arrays are converted to UTF-8 strings and may contain invalid UTF-8 sequences.
heystefan 5y ago
Oooh I know this one, I've read it here last year: https://tonsky.me/blog/emoji/ https://tonsky.me/blog/emoji/
zarzavat 5y ago
This reminds me of an interesting bug I saw where I was seeing a strange flag in some Arabic text. However when I copied the string and pasted it into a text editor, the flag of Saudi Arabia appeared instead (which made much more sense). After some vexillologic research on Wikipedia I identified the original flag as American Samoa and it suddenly all made sense. Turns out some broken RTL support was flipping the SA into AS at presentation.
- zarzavat 5y ago
  After writing this comment I did some more research. Apparently this is actually a bug in Chrome itself (!). https://bugs.chromium.org/p/chromium/issues/detail?id=1272436 https://bugs.chromium.org/p/chromium/issues/detail?id=127243...
kart23 5y ago
Google also interprets emojis funny. Google the Estonian and South Sudan flag (f"{chr(127466)*2+chr(127480)*2}") and you get results for Spain.
smegsicle 5y ago
did they think all those skintone emojis are individual codepoints?
- advisedwang 5y ago
  They might have thought that `reverse()` had some kind of unicode-aware handling. I believe `upper()`/`lower()` do.
- daveslash 5y ago
  When I first realized that the skin tone emojis were a code-point + a color code-point modifier, I tried to see what other colors there were and if I could apply those to other emojis. The immature child in me looked to see if there was a red color code point and if so, could I use it to make a "blood poop" emoji. Turns out.... no.
yoyohello13 5y ago
Maybe I'm missing some prerequisite knowledge here, but why would I assume `flag="us"` is an emoji? Looking at that first block of code, there is no reason for me to think "us" is a single character. Edit: Turns out my browser wasn't rendering the flags.
- Benlights 5y ago
  I had the same issue when I read the article, I kept on getting stuck and asking myself what I was missing.
- greenyoda 5y ago
  In my browser (Firefox on Windows), the thing between the quotes in the first block of code looks like a picture of the US flag cropped to a circle, not like the characters "us".
  - yoyohello13 5y ago
    Ah I see, I just opened it in firefox. It looks like some JS library is not getting loaded in Edge. The author was talking about "us", "so", etc. looking like one character and I thought I was going crazy, lol.
    - bialpio 5y ago
      Reminds me of an image that renders differently on Macs (https://www.bleepingcomputer.com/news/technology/this-image-looks-very-different-on-apple-devices-see-for-yourself/ https://www.bleepingcomputer.com/news/technology/this-image-...), I bet it'd make for a fun conversation that could make the participants question their sanity. :-)
    - jfk13 5y ago
      I don't think that's about a JS library. Firefox bundles an emoji font that supports some things -- such as the flags -- that aren't supported by Segoe UI Emoji on Windows, so it has additional coverage for such character sequences.
happytoexplain 5y ago
I guessed that it would become the USSR flag (US -> SU), but apparently Unicode doesn't define that one! I wonder why. That would have been humorous.
- ts4z 5y ago
  IIRC Unicode doesn't define country codes. It was a workaround for a political issue of which countries recognize which other countries. It would have been difficult to get the CN delegation to sign off on a list that contained TW, although there are probably others.
  - andylynch 5y ago
    There are many more than I realised - Wikipedia has a decent list https://en.m.wikipedia.org/wiki/List_of_states_with_limited_recognition https://en.m.wikipedia.org/wiki/List_of_states_with_limited_...
- chungy 5y ago
  Unicode doesn't define any flags, really. That's up to the font rendering on systems/libraries.
  - happytoexplain 5y ago
    True, but Unicode explicitly defines "SU" as a deprecated combination, regardless of flags. Seems like they omit everything from the list of "no longer used" country codes, with some exceptions. I would think they would have no reason not to allow historical regions.
- bloak 5y ago
  As I understand it, there is no two-letter ISO code for the USSR because when they update the standard they remove countries that no longer exist. In at least one case they have reused a code point: CS has been both "Czechoslovakia" and "Serbia and Montenegro", neither of which currently exist. As a result, two-letter ISO codes are useless for many potential applications, such as, for example, recording which country a book was published in, unless you supplement them with a reference to a particular version of the standard. Is there a way of getting the Czechoslovakian flag as an emoji? And did Serbia and Montenegro get round to making a flag?
jerf 5y ago
So, in terms of acing interviews, increasingly one of the best answers to the question "Write some code that reverses a string" is that in a world of unicode, "reversing a string" is no longer possible or meaningful. You'll probably be told "oh, assume US ASCII" or something, but in the meantime, if you can back that up when they dig into it, you'll look really smart.
- paxys 5y ago
  UTF-8 reverse string has been a thing for a long time in most/all programming languages. It may not work perfectly in 100% of the cases, but that doesn't mean reversing a string is no longer possible.
  - jerf 5y ago
    "It may not work perfectly in 100% of the cases, but that doesn't mean reversing a string is no longer possible." It depends on your point of view. From a strict point of view, it does exactly mean it is no longer possible. By contrast, we all 100% knew what reversing an ASCII string meant, with no ambiguity. It also depends on the version of Unicode you are using, and oh by the way, unicode strings do not come annotated with the version they are in. Since it's supposed to be backwards compatible hopefully the latest works, but I'd be unsurprised if someone can name something whose correct reversal depends on the version of Unicode. And, if not now, then in some later not-yet-existing pair of Unicode standards.
    - pwdisswordfish9 5y ago
      > By contrast, we all 100% knew what reversing an ASCII string meant, with no ambiguity. Not if the ASCII string employed the backspace control character to accomplish what is today done with Unicode combining characters. Or, in fact, if it employed any other kind of control sequence.
      - thaumasiotes 5y ago
        I always thought it was interesting that ASCII is transparently just a bunch of control codes for a typewriter (where "strike an 'a'" is a mechanical instruction no different from "reset the carriage position"), but when we wanted to represent symbolic data we copied it and included all of the nonsensical mechanical instructions.
michaelsbradley 5y ago
See chapter 7 in Hacking the Planet (with Notcurses) for a short treatment of encodings, extended grapheme clusters, etc. https://nick-black.com/htp-notcurses.pdf#page53 https://nick-black.com/htp-notcurses.pdf#page53
xmprt 5y ago
This is a cool article about Unicode encoding however I still feel like it should be possible to reverse strings with Flag emojis. I don't see why computers can't handle multi rune symbols in the same way that they handle multi byte runes. We could combine all the runes that should be a single symbol and make sure that we're maintaining the ordering of those runes in the reversed string. Of course that means that naive string reversing doesn't work anymore but naive string reversing wouldn't work in the world of UTF-8 if we just went byte by byte.
- nitely 5y ago
  Of course it's possible, the Unicode standard even has a table[0] you can use to build a DFA (Deterministic Finite Automata) to break up a string into grapheme clusters. You can reverse the DFA to match and yield the graphemes backwards as well, which will give you the reversed unicode string. [0]: http://www.unicode.org/reports/tr29/#Table_Combining_Char_Sequences_and_Grapheme_Clusters http://www.unicode.org/reports/tr29/#Table_Combining_Char_Se...
- happytoexplain 5y ago
  Swift, for example, does what you're saying. I thought that the reason many languages don't do it that way is that part of the definition of an array (or at least expected-by-convention) is constant-time operations. If you treat a string as an array, then having to deal with variable-length units breaks that rule. That's why, when there is an API for dealing with grapheme clusters, it is usually a special case that duplicates an array-like API, instead of literally using an array. I actually don't know how/why Python is apparently using code points, since they are variable length. That seems like a compromise between using code units and using grapheme clusters that gets you the worst of both worlds. Edit: Maybe it uses UTF-32 under the hood when it's doing array operations on code points?
  - nitely 5y ago
    CPython 3 does use UTF-32 under the hood for strings (there is bytes for plain sequence of bytes). As you say, it's the worst of both worlds. High memory usage, and not really useful if you are dealing with unicode characters (grapheme clusters). My impression is most modern languages that bother with unicode (swift, rust, nim) are using utf-8, and doing linear time operations to handle unicode. I think that's the right approach, as I don't recall ever needing random access on a unicode string.
kevin_thibedeau 5y ago
This misses the real problem with flag emoji in that they are composed of codepoints that can be in any order. With other emoji you get a base codepoint with potential combining characters. Using a table of combining character ranges you can skip over them and isolate the logical glyph sequences. You don't need surrounding context to parse them out like flags need.
- uniqueuid 5y ago
  Thanks for that interesting detail! If such re-purposing continues, it might be easier to go straight to utf-32 for some use cases.
  - dhosek 5y ago
    Nope, because the repurposing is independent of how the Unicode is represented. There's absolutely no advantage to having a string in UTF-32 over UTF-8 since you'll still need to examine every character and the added overhead for converting byte strings in UTF-8 to 32-bit code points is by far offset by the huge memory increase necessary to store UTF-32. What's more, it's really not that difficult to start at the end of a valid UTF-8 string and get the characters in reverse order. UTF-8 is well-designed that way in that there's never ambiguity about whether you're looking at the beginning byte of a code point.
    - colejohnson66 5y ago
      > UTF-8 is well-designed that way in that there's never ambiguity about whether you're looking at the beginning byte of a code point. To expand, if the most-significant-bit is a 0, it's an ASCII codepoint. If the top two are '10', it's a continuation byte, and if they're '11', it's the start of a multibyte codepoint (the other most-significant-bits specify how long it is to facilitate easy codepoint counting). So a naive codepoint reversal algorithm would start at the end, and move backwards until it sees either an ASCII codepoint or the start of a multibyte one. Upon reaching it, copy those 1-4 bytes to the start of a new buffer. Continue until you reach the start. [0]: https://en.wikipedia.org/wiki/UTF-8#Encoding https://en.wikipedia.org/wiki/UTF-8#Encoding
uniqueuid 5y ago
Upper and lower codepoints are really way too obscure and can create issues you didn't even know you had. I once had the very unpleasant experience of debugging a case where data saved with R on windows and loaded on macOS ended up with individually double-encoded codepoints. Not fun.
dhosek 5y ago
On the challenge front, there are things like á which might be a single code point or two code points (a+´). Then there are the really challenging things like ᾷ where if the components are individual characters, the order of ͺ and ῀ are not guaranteed to be consistent.
- happytoexplain 5y ago
  Which is why these APIs should always make normalization available: https://unicode.org/reports/tr15/ https://unicode.org/reports/tr15/
- saltminer 5y ago
  Then you have stuff like zalgo text (http://eeemo.net/ http://eeemo.net/) which takes pride in abusing code points
sltkr 5y ago
So what was the deal with the Scottish flag?
- gsnedders 5y ago
  From Wikipedia: > A separate mechanism (emoji tag sequences) is used for regional flags, such as England 󠁧󠁢󠁥󠁮󠁧󠁿, Scotland 󠁧󠁢󠁳󠁣󠁴󠁿, Wales 󠁧󠁢󠁷󠁬󠁳󠁿, Texas 󠁵󠁳󠁴󠁸󠁿 or California 󠁵󠁳󠁣󠁡󠁿. It uses U+1F3F4 WAVING BLACK FLAG and formatting tag characters instead of regional indicator symbols. It is based on ISO 3166-2 regions with hyphen removed and lowercase, e.g. GB-ENG → gbeng, terminating with U+E007F CANCEL TAG. Flag of England is therefore represented by a sequence U+1F3F4, U+E0067, U+E0062, U+E0065, U+E006E, U+E0067, U+E007F.
  - ghostly_s 5y ago
    This was the only part that was surprising to me, and as it turns out my surprise mostly stems from still not really understanding how the United Kingdom works.
    - tialaramex 5y ago
      Don't worry, "How the United Kingdom works" is a political question and so subject to change. For example, Wales was essentially just straight up conquered, and so for long periods Wales did not have any distinct legal identity from England. You'll see that today there's a bunch of laws which are for England and Wales but notably not Scotland, including criminal laws. In living memory Wales got some measure of independent control over its own affairs, via an elected "Assembly" but what powers are "devolved" to this assembly are in effect the gift of the Parliament, in Westminster, which is sovereign. Whether taking away those powers would go well is a good question. On the other hand, Northern Ireland is what's left of English/ British dominion over the entire island of Ireland, most of which today is the Republic of Ireland, a sovereign entity with its own everything. It's only existed for about a century, and is a result of the agreed "partition" when the Irish rebelled because most of the Irish wanted independence but those in the North not so much. Feel free to read about euphemistically named "Troubles". In the modern era, Northern Ireland, like Wales, gets a devolved government in Stormont. Unlike Wales, the Northern Ireland government is a total mess, and e.g. they have abortion (like the rest of the UK, and like the rest of Ireland) only because Stormont was so broken that Westminster imposed abortion legalisation on them since they weren't actually governing. If you think the US Congress is dysfunctional, check out Stormont... Finally Scotland was for a very long time an independent but closely related sovereign nation. It agreed to join this United Kingdom about three hundred years ago in the Acts of Union after about a century with the same Monarch ruling both countries. However, it too got a devolved government, a Parliament, probably the most powerful of the three, in Holyrood, Edingburgh in the 20th century and it has a relatively powerful pro-independence politics, the Scottish National Party is the dominant power in Scottish politics, although how many of its voters actually support independence per se is tricky to judge. Brexit changed all this again, because as part of the EU a bunch of the powers you could reasonably localise, and so were "devolved" to Wales, Scotland and Northern Ireland, had been controlled by EU law. So Westminster could say they were devolved, knowing that the constituent entities couldn't actually do much with this supposed power. Having left the EU, those powers were among the thing Brexiteers seemed to have imagined now lay at Westminster, but of course the devolved countries said no, these are our powers, we get to decide e.g. how agricultural subsidies are distributed to suit our farmers. That's even more fun in Northern Ireland, because they share a border with the Republic, an EU member, and so they're not allowed to have certain rules that would obviously result in a physical border with guards and so on. Their Unionists (the people who are why it isn't just part of the Republic of Ireland because they want to be in the United Kingdom) feel like they were sold out by Westminster politicians, while the Republicans (those who'd rather be part of the Republic) see this as potentially a further argument in favour of that. All of which isn't helping at all to keep the peace between these rivals, that peace being the whole reason we don't want to put up a border...
deleted 5y ago
[deleted]
codezero 5y ago
You also can't URL Encode a string (In JS at least) if you truncate an emoji at the beginning or end of it.
- account42 5y ago
  URL Encoding works on bytes and does not concern itself with the character encoding of those bytes (except assuming that it is an ASCII superset) so this is only a limitation of the JS implementation.
  - codezero 5y ago
    JS isn’t the only language that does this.
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
qqii 5y ago
> Challenge: How would you go about writing a function that reverses a string while leaving symbols encoded as sequences of code points intact? Can you do it from scratch? Is there a package available in your language that can do it for you? How did that package solve the problem? So are there any good libraries that can deal with code points that are merged together into a single pictographic and reverse them "as expected"?
- da12 5y ago
  If you're using Python, check out grapheme: https://github.com/alvinlindstam/grapheme https://github.com/alvinlindstam/grapheme
treesknees 5y ago
But you can, and did, reverse a string. It seems you would need more details, such as a request to reverse the meaning or interpretation of the string, which is what the author is getting at. If someone challenges you to reverse an image, what do you do? Do you invert the colors? Mirror horizontally? Mirror vertically? Just reverse the byte order?
- egypturnash 5y ago
  Galaxy brain image reversal: completely redraw it from scratch, with a viewpoint 180º from the original.
  - ravi-delia 5y ago
    New computer vision challenge
  - McBeige 5y ago
    If the FoV is less than 180deg then any image would be a realistic solution as long as it doesn't depict anything from the original.
    - egypturnash 5y ago
      My intent here is "same subject, but now you're standing on the other side of it", not "same viewer location but turned 180º". For instance, if you started with an image of the Washington Monument with the Lincoln Memorial in the background, the "reverse" would be an image of the Washington Monument with assorted Smithsonian museum buildings behind it. Or whatever you theorize would be on the east of the Washington Monument if no reference is available.
- wahern 5y ago
  There's a specification problem here. I like to say that a "string" isn't a data structure, it's the absence of one. Discussing "strings" is pointless. It follows that comparing programming languages by their "string" handling is likewise pointless. Case in point: a "struct" in languages like C and Rust is literally a specification of how to treat segments of a "string" of contiguous bytes.
Beldin 5y ago
Interestingly, on my phone the so-called flag is not a flag at all, but "US" in outline. So python behaves as expected: the 2 character string, when reversed, becomes "SU". Similar stuff happens with the other "flag" strings. I'm sure emojis in my phone are outdated. I'm not sure how that affects whether I see a flag or letters.
- pilsetnieks 5y ago
  Thankfully, there isn't an assigned ISO 3166-1 2-letter country code for SU currently; people may have interesting reactions seeing what happens when reversing a US flag emoji if there were.
  - easrng 5y ago
    If this was 1990 (and we somehow had the current emoji standard) SU would be the USSR flag.
- kingcharles 5y ago
  Out of interest, what phone and browser? The only platform I've seen that doesn't render the flags is Windows.
  - Beldin 5y ago
    An android phone from 2014, with a year out of date chrome. To update chrome, I'd have to give it permission to access my contacts. That ain't happening. (Phone OS is too old for per-app permissions)
nextstep 5y ago
Compare all of this nonsense to how it’s done in Swift. String APIs in Swift are great: intuitive and do what you expect.
zanzibar735 5y ago
Of course you can reverse a string with a flag emoji. You just need to treat a "string" as a collected of Extended Grapheme Clusters, and then you reverse the order of the EGCs. So if the string is `a<flag unicode bytes>b`, the output should be `b<flag unicode bytes>a`.
coreyp_1 5y ago
If you think the Unicode flag emoji take a lot of bytes, then consider the family emoji! (https://unicode.org/emoji/charts/full-emoji-list.html#family https://unicode.org/emoji/charts/full-emoji-list.html#family) I'm in the process of designing a scripting language and implementing it in C++. I plan to put together a YouTube series about it. (Doesn't everyone want to see Bison and Flex mixed with proper unit tests and C++20 code?) Due to my future intended use case, I needed good support for Unicode. I thought that I could write it myself, and I was wrong. I wasted two weeks (in my spare time, mostly evenings) trying to cobble together things that should work, identifying patterns, figuring out how to update it as Unicode itself is updated, thinking about edge cases, i18n, zalgo text, etc. And then I finally reached the point where I knew enough to know that I was making the wrong choice. I'm now using ICU. (https://icu.unicode.org/ https://icu.unicode.org/) It's huge, it was hard to get it working in my environment, and there are very few examples of it's usage online, but after the initial setup dues are paid, it WORKS. Aside: Yes, I know I'm crazy for implementing a programming language that I intend for serious usage. Yes, I have good reasons for doing it, and yes I have considered alternatives. But it's fun, so I'm doing it anyways. Moral of the story: Dealing with Unicode is hard, and if you think it shouldn't be that hard, then you probably don't know enough about the problem!
- Gigachad 5y ago
  It always feels like the most amount of work goes to the least used emoji. So many revisions and additions to the family emoji and yet it’s one of the ones I don’t recall anyone ever using. I think the trap Unicode got in to is technically they can have infinite emoji so they just don’t ever have a way to say no to new proposals.
  - laumars 5y ago
    They do say no though. Frequently too. The problem with Unicode is simply that it’s trying to solve a very hard problem.
    - tialaramex 5y ago
      Exactly this. Humans have incredibly complicated writing systems, and all Unicode wants to do is encode them all. Keep in mind that the trivial toy system we're more familiar with, ASCII, already has some pretty strange features because even to half-arse one human writing system they needed those features. Case is totally wild, it only applies to like 5% of the symbols in ASCII, but in the process it means they each need two codepoints and you're expected to carry around tech for switching back and forth between cases. And then there are several distinct types of white space, each gets a codepoint, some of them try to mess with your text's "position" which may not make any sense in the context where you wanted to use it. What does it mean to have a "horizontal tab" between two parts of the text I wanted to draw on this mug? I found a document which says it is the same as "eight spaces" which seems wrong because surely if you wanted eight spaces you'd just write eight spaces. And after all that ASCII doesn't have working quotation marks, it doesn't understand how to spell a bunch of common English words like naïve or café, pretty disappointing.
mappu 5y ago
If you like this, you may also like why len(emoji) is still not 1 in Python 3 despite all the unicode breakage: https://storytime.ivysaur.me/posts/grapheme-clusters/ https://storytime.ivysaur.me/posts/grapheme-clusters/ I do feel like these are all 'gotcha' questions - I haven't seen any real-world requirement to reverse a string and then have it be displayed in a useful way.
otagekki 5y ago
If flag emojis are really a combination of 2 special characters, the reversal of the U.S. flag should result in having the Soviet Union flag.
- TonyTrapp 5y ago
  It's up to the installed fonts really. I don't know if the combination of S + U is standardized as a Soviet Union flag emoji, but even if it is, your locally installed fonts may not contain every single flag emoji, so the browser would still fall back to rendering the two letters instead.
- masklinn 5y ago
  > the reversal of the U.S. flag should result in having the Soviet Union flag. Except it has been deleted from the ISO 3166-2 registry, so not having it is perfectly valid (arguably more so than having it).
  - account42 5y ago
    No, that only shows that the ISO 3166-2 registry is a bad basis for Unicode flags since having things lose meaning over time should not be acceptable for a text encoding. Flags have another issue here in that they can change even when the country stays the same - a recent example here being Afghanistan, but also France who recently changed the official shades of the colors in their flag. Ideally you'd want a new Unicode representation for any changed flags in order to not retroactively change the meaning in old documents.
- brewmarche 5y ago
  Just tried reversing a Spanish flag with Python and indeed I got Sweden back
  - kingcharles 5y ago
    No-one expects the Swedish flag!
- 5y ago
ts4z 5y ago
Let me cheat a bit and say Unicode comes in three flavors: UTF-8, UCS-2 aka UTF-16, and UTF-32. UTF-8 is byte-oriented, UTF-16 is double-byte oriented, and UTF-32 nobody uses because you waste half the word almost all of the time. You can't reduce the bytes in UTF-8 or UTF-16, because you'll scramble the encoding. But you could parsing the string, codepoint-at-a-time, handling the specifics of UTF-8, or UTF-16 with its surrogate pairs, and reversing those. This sounds equivalent to reversing UTF-32, and I believe is what the original poster was imagining. Except you can't do that, because Unicode has composing characters. Now, I'm American and too stupid to type anything other than ASCII, but I know about n+~ = ñ. If you have the pre-composed version of ñ, you can reverse the codepoint (it's one codepoint). If you don't have it, and you have n+dead ~, you can't reverse it, or in the word "año" you might put the ~ on the "o". (Even crazier things happen when you get to the ligatures in Arabic; IIRC one of those is about 20 codepoints.) So we can't just reverse codepoints, even ancient versions of Unicode. Other posters have talked about the even more exotic stuff like Emoji + skin tone. It's necessary to be very careful. Now, the old fart in me says that ASCII never had this problem. But the old fart in me knows about CRLF in text protocols, and that's never LFCR; and that if you want to make a ñ in ASCII you must send n ^H ~. I guess you can reverse that, but if you want to do more exotic things it becomes less obvious. (IIRC UCS-2 is the deadname, now we call it UTF-16 to remind us to always handle surrogate pairs correctly, which we don't.) TLDR: Strings are hard.
progbits 5y ago
Semi-related (about length of emoji "characters", not reversing): https://hsivonen.fi/string-length/ https://hsivonen.fi/string-length/ Previously discussed: https://news.ycombinator.com/item?id=20914184 https://news.ycombinator.com/item?id=20914184 https://news.ycombinator.com/item?id=26591373 https://news.ycombinator.com/item?id=26591373 As for this article & Python - as usual it is biasing towards convenience and implicit behavior rather than properly handling all edge cases. Compare with Rust where you can't "reverse" a string - that is not a defined operation. But you can either break it into a sequence of characters or graphemes and then reverse that, with expected results: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=6335bb2c5cacf308ed5190deaba0121f https://play.rust-lang.org/?version=stable&mode=debug&editio... (Sadly the grapheme segmentation is not part of standard library, at least yet)
- account42 5y ago
  > Sadly the grapheme segmentation is not part of standard library, at least yet Seeing as grapheme segmentation is a moving target that only makes sense.
aidenn0 5y ago
> The answer is: it depends. There isn't a canonical way to reverse a string, at least that I'm aware of. Unicode defines grapheme clusters[1] that represent "user-perceived characters" separating a string into those and reversing seems like a pretty good way to go about it. 1: http://www.unicode.org/reports/tr29/ http://www.unicode.org/reports/tr29/
tl 5y ago
This is a nice dive into limitations in Python's unicode handling and at the end, how to work around some problems. But you could use languages with proper unicode support like Swift or Elixir (weirdly HN is fighting flags in comment code which makes examples header to demonstrate).
- anamexis 5y ago
  HN doesn't allow any emoji.
jug 5y ago
I'm not surprised the flag had two components, but I _was_ surprised the US flag was made by literally U and S, haha! I definitely thought it'd be something like [I am a Flag] and [The flag ID between 0 and 65535]. And reversing it would be [Flag ID] + [I am a Flag] which would not be a defined "component" and instead rendered as the individual two nonsense characters.
- andylynch 5y ago
  You might also have noticed this is partly a very well thought out hack to make Unicode less sensitive to disagreements and changes in consensus on which flags are encoded, or even the names of the countries concerned!
bandyaboot 5y ago
Would be interesting to see the list of flag emojis that, when reversed, become a different flag emoji.
- jfk13 5y ago
  There are plenty of country codes that when reversed become a different, valid country code: e.g. Israel (IL) when reversed is Lithuania (LI); Australia (AU) becomes Ukraine (UA). Whether "reversing flag emojis" causes such transformations will depend on what is meant by "reversing", which is kind of the whole point here: there are a number of possible interpretations of "reverse".
alfredxing 5y ago
Related — I did a deep dive a couple years ago on emoji codepoints and how they're encoded in the Apple emoji font file, with the end goal of extracting the embedded images — https://github.com/alfredxing/emoji https://github.com/alfredxing/emoji
zwerdlds 5y ago
In normal conditions you can check for a ZWJ, but with regional coding chars, you would have to consider the regional chars block as a single char in the reversal. Given that is isn't necessarily locale dependant but presentation layer dependant, there might not be anough info to decide how to act.
jiveturkey 5y ago
Interesting article. Written for beginners, conversationally. Has excessive amounts of whitespace, for "readability" I guess. But at the same time, it dives quite deep, which I don't think this "style" of presentation matches up with the amount of time a more novice reader is going to devote to a single long form article. As to the content, for all the deep dive, a simple link to https://unicode.org/reports/tr51/#Flags https://unicode.org/reports/tr51/#Flags and what an emoji is, would have saved so much exposition. I also wish he'd touched on normalization. With the amount of time he's demanding from readers he could have mentioned this important subject. Because then he could discuss why (starting from his emoji example) a-grave (à) might or might not be reversible, depending how the character is composed. Also wish he'd pointed to some libraries that can do such reversals.
deleted 5y ago
[deleted]
techwiz137 5y ago
It's pretty funny that reversing the American flag yields Soviet Union(SU).
Crazyontap 5y ago
This section on the linked Wikipedia article(1) is quite amazing on how the family emoji is rendered using a zero-width joiner (1) https://en.wikipedia.org/wiki/Emoji#Joining https://en.wikipedia.org/wiki/Emoji#Joining edit: forgot HN doesn't render emojis. Better read it directly on Wikipedia i guess.
hougaard 5y ago
In other news, water is wet :)
mlindner 5y ago
The person tries to define character when there isn't actually any definition of what that even means. Character is a term limited to languages that actually use them and not all text is made up of characters.
emodendroket 5y ago
What I'd like to know is, given the explosion of the character set for emoji, does the rationale for Han unification still make sense? The case for not allowing national variants seems less and less compelling with every emoji they add. This is a bit of a hobby horse, but imagine if every time you read an article in English on your phone some of the letters were replaced with "equivalent" Greek or Cyrillic one and you can get an idea of the annoyance. Yeah, you can still read it with a bit of thought, but who wants to read that way?
- digisign 5y ago
  > were replaced with "equivalent" Greek or Cyrillic one The subset of equivalent letters, or different ones? If they looked the same, it wouldn't bother me if the letters in the center were a single codepoint between European languages: https://upload.wikimedia.org/wikipedia/commons/8/84/Venn_diagram_showing_Greek%2C_Latin_and_Cyrillic_letters.svg https://upload.wikimedia.org/wikipedia/commons/8/84/Venn_dia...
  - account42 5y ago
    I am disappointed that that diagram omits ꙮ [0] [0] https://en.wikipedia.org/wiki/Multiocular_O https://en.wikipedia.org/wiki/Multiocular_O
  - emodendroket 5y ago
    The problem is they don't look the same. So imagine, for instance, Я instead of "R" or И instead of "N" (I don't think the sounds are actually equivalent but let's run with it for the sake of example). Not insurmountable. One could still read a text with these substitutions. But it'd be distracting, and extra detrimental for people who don't speak English as their first language.
    - digisign 5y ago
      The ones in the center are in all three sets, they do look the same. The outer areas are out of bounds.
- 5y ago
raffy 5y ago
Kinda related: I am developing a library for ENS (Ethereum Name Service) name normalization: https://github.com/adraffy/ens-normalize.js https://github.com/adraffy/ens-normalize.js I'm trying to find the best combination of UTS-46, UTS-51, UTS-39, and prior work on IDN resolution w/r/t confusables: https://adraffy.github.io/ens-normalize.js/test/report-confusables.html https://adraffy.github.io/ens-normalize.js/test/report-confu... Personally, I found the Unicode spec very messy. Critical information is all over the place. You can see the direct effect of this when you compare various packages across different languages and discover that every library disagrees in multiple places. Even JS String.normalize() isn't consistent in the latest version of most browsers: https://adraffy.github.io/ens-normalize.js/test/report-nf.html https://adraffy.github.io/ens-normalize.js/test/report-nf.ht... (fails in Chrome, Safari) The major difference between ENS and DNS is emoji are front and center. ENS resolves by computing a hash of a name in a canonicalized form. Since resolution must happen decentralized, simply punting to punycode and relying custom logic for Unicode-handling isn't possible. On-chain records are 1:1, so there's no fuzzy matching either. Additionally, ENS is actively registering names, so any improvement to the system must preserve as many names as possible. At the moment, I'm attempting to improve upon the confusables in the Common/Greek/Latin/Cyrillic scripts, and will combine these new grouping with the mixed-script limitations similar to IDN handling in Chromium. Interactive Demo: https://adraffy.github.io/ens-normalize.js/test/resolver.html https://adraffy.github.io/ens-normalize.js/test/resolver.htm... Also this emoji report is pretty cool: https://adraffy.github.io/ens-normalize.js/test/report-emoji.html https://adraffy.github.io/ens-normalize.js/test/report-emoji...
nottorp 5y ago
So basically unicode along with c++ are great job security if you do bother to learn them. There's another word that comes to mind when thinking about those two: metastasis.
utopcell 5y ago
There are unicode characters that reverse parsing order themselves. This has been the basis of a code injection attack, analyzed in [1]. [1] ``Trojan Source: Invisible Vulnerabilities'': https://trojansource.codes/trojan-source.pdf https://trojansource.codes/trojan-source.pdf
randpx 5y ago
Try reversing the Canadian flag (CA) and you get the Ascension Island Flag (AC). Great article, but completely misses the point.
faebi 5y ago
Why reverse them if one barely can implement, display and edit them correctly. I never could make them work perfectly in VIM. Also I had to open a bug in Firefox recently: Flag emojis and others are displayed in double the size on Windows 10 using Firefox Nightly https://bugzilla.mozilla.org/show_bug.cgi?id=1746795 https://bugzilla.mozilla.org/show_bug.cgi?id=1746795
- easrng 5y ago
  Windows doesn't even have flag emojis, they just show up as the country code. Edit: Actually Firefox ships a copy of twemoji for fallback purposes, so flags will still render.
Waterluvian 5y ago
I wish languages did a far better job clearly distinguishing between their operations: 1. You are acting in byte space and it’s pretty unambiguous what should happen. We are not acknowledging the semantics of language and alphabets. 2. You’re acting in language space and these operations will behave the way you probably think they should (depending on your cultural expectations, probably)
cmyr 5y ago
Something I haven't seen mentioned yet is one of the most annoying things about regional indicator symbols, which is that interpreting them correctly requires arbitrary backtracking, and handling this correctly is very annoying for things like text fields. Basically: A single, unpaired RIS counts as a single grapheme. Similarly, a pair of RIS count as a single grapheme. Now imagine if your cursor position is after an RTS, and you arrow backwards (assuming LTR text, imagine your cursor is to the right of an RIS, and you press the left arrow.) Your textbox should now move the cursor to the left by one grapheme. How do you figure out where this is, in code units? You basically have to scan backwards until you find the first non-RIS codepoint, and then you have to match them up into pairs to figure out if your left-arrow movement should correspond to a movement of one or two codepoints. This is a longstanding source of bugs, and if you're bored you can play around with pasting a huge sequence of flags into a textfield and then trying to navigate around it with the arrow keys/mouse. There are some broken implementations out there. edit: while I'm thinking about this I will point out that an alternative design, which would have solved this problem (and which was first pointed out to me by @raphlinus) would have been to have two separate sets of RI symbols, one for 'first position' and one for 'second position'; then you could always determine the appropriate cursor position without needing context. Isn't hindsight a wonderful thing?
- account42 5y ago
  > Isn't hindsight a wonderful thing? Gladly, the creators of UTF-18 did have that foresight so at least we don't have this problem at the code unit -> code point level.
a_c 5y ago
Understanding unicode would make the question more obvious https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
sundarurfriend 5y ago
Julia docs do a (surprisingly) good job of being clear and explicit about this: the docstring for `reverse(AbstractString)` says: > Reverses a string. Technically, this function reverses the codepoints in a string and its main utility is for reversed-order string processing [...]. See also [...] `graphemes` from module Unicode to operate on user-visible "characters" (graphemes) rather than codepoints. Properly reversing a string of flags (or any other grapheme clusters) is just a `using Unicode: grapheme` away.
ineedasername 5y ago
It's an emoji... Are there any emojis with only one character? My assumption going in would be that any emoji is > 1 character. Admittedly, despite lots of string processing, I never have to deal with emojis so I guess I'm not sure. An interesting exercise would be emoji detection during string reversal to preserve the original emoji. I though something like that would be the crux of the article. Am I wrong about single character emojis?
- easrng 5y ago
  It depends what you mean by character, there are lots of single codepoint emojis though.
DamnGirl 5y ago
nitely 5y ago
You can, but you need to break the string into graphemes first.
qwerty456127 5y ago
If the US flag is 2 special symbols saying US, why doesn't reversing it just produce the flag of the Soviet Union?
- NullPrefix 5y ago
  Same reason why there is no Nazi Germany flag - they are not included in Unicode.
  - qwerty456127 5y ago
    Quite unfortunate - the US/SU case would make a nice Unicode+political pun.
demetrius 5y ago
It's sad that Unicode doesn't include flags for dissolved countries. If it did, reversing an US flag would make a Soviet Union flag (code SU). This would make the text much more fun
- kragen 5y ago
  The whole reason for handling the flag emojis that way was so that the Unicode Consortium wouldn't have to decide which countries should or should not be recognized. It is totally valid for you to configure your computer to display SU as a Soviet flag.
midjji 5y ago
And this is why char should have been byte from the start.
chrismorgan 5y ago
UTF-8 does not represent Unicode code points, but rather Unicode scalar values. The difference between the two is surrogates, the way that UTF-16 ruined Unicode: code points are 0₁₆ to 10FFFF₁₆, scalar values are 0₁₆ to D7FF₁₆ and E000₁₆ to 10FFFF₁₆. Yes, the author quoted Wikipedia, but Wikipedia is wrong on this point; surprisingly comprehensively wrong: the UTF-8 page completely ignores the distinction, and even the page on code points doesn’t mention scalar values! This error propagates to other places, too: for example, “and there are a total of 1,112,064 possible code points”: no, that’s how many scalar values there are; code points also include the 2,048 surrogates, so there are 1,114,112 code points.

189 points by da12 5y ago

I feel like I'm obligated to share this almost 20 year old Spolsky post that gave me my understanding of characters. https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...

xmprt 5y ago
In that same vein, here's my introduction to Unicode about 10 years ago from Tom Scott. https://www.youtube.com/watch?v=MijmeoH9LT4 https://www.youtube.com/watch?v=MijmeoH9LT4
- zerox7felf 5y ago
  poor man gave me and many others something like half of our introduction to computer science, but has gotten far more fame as the "emoji guy" for his repeated bouts with this particular part of unicode :)
- ciupicri 5y ago
  That's more about the UTF-8 encoding than Unicode itself.

codingkev 5y ago

Yes, this allows for easy building of flag emojis as long as you know the ISO 3166 two-letter country code. Example: https://github.com/kennell/flagz/blob/master/flagz.py https://github.com/kennell/flagz/blob/master/flagz.py

exdsq 5y ago

Am I missing something or is this Day 1 of a programming course in C?

Mesopropithecus 5y ago

Unfortunately the HN text input won't let me do this, but a funny starter for the article would have been this: '(Spanish flag)'[::-1] basically ''.join([chr(127466), chr(127480)]) vs. ''.join([chr(127466), chr(127480)])[::-1] I'll add this to my collection of party tricks and show myself out. Cool article!

deleted 5y ago

[deleted]

architectdrone 5y ago

humorously, on my local machine, I only see the string "us", and was rather confused when he was asserting that it was a single character :D

kingcharles 5y ago
You're on Windows? Windows doesn't render flag emojis as flags.

hffftz 5y ago

jupp0r 5y ago

With all the criticism I normally have for Rust, I must say that its type safe handling of UTF-8 and its unambiguous distinction between byte strings and UTF-8 strings are extremely helpful in handling situations mentioned in the article correctly (and also efficiently). Yes it's a pain, but the way the standard library designed its types force you to handle conversions correctly, for example when byte arrays are converted to UTF-8 strings and may contain invalid UTF-8 sequences.

heystefan 5y ago

Oooh I know this one, I've read it here last year: https://tonsky.me/blog/emoji/ https://tonsky.me/blog/emoji/

zarzavat 5y ago

This reminds me of an interesting bug I saw where I was seeing a strange flag in some Arabic text. However when I copied the string and pasted it into a text editor, the flag of Saudi Arabia appeared instead (which made much more sense). After some vexillologic research on Wikipedia I identified the original flag as American Samoa and it suddenly all made sense. Turns out some broken RTL support was flipping the SA into AS at presentation.

zarzavat 5y ago
After writing this comment I did some more research. Apparently this is actually a bug in Chrome itself (!). https://bugs.chromium.org/p/chromium/issues/detail?id=1272436 https://bugs.chromium.org/p/chromium/issues/detail?id=127243...

kart23 5y ago

Google also interprets emojis funny. Google the Estonian and South Sudan flag (f"{chr(127466)*2+chr(127480)*2}") and you get results for Spain.

smegsicle 5y ago

did they think all those skintone emojis are individual codepoints?

advisedwang 5y ago
They might have thought that `reverse()` had some kind of unicode-aware handling. I believe `upper()`/`lower()` do.
daveslash 5y ago
When I first realized that the skin tone emojis were a code-point + a color code-point modifier, I tried to see what other colors there were and if I could apply those to other emojis. The immature child in me looked to see if there was a red color code point and if so, could I use it to make a "blood poop" emoji. Turns out.... no.

yoyohello13 5y ago

Maybe I'm missing some prerequisite knowledge here, but why would I assume `flag="us"` is an emoji? Looking at that first block of code, there is no reason for me to think "us" is a single character. Edit: Turns out my browser wasn't rendering the flags.

Benlights 5y ago
I had the same issue when I read the article, I kept on getting stuck and asking myself what I was missing.
greenyoda 5y ago
In my browser (Firefox on Windows), the thing between the quotes in the first block of code looks like a picture of the US flag cropped to a circle, not like the characters "us".
- yoyohello13 5y ago
  Ah I see, I just opened it in firefox. It looks like some JS library is not getting loaded in Edge. The author was talking about "us", "so", etc. looking like one character and I thought I was going crazy, lol.
  - bialpio 5y ago
    Reminds me of an image that renders differently on Macs (https://www.bleepingcomputer.com/news/technology/this-image-looks-very-different-on-apple-devices-see-for-yourself/ https://www.bleepingcomputer.com/news/technology/this-image-...), I bet it'd make for a fun conversation that could make the participants question their sanity. :-)
  - jfk13 5y ago
    I don't think that's about a JS library. Firefox bundles an emoji font that supports some things -- such as the flags -- that aren't supported by Segoe UI Emoji on Windows, so it has additional coverage for such character sequences.

happytoexplain 5y ago

I guessed that it would become the USSR flag (US -> SU), but apparently Unicode doesn't define that one! I wonder why. That would have been humorous.

ts4z 5y ago
IIRC Unicode doesn't define country codes. It was a workaround for a political issue of which countries recognize which other countries. It would have been difficult to get the CN delegation to sign off on a list that contained TW, although there are probably others.
- andylynch 5y ago
  There are many more than I realised - Wikipedia has a decent list https://en.m.wikipedia.org/wiki/List_of_states_with_limited_recognition https://en.m.wikipedia.org/wiki/List_of_states_with_limited_...
chungy 5y ago
Unicode doesn't define any flags, really. That's up to the font rendering on systems/libraries.
- happytoexplain 5y ago
  True, but Unicode explicitly defines "SU" as a deprecated combination, regardless of flags. Seems like they omit everything from the list of "no longer used" country codes, with some exceptions. I would think they would have no reason not to allow historical regions.
bloak 5y ago
As I understand it, there is no two-letter ISO code for the USSR because when they update the standard they remove countries that no longer exist. In at least one case they have reused a code point: CS has been both "Czechoslovakia" and "Serbia and Montenegro", neither of which currently exist. As a result, two-letter ISO codes are useless for many potential applications, such as, for example, recording which country a book was published in, unless you supplement them with a reference to a particular version of the standard. Is there a way of getting the Czechoslovakian flag as an emoji? And did Serbia and Montenegro get round to making a flag?

jerf 5y ago

So, in terms of acing interviews, increasingly one of the best answers to the question "Write some code that reverses a string" is that in a world of unicode, "reversing a string" is no longer possible or meaningful. You'll probably be told "oh, assume US ASCII" or something, but in the meantime, if you can back that up when they dig into it, you'll look really smart.

paxys 5y ago
UTF-8 reverse string has been a thing for a long time in most/all programming languages. It may not work perfectly in 100% of the cases, but that doesn't mean reversing a string is no longer possible.
- jerf 5y ago
  "It may not work perfectly in 100% of the cases, but that doesn't mean reversing a string is no longer possible." It depends on your point of view. From a strict point of view, it does exactly mean it is no longer possible. By contrast, we all 100% knew what reversing an ASCII string meant, with no ambiguity. It also depends on the version of Unicode you are using, and oh by the way, unicode strings do not come annotated with the version they are in. Since it's supposed to be backwards compatible hopefully the latest works, but I'd be unsurprised if someone can name something whose correct reversal depends on the version of Unicode. And, if not now, then in some later not-yet-existing pair of Unicode standards.
  - pwdisswordfish9 5y ago
    > By contrast, we all 100% knew what reversing an ASCII string meant, with no ambiguity. Not if the ASCII string employed the backspace control character to accomplish what is today done with Unicode combining characters. Or, in fact, if it employed any other kind of control sequence.
    - thaumasiotes 5y ago
      I always thought it was interesting that ASCII is transparently just a bunch of control codes for a typewriter (where "strike an 'a'" is a mechanical instruction no different from "reset the carriage position"), but when we wanted to represent symbolic data we copied it and included all of the nonsensical mechanical instructions.

michaelsbradley 5y ago

See chapter 7 in Hacking the Planet (with Notcurses) for a short treatment of encodings, extended grapheme clusters, etc. https://nick-black.com/htp-notcurses.pdf#page53 https://nick-black.com/htp-notcurses.pdf#page53

xmprt 5y ago

This is a cool article about Unicode encoding however I still feel like it should be possible to reverse strings with Flag emojis. I don't see why computers can't handle multi rune symbols in the same way that they handle multi byte runes. We could combine all the runes that should be a single symbol and make sure that we're maintaining the ordering of those runes in the reversed string. Of course that means that naive string reversing doesn't work anymore but naive string reversing wouldn't work in the world of UTF-8 if we just went byte by byte.

nitely 5y ago
Of course it's possible, the Unicode standard even has a table[0] you can use to build a DFA (Deterministic Finite Automata) to break up a string into grapheme clusters. You can reverse the DFA to match and yield the graphemes backwards as well, which will give you the reversed unicode string. [0]: http://www.unicode.org/reports/tr29/#Table_Combining_Char_Sequences_and_Grapheme_Clusters http://www.unicode.org/reports/tr29/#Table_Combining_Char_Se...
happytoexplain 5y ago
Swift, for example, does what you're saying. I thought that the reason many languages don't do it that way is that part of the definition of an array (or at least expected-by-convention) is constant-time operations. If you treat a string as an array, then having to deal with variable-length units breaks that rule. That's why, when there is an API for dealing with grapheme clusters, it is usually a special case that duplicates an array-like API, instead of literally using an array. I actually don't know how/why Python is apparently using code points, since they are variable length. That seems like a compromise between using code units and using grapheme clusters that gets you the worst of both worlds. Edit: Maybe it uses UTF-32 under the hood when it's doing array operations on code points?
- nitely 5y ago
  CPython 3 does use UTF-32 under the hood for strings (there is bytes for plain sequence of bytes). As you say, it's the worst of both worlds. High memory usage, and not really useful if you are dealing with unicode characters (grapheme clusters). My impression is most modern languages that bother with unicode (swift, rust, nim) are using utf-8, and doing linear time operations to handle unicode. I think that's the right approach, as I don't recall ever needing random access on a unicode string.

kevin_thibedeau 5y ago

This misses the real problem with flag emoji in that they are composed of codepoints that can be in any order. With other emoji you get a base codepoint with potential combining characters. Using a table of combining character ranges you can skip over them and isolate the logical glyph sequences. You don't need surrounding context to parse them out like flags need.

uniqueuid 5y ago
Thanks for that interesting detail! If such re-purposing continues, it might be easier to go straight to utf-32 for some use cases.
- dhosek 5y ago
  Nope, because the repurposing is independent of how the Unicode is represented. There's absolutely no advantage to having a string in UTF-32 over UTF-8 since you'll still need to examine every character and the added overhead for converting byte strings in UTF-8 to 32-bit code points is by far offset by the huge memory increase necessary to store UTF-32. What's more, it's really not that difficult to start at the end of a valid UTF-8 string and get the characters in reverse order. UTF-8 is well-designed that way in that there's never ambiguity about whether you're looking at the beginning byte of a code point.
  - colejohnson66 5y ago
    > UTF-8 is well-designed that way in that there's never ambiguity about whether you're looking at the beginning byte of a code point. To expand, if the most-significant-bit is a 0, it's an ASCII codepoint. If the top two are '10', it's a continuation byte, and if they're '11', it's the start of a multibyte codepoint (the other most-significant-bits specify how long it is to facilitate easy codepoint counting). So a naive codepoint reversal algorithm would start at the end, and move backwards until it sees either an ASCII codepoint or the start of a multibyte one. Upon reaching it, copy those 1-4 bytes to the start of a new buffer. Continue until you reach the start. [0]: https://en.wikipedia.org/wiki/UTF-8#Encoding https://en.wikipedia.org/wiki/UTF-8#Encoding

uniqueuid 5y ago

Upper and lower codepoints are really way too obscure and can create issues you didn't even know you had. I once had the very unpleasant experience of debugging a case where data saved with R on windows and loaded on macOS ended up with individually double-encoded codepoints. Not fun.

dhosek 5y ago

On the challenge front, there are things like á which might be a single code point or two code points (a+´). Then there are the really challenging things like ᾷ where if the components are individual characters, the order of ͺ and ῀ are not guaranteed to be consistent.

happytoexplain 5y ago
Which is why these APIs should always make normalization available: https://unicode.org/reports/tr15/ https://unicode.org/reports/tr15/
saltminer 5y ago
Then you have stuff like zalgo text (http://eeemo.net/ http://eeemo.net/) which takes pride in abusing code points

sltkr 5y ago

So what was the deal with the Scottish flag?

gsnedders 5y ago
From Wikipedia: > A separate mechanism (emoji tag sequences) is used for regional flags, such as England 󠁧󠁢󠁥󠁮󠁧󠁿, Scotland 󠁧󠁢󠁳󠁣󠁴󠁿, Wales 󠁧󠁢󠁷󠁬󠁳󠁿, Texas 󠁵󠁳󠁴󠁸󠁿 or California 󠁵󠁳󠁣󠁡󠁿. It uses U+1F3F4 WAVING BLACK FLAG and formatting tag characters instead of regional indicator symbols. It is based on ISO 3166-2 regions with hyphen removed and lowercase, e.g. GB-ENG → gbeng, terminating with U+E007F CANCEL TAG. Flag of England is therefore represented by a sequence U+1F3F4, U+E0067, U+E0062, U+E0065, U+E006E, U+E0067, U+E007F.
- ghostly_s 5y ago
  This was the only part that was surprising to me, and as it turns out my surprise mostly stems from still not really understanding how the United Kingdom works.
  - tialaramex 5y ago
    Don't worry, "How the United Kingdom works" is a political question and so subject to change. For example, Wales was essentially just straight up conquered, and so for long periods Wales did not have any distinct legal identity from England. You'll see that today there's a bunch of laws which are for England and Wales but notably not Scotland, including criminal laws. In living memory Wales got some measure of independent control over its own affairs, via an elected "Assembly" but what powers are "devolved" to this assembly are in effect the gift of the Parliament, in Westminster, which is sovereign. Whether taking away those powers would go well is a good question. On the other hand, Northern Ireland is what's left of English/ British dominion over the entire island of Ireland, most of which today is the Republic of Ireland, a sovereign entity with its own everything. It's only existed for about a century, and is a result of the agreed "partition" when the Irish rebelled because most of the Irish wanted independence but those in the North not so much. Feel free to read about euphemistically named "Troubles". In the modern era, Northern Ireland, like Wales, gets a devolved government in Stormont. Unlike Wales, the Northern Ireland government is a total mess, and e.g. they have abortion (like the rest of the UK, and like the rest of Ireland) only because Stormont was so broken that Westminster imposed abortion legalisation on them since they weren't actually governing. If you think the US Congress is dysfunctional, check out Stormont... Finally Scotland was for a very long time an independent but closely related sovereign nation. It agreed to join this United Kingdom about three hundred years ago in the Acts of Union after about a century with the same Monarch ruling both countries. However, it too got a devolved government, a Parliament, probably the most powerful of the three, in Holyrood, Edingburgh in the 20th century and it has a relatively powerful pro-independence politics, the Scottish National Party is the dominant power in Scottish politics, although how many of its voters actually support independence per se is tricky to judge. Brexit changed all this again, because as part of the EU a bunch of the powers you could reasonably localise, and so were "devolved" to Wales, Scotland and Northern Ireland, had been controlled by EU law. So Westminster could say they were devolved, knowing that the constituent entities couldn't actually do much with this supposed power. Having left the EU, those powers were among the thing Brexiteers seemed to have imagined now lay at Westminster, but of course the devolved countries said no, these are our powers, we get to decide e.g. how agricultural subsidies are distributed to suit our farmers. That's even more fun in Northern Ireland, because they share a border with the Republic, an EU member, and so they're not allowed to have certain rules that would obviously result in a physical border with guards and so on. Their Unionists (the people who are why it isn't just part of the Republic of Ireland because they want to be in the United Kingdom) feel like they were sold out by Westminster politicians, while the Republicans (those who'd rather be part of the Republic) see this as potentially a further argument in favour of that. All of which isn't helping at all to keep the peace between these rivals, that peace being the whole reason we don't want to put up a border...

deleted 5y ago

[deleted]

codezero 5y ago

You also can't URL Encode a string (In JS at least) if you truncate an emoji at the beginning or end of it.

account42 5y ago
URL Encoding works on bytes and does not concern itself with the character encoding of those bytes (except assuming that it is an ASCII superset) so this is only a limitation of the JS implementation.
- codezero 5y ago
  JS isn’t the only language that does this.

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

qqii 5y ago

> Challenge: How would you go about writing a function that reverses a string while leaving symbols encoded as sequences of code points intact? Can you do it from scratch? Is there a package available in your language that can do it for you? How did that package solve the problem? So are there any good libraries that can deal with code points that are merged together into a single pictographic and reverse them "as expected"?

da12 5y ago
If you're using Python, check out grapheme: https://github.com/alvinlindstam/grapheme https://github.com/alvinlindstam/grapheme

treesknees 5y ago

But you can, and did, reverse a string. It seems you would need more details, such as a request to reverse the meaning or interpretation of the string, which is what the author is getting at. If someone challenges you to reverse an image, what do you do? Do you invert the colors? Mirror horizontally? Mirror vertically? Just reverse the byte order?

egypturnash 5y ago
Galaxy brain image reversal: completely redraw it from scratch, with a viewpoint 180º from the original.
- ravi-delia 5y ago
  New computer vision challenge
- McBeige 5y ago
  If the FoV is less than 180deg then any image would be a realistic solution as long as it doesn't depict anything from the original.
  - egypturnash 5y ago
    My intent here is "same subject, but now you're standing on the other side of it", not "same viewer location but turned 180º". For instance, if you started with an image of the Washington Monument with the Lincoln Memorial in the background, the "reverse" would be an image of the Washington Monument with assorted Smithsonian museum buildings behind it. Or whatever you theorize would be on the east of the Washington Monument if no reference is available.
wahern 5y ago
There's a specification problem here. I like to say that a "string" isn't a data structure, it's the absence of one. Discussing "strings" is pointless. It follows that comparing programming languages by their "string" handling is likewise pointless. Case in point: a "struct" in languages like C and Rust is literally a specification of how to treat segments of a "string" of contiguous bytes.

Beldin 5y ago

Interestingly, on my phone the so-called flag is not a flag at all, but "US" in outline. So python behaves as expected: the 2 character string, when reversed, becomes "SU". Similar stuff happens with the other "flag" strings. I'm sure emojis in my phone are outdated. I'm not sure how that affects whether I see a flag or letters.

pilsetnieks 5y ago
Thankfully, there isn't an assigned ISO 3166-1 2-letter country code for SU currently; people may have interesting reactions seeing what happens when reversing a US flag emoji if there were.
- easrng 5y ago
  If this was 1990 (and we somehow had the current emoji standard) SU would be the USSR flag.
kingcharles 5y ago
Out of interest, what phone and browser? The only platform I've seen that doesn't render the flags is Windows.
- Beldin 5y ago
  An android phone from 2014, with a year out of date chrome. To update chrome, I'd have to give it permission to access my contacts. That ain't happening. (Phone OS is too old for per-app permissions)

nextstep 5y ago

Compare all of this nonsense to how it’s done in Swift. String APIs in Swift are great: intuitive and do what you expect.

zanzibar735 5y ago

Of course you can reverse a string with a flag emoji. You just need to treat a "string" as a collected of Extended Grapheme Clusters, and then you reverse the order of the EGCs. So if the string is `a<flag unicode bytes>b`, the output should be `b<flag unicode bytes>a`.

coreyp_1 5y ago

If you think the Unicode flag emoji take a lot of bytes, then consider the family emoji! (https://unicode.org/emoji/charts/full-emoji-list.html#family https://unicode.org/emoji/charts/full-emoji-list.html#family) I'm in the process of designing a scripting language and implementing it in C++. I plan to put together a YouTube series about it. (Doesn't everyone want to see Bison and Flex mixed with proper unit tests and C++20 code?) Due to my future intended use case, I needed good support for Unicode. I thought that I could write it myself, and I was wrong. I wasted two weeks (in my spare time, mostly evenings) trying to cobble together things that should work, identifying patterns, figuring out how to update it as Unicode itself is updated, thinking about edge cases, i18n, zalgo text, etc. And then I finally reached the point where I knew enough to know that I was making the wrong choice. I'm now using ICU. (https://icu.unicode.org/ https://icu.unicode.org/) It's huge, it was hard to get it working in my environment, and there are very few examples of it's usage online, but after the initial setup dues are paid, it WORKS. Aside: Yes, I know I'm crazy for implementing a programming language that I intend for serious usage. Yes, I have good reasons for doing it, and yes I have considered alternatives. But it's fun, so I'm doing it anyways. Moral of the story: Dealing with Unicode is hard, and if you think it shouldn't be that hard, then you probably don't know enough about the problem!

Gigachad 5y ago
It always feels like the most amount of work goes to the least used emoji. So many revisions and additions to the family emoji and yet it’s one of the ones I don’t recall anyone ever using. I think the trap Unicode got in to is technically they can have infinite emoji so they just don’t ever have a way to say no to new proposals.
- laumars 5y ago
  They do say no though. Frequently too. The problem with Unicode is simply that it’s trying to solve a very hard problem.
  - tialaramex 5y ago
    Exactly this. Humans have incredibly complicated writing systems, and all Unicode wants to do is encode them all. Keep in mind that the trivial toy system we're more familiar with, ASCII, already has some pretty strange features because even to half-arse one human writing system they needed those features. Case is totally wild, it only applies to like 5% of the symbols in ASCII, but in the process it means they each need two codepoints and you're expected to carry around tech for switching back and forth between cases. And then there are several distinct types of white space, each gets a codepoint, some of them try to mess with your text's "position" which may not make any sense in the context where you wanted to use it. What does it mean to have a "horizontal tab" between two parts of the text I wanted to draw on this mug? I found a document which says it is the same as "eight spaces" which seems wrong because surely if you wanted eight spaces you'd just write eight spaces. And after all that ASCII doesn't have working quotation marks, it doesn't understand how to spell a bunch of common English words like naïve or café, pretty disappointing.

mappu 5y ago

If you like this, you may also like why len(emoji) is still not 1 in Python 3 despite all the unicode breakage: https://storytime.ivysaur.me/posts/grapheme-clusters/ https://storytime.ivysaur.me/posts/grapheme-clusters/ I do feel like these are all 'gotcha' questions - I haven't seen any real-world requirement to reverse a string and then have it be displayed in a useful way.

otagekki 5y ago

If flag emojis are really a combination of 2 special characters, the reversal of the U.S. flag should result in having the Soviet Union flag.

TonyTrapp 5y ago
It's up to the installed fonts really. I don't know if the combination of S + U is standardized as a Soviet Union flag emoji, but even if it is, your locally installed fonts may not contain every single flag emoji, so the browser would still fall back to rendering the two letters instead.
masklinn 5y ago
> the reversal of the U.S. flag should result in having the Soviet Union flag. Except it has been deleted from the ISO 3166-2 registry, so not having it is perfectly valid (arguably more so than having it).
- account42 5y ago
  No, that only shows that the ISO 3166-2 registry is a bad basis for Unicode flags since having things lose meaning over time should not be acceptable for a text encoding. Flags have another issue here in that they can change even when the country stays the same - a recent example here being Afghanistan, but also France who recently changed the official shades of the colors in their flag. Ideally you'd want a new Unicode representation for any changed flags in order to not retroactively change the meaning in old documents.
brewmarche 5y ago
Just tried reversing a Spanish flag with Python and indeed I got Sweden back
- kingcharles 5y ago
  No-one expects the Swedish flag!
5y ago

ts4z 5y ago

Let me cheat a bit and say Unicode comes in three flavors: UTF-8, UCS-2 aka UTF-16, and UTF-32. UTF-8 is byte-oriented, UTF-16 is double-byte oriented, and UTF-32 nobody uses because you waste half the word almost all of the time. You can't reduce the bytes in UTF-8 or UTF-16, because you'll scramble the encoding. But you could parsing the string, codepoint-at-a-time, handling the specifics of UTF-8, or UTF-16 with its surrogate pairs, and reversing those. This sounds equivalent to reversing UTF-32, and I believe is what the original poster was imagining. Except you can't do that, because Unicode has composing characters. Now, I'm American and too stupid to type anything other than ASCII, but I know about n+~ = ñ. If you have the pre-composed version of ñ, you can reverse the codepoint (it's one codepoint). If you don't have it, and you have n+dead ~, you can't reverse it, or in the word "año" you might put the ~ on the "o". (Even crazier things happen when you get to the ligatures in Arabic; IIRC one of those is about 20 codepoints.) So we can't just reverse codepoints, even ancient versions of Unicode. Other posters have talked about the even more exotic stuff like Emoji + skin tone. It's necessary to be very careful. Now, the old fart in me says that ASCII never had this problem. But the old fart in me knows about CRLF in text protocols, and that's never LFCR; and that if you want to make a ñ in ASCII you must send n ^H ~. I guess you can reverse that, but if you want to do more exotic things it becomes less obvious. (IIRC UCS-2 is the deadname, now we call it UTF-16 to remind us to always handle surrogate pairs correctly, which we don't.) TLDR: Strings are hard.

progbits 5y ago

Semi-related (about length of emoji "characters", not reversing): https://hsivonen.fi/string-length/ https://hsivonen.fi/string-length/ Previously discussed: https://news.ycombinator.com/item?id=20914184 https://news.ycombinator.com/item?id=20914184 https://news.ycombinator.com/item?id=26591373 https://news.ycombinator.com/item?id=26591373 As for this article & Python - as usual it is biasing towards convenience and implicit behavior rather than properly handling all edge cases. Compare with Rust where you can't "reverse" a string - that is not a defined operation. But you can either break it into a sequence of characters or graphemes and then reverse that, with expected results: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=6335bb2c5cacf308ed5190deaba0121f https://play.rust-lang.org/?version=stable&mode=debug&editio... (Sadly the grapheme segmentation is not part of standard library, at least yet)

account42 5y ago
> Sadly the grapheme segmentation is not part of standard library, at least yet Seeing as grapheme segmentation is a moving target that only makes sense.

aidenn0 5y ago

> The answer is: it depends. There isn't a canonical way to reverse a string, at least that I'm aware of. Unicode defines grapheme clusters[1] that represent "user-perceived characters" separating a string into those and reversing seems like a pretty good way to go about it. 1: http://www.unicode.org/reports/tr29/ http://www.unicode.org/reports/tr29/

tl 5y ago

This is a nice dive into limitations in Python's unicode handling and at the end, how to work around some problems. But you could use languages with proper unicode support like Swift or Elixir (weirdly HN is fighting flags in comment code which makes examples header to demonstrate).

anamexis 5y ago
HN doesn't allow any emoji.

jug 5y ago

I'm not surprised the flag had two components, but I _was_ surprised the US flag was made by literally U and S, haha! I definitely thought it'd be something like [I am a Flag] and [The flag ID between 0 and 65535]. And reversing it would be [Flag ID] + [I am a Flag] which would not be a defined "component" and instead rendered as the individual two nonsense characters.

andylynch 5y ago
You might also have noticed this is partly a very well thought out hack to make Unicode less sensitive to disagreements and changes in consensus on which flags are encoded, or even the names of the countries concerned!

bandyaboot 5y ago

Would be interesting to see the list of flag emojis that, when reversed, become a different flag emoji.

jfk13 5y ago
There are plenty of country codes that when reversed become a different, valid country code: e.g. Israel (IL) when reversed is Lithuania (LI); Australia (AU) becomes Ukraine (UA). Whether "reversing flag emojis" causes such transformations will depend on what is meant by "reversing", which is kind of the whole point here: there are a number of possible interpretations of "reverse".

alfredxing 5y ago

Related — I did a deep dive a couple years ago on emoji codepoints and how they're encoded in the Apple emoji font file, with the end goal of extracting the embedded images — https://github.com/alfredxing/emoji https://github.com/alfredxing/emoji

zwerdlds 5y ago

In normal conditions you can check for a ZWJ, but with regional coding chars, you would have to consider the regional chars block as a single char in the reversal. Given that is isn't necessarily locale dependant but presentation layer dependant, there might not be anough info to decide how to act.

jiveturkey 5y ago

Interesting article. Written for beginners, conversationally. Has excessive amounts of whitespace, for "readability" I guess. But at the same time, it dives quite deep, which I don't think this "style" of presentation matches up with the amount of time a more novice reader is going to devote to a single long form article. As to the content, for all the deep dive, a simple link to https://unicode.org/reports/tr51/#Flags https://unicode.org/reports/tr51/#Flags and what an emoji is, would have saved so much exposition. I also wish he'd touched on normalization. With the amount of time he's demanding from readers he could have mentioned this important subject. Because then he could discuss why (starting from his emoji example) a-grave (à) might or might not be reversible, depending how the character is composed. Also wish he'd pointed to some libraries that can do such reversals.

deleted 5y ago

[deleted]

techwiz137 5y ago

It's pretty funny that reversing the American flag yields Soviet Union(SU).

Crazyontap 5y ago

This section on the linked Wikipedia article(1) is quite amazing on how the family emoji is rendered using a zero-width joiner (1) https://en.wikipedia.org/wiki/Emoji#Joining https://en.wikipedia.org/wiki/Emoji#Joining edit: forgot HN doesn't render emojis. Better read it directly on Wikipedia i guess.

hougaard 5y ago

In other news, water is wet :)

mlindner 5y ago

The person tries to define character when there isn't actually any definition of what that even means. Character is a term limited to languages that actually use them and not all text is made up of characters.

emodendroket 5y ago

What I'd like to know is, given the explosion of the character set for emoji, does the rationale for Han unification still make sense? The case for not allowing national variants seems less and less compelling with every emoji they add. This is a bit of a hobby horse, but imagine if every time you read an article in English on your phone some of the letters were replaced with "equivalent" Greek or Cyrillic one and you can get an idea of the annoyance. Yeah, you can still read it with a bit of thought, but who wants to read that way?

digisign 5y ago
> were replaced with "equivalent" Greek or Cyrillic one The subset of equivalent letters, or different ones? If they looked the same, it wouldn't bother me if the letters in the center were a single codepoint between European languages: https://upload.wikimedia.org/wikipedia/commons/8/84/Venn_diagram_showing_Greek%2C_Latin_and_Cyrillic_letters.svg https://upload.wikimedia.org/wikipedia/commons/8/84/Venn_dia...
- account42 5y ago
  I am disappointed that that diagram omits ꙮ [0] [0] https://en.wikipedia.org/wiki/Multiocular_O https://en.wikipedia.org/wiki/Multiocular_O
- emodendroket 5y ago
  The problem is they don't look the same. So imagine, for instance, Я instead of "R" or И instead of "N" (I don't think the sounds are actually equivalent but let's run with it for the sake of example). Not insurmountable. One could still read a text with these substitutions. But it'd be distracting, and extra detrimental for people who don't speak English as their first language.
  - digisign 5y ago
    The ones in the center are in all three sets, they do look the same. The outer areas are out of bounds.
5y ago

raffy 5y ago

Kinda related: I am developing a library for ENS (Ethereum Name Service) name normalization: https://github.com/adraffy/ens-normalize.js https://github.com/adraffy/ens-normalize.js I'm trying to find the best combination of UTS-46, UTS-51, UTS-39, and prior work on IDN resolution w/r/t confusables: https://adraffy.github.io/ens-normalize.js/test/report-confusables.html https://adraffy.github.io/ens-normalize.js/test/report-confu... Personally, I found the Unicode spec very messy. Critical information is all over the place. You can see the direct effect of this when you compare various packages across different languages and discover that every library disagrees in multiple places. Even JS String.normalize() isn't consistent in the latest version of most browsers: https://adraffy.github.io/ens-normalize.js/test/report-nf.html https://adraffy.github.io/ens-normalize.js/test/report-nf.ht... (fails in Chrome, Safari) The major difference between ENS and DNS is emoji are front and center. ENS resolves by computing a hash of a name in a canonicalized form. Since resolution must happen decentralized, simply punting to punycode and relying custom logic for Unicode-handling isn't possible. On-chain records are 1:1, so there's no fuzzy matching either. Additionally, ENS is actively registering names, so any improvement to the system must preserve as many names as possible. At the moment, I'm attempting to improve upon the confusables in the Common/Greek/Latin/Cyrillic scripts, and will combine these new grouping with the mixed-script limitations similar to IDN handling in Chromium. Interactive Demo: https://adraffy.github.io/ens-normalize.js/test/resolver.html https://adraffy.github.io/ens-normalize.js/test/resolver.htm... Also this emoji report is pretty cool: https://adraffy.github.io/ens-normalize.js/test/report-emoji.html https://adraffy.github.io/ens-normalize.js/test/report-emoji...

nottorp 5y ago

So basically unicode along with c++ are great job security if you do bother to learn them. There's another word that comes to mind when thinking about those two: metastasis.

utopcell 5y ago

There are unicode characters that reverse parsing order themselves. This has been the basis of a code injection attack, analyzed in [1]. [1] ``Trojan Source: Invisible Vulnerabilities'': https://trojansource.codes/trojan-source.pdf https://trojansource.codes/trojan-source.pdf

randpx 5y ago

Try reversing the Canadian flag (CA) and you get the Ascension Island Flag (AC). Great article, but completely misses the point.

faebi 5y ago

Why reverse them if one barely can implement, display and edit them correctly. I never could make them work perfectly in VIM. Also I had to open a bug in Firefox recently: Flag emojis and others are displayed in double the size on Windows 10 using Firefox Nightly https://bugzilla.mozilla.org/show_bug.cgi?id=1746795 https://bugzilla.mozilla.org/show_bug.cgi?id=1746795

easrng 5y ago
Windows doesn't even have flag emojis, they just show up as the country code. Edit: Actually Firefox ships a copy of twemoji for fallback purposes, so flags will still render.

Waterluvian 5y ago

I wish languages did a far better job clearly distinguishing between their operations: 1. You are acting in byte space and it’s pretty unambiguous what should happen. We are not acknowledging the semantics of language and alphabets. 2. You’re acting in language space and these operations will behave the way you probably think they should (depending on your cultural expectations, probably)

cmyr 5y ago

Something I haven't seen mentioned yet is one of the most annoying things about regional indicator symbols, which is that interpreting them correctly requires arbitrary backtracking, and handling this correctly is very annoying for things like text fields. Basically: A single, unpaired RIS counts as a single grapheme. Similarly, a pair of RIS count as a single grapheme. Now imagine if your cursor position is after an RTS, and you arrow backwards (assuming LTR text, imagine your cursor is to the right of an RIS, and you press the left arrow.) Your textbox should now move the cursor to the left by one grapheme. How do you figure out where this is, in code units? You basically have to scan backwards until you find the first non-RIS codepoint, and then you have to match them up into pairs to figure out if your left-arrow movement should correspond to a movement of one or two codepoints. This is a longstanding source of bugs, and if you're bored you can play around with pasting a huge sequence of flags into a textfield and then trying to navigate around it with the arrow keys/mouse. There are some broken implementations out there. edit: while I'm thinking about this I will point out that an alternative design, which would have solved this problem (and which was first pointed out to me by @raphlinus) would have been to have two separate sets of RI symbols, one for 'first position' and one for 'second position'; then you could always determine the appropriate cursor position without needing context. Isn't hindsight a wonderful thing?

account42 5y ago
> Isn't hindsight a wonderful thing? Gladly, the creators of UTF-18 did have that foresight so at least we don't have this problem at the code unit -> code point level.

a_c 5y ago

Understanding unicode would make the question more obvious https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...

sundarurfriend 5y ago

Julia docs do a (surprisingly) good job of being clear and explicit about this: the docstring for `reverse(AbstractString)` says: > Reverses a string. Technically, this function reverses the codepoints in a string and its main utility is for reversed-order string processing [...]. See also [...] `graphemes` from module Unicode to operate on user-visible "characters" (graphemes) rather than codepoints. Properly reversing a string of flags (or any other grapheme clusters) is just a `using Unicode: grapheme` away.

ineedasername 5y ago

It's an emoji... Are there any emojis with only one character? My assumption going in would be that any emoji is > 1 character. Admittedly, despite lots of string processing, I never have to deal with emojis so I guess I'm not sure. An interesting exercise would be emoji detection during string reversal to preserve the original emoji. I though something like that would be the crux of the article. Am I wrong about single character emojis?

easrng 5y ago
It depends what you mean by character, there are lots of single codepoint emojis though.

DamnGirl 5y ago

nitely 5y ago

You can, but you need to break the string into graphemes first.

qwerty456127 5y ago

If the US flag is 2 special symbols saying US, why doesn't reversing it just produce the flag of the Soviet Union?

NullPrefix 5y ago
Same reason why there is no Nazi Germany flag - they are not included in Unicode.
- qwerty456127 5y ago
  Quite unfortunate - the US/SU case would make a nice Unicode+political pun.

demetrius 5y ago

It's sad that Unicode doesn't include flags for dissolved countries. If it did, reversing an US flag would make a Soviet Union flag (code SU). This would make the text much more fun

kragen 5y ago
The whole reason for handling the flag emojis that way was so that the Unicode Consortium wouldn't have to decide which countries should or should not be recognized. It is totally valid for you to configure your computer to display SU as a Soviet flag.

midjji 5y ago

And this is why char should have been byte from the start.

chrismorgan 5y ago

UTF-8 does not represent Unicode code points, but rather Unicode scalar values. The difference between the two is surrogates, the way that UTF-16 ruined Unicode: code points are 0₁₆ to 10FFFF₁₆, scalar values are 0₁₆ to D7FF₁₆ and E000₁₆ to 10FFFF₁₆. Yes, the author quoted Wikipedia, but Wikipedia is wrong on this point; surprisingly comprehensively wrong: the UTF-8 page completely ignores the distinction, and even the page on code points doesn’t mention scalar values! This error propagates to other places, too: for example, “and there are a total of 1,112,064 possible code points”: no, that’s how many scalar values there are; code points also include the 2,048 surrogates, so there are 1,114,112 code points.