11 ms·
Node's Unicode Dragon
- baddox 13y agoDespite that being a rather interesting technical article, I am upset that my expectation of an actual Unicode depiction of a dragon was not met.
- greenyoda 13y agoThere is actually a Unicode dragon character at code point U+1F409: http://www.fileformat.info/info/unicode/char/1f409/index.htm http://www.fileformat.info/info/unicode/char/1f409/index.htm Also, since any ASCII dragon is also a valid Unicode dragon (in UTF-8, at least), the following might satisfy your needs: http://www.dougsartgallery.com/ascii-art-dragon.html http://www.dougsartgallery.com/ascii-art-dragon.html
- deleted 13y ago[deleted]
- cirwin 13y ago🐉 To see this dragon, either: 1. Use Safari or Firefox on OS X. 2. Install custom fonts for Linux or Windows. 3. Install https://chrome.google.com/webstore/detail/chromoji-emoji-for-google/cahedbegdkagmcjfolhdlechbkeaieki https://chrome.google.com/webstore/detail/chromoji-emoji-for... for Chrome
- Wilya 13y agoNext time I have some "Here be dragons" code, I'm going to use this.
- pavlov 13y agoThe dragon glyph is rendered correctly in IE10 on Windows 8 without any custom fonts. Hooray for the most underestimated browser ever ;)
- city41 13y agoAlso true of mobile IE10
- drivebyacct2 13y agoWorks fine in IE11.0.9600.16384 as well as Firefox{,-ux,-nightly} in Windows and every browser in my Arch machine.
- lelf 13y agoThere is also 🐲 U+1F432 DRAGON FACE Also: didn't know that for every emoji there is https://en.wikipedia.org/wiki/🐉 https://en.wikipedia.org/wiki/🐉
- nonchalance 13y agoString encoding in general is a mess. Wait till you get to code pages. Incidentally, the largest JS script I've ever seen pertained to encoding and decoding characters under various codepages: https://raw.github.com/Niggler/js-codepage/master/cptable.js https://raw.github.com/Niggler/js-codepage/master/cptable.js [github complains "(Sorry about that, but we can't show files that are this big right now.)"]
- shawnz 13y ago> Unfortunately for us, Javascript has never been updated to support UTF-16. Instead it continues to treat strings as UCS-2. So really, they were parsing the JSON as if it were UTF-16, but really it was UCS-2. How is that an error in Node?
- justincormack 13y agoJSON is defined as UTF8, 16 or 32 [1]. The escaped characters are UTF-16 not UCS2. It is unfortunate of JavaScript can't parse it correctly! [1] http://www.ietf.org/rfc/rfc4627.txt http://www.ietf.org/rfc/rfc4627.txt
- kansface 13y agoThis is true of JSON, but its not true of Javascript which gives no fucks about utf16 (or valid surrogate pairs). Its a very strange world where JSON and Javascript have incompatible interpretations of strings. http://mathiasbynens.be/notes/javascript-encoding http://mathiasbynens.be/notes/javascript-encoding
- gnaritas 13y agoNot really as JSON is not valid JavaScript and requires its own parser. It's based on JavaScript, but it is not JavaScript.
- daxelrod 13y agoI was skeptical, but I did some searching, and you appear to be right! The difference seems to come down to string handling: http://timelessrepo.com/json-isnt-a-javascript-subset http://timelessrepo.com/json-isnt-a-javascript-subset
- gnaritas 13y agoHa, same article where I first learned this.
- state 13y agoWhew. This explains a bug from six months ago that drove me up the wall. I could never figure it out.
- dsj36 13y agohow did the error JSON include the undecodable bytes? JSON strings are all unicode sequences, so there would have had to be some way that the raw bytes were mapped into codepoints. on the other hand, if the offending bytes were blindly substituted into the JSON, then it's not surprising that there were decoding issues down the line...
- cirwin 13y agoIn the example I looked at to debug this, the sequence of events was: 1. One of our customer's javascript apps sent a truncated string to their web-server in a JSON payload. This string ended with a leading surrogate (this is another instance of V8 bug discussed in the blog post). 2. Their ruby backend exploded when they tried to use a regular expression on the string (because ruby's regexp library is strict about valid utf-8). 3. The bugsnag exception notifier copied the bytes from the incoming parameter into the JSON exception notification payload (ruby didn't notice because its string library unconditionally believes you if you tell it a string is valid utf8 — another bug :p).
- sujayakar 13y agoah yeah step 3 seems pretty bad -- cool that you found that bug!
- jlarocco 13y agoFrom the article: > The exceptions that were crashing us were caused by people using String.prototype.substr. That function works perfectly on strings that only contain Unicode 1.0 data, but as soon as you're storing UTF-16 in your UCS-2 string there's a possibility that when you take a slice you'll split a valid surrogate pair into two invalid lonely surrogates. To me, it seems like it'd be nearly impossible for somebody to trigger, but there's always Murphy's law...
- twoodfin 13y agoThese kinds of isolated surrogate pairs are pretty easy to create if you're doing the right kind of processing on the right kind of data. Suppose you receive a long piece of text wrapped in JSON, unpack it into a JS String, then start processing it in fixed size chunks. If your source text contains any significant percentage of surrogate pair-represented characters, you'll eventually break one.
- jrochkind1 13y agoThe OP describes an environment where data goes from node to Rails. If you want to check a string for valid encoding and/or replace bad bytes with replacement char on the _ruby_ end... it's not very obvious how you do that with the ruby stdlib api, and it takes a few tricks to do right. So I wrote a gem for it: https://github.com/jrochkind/ensure_valid_encoding https://github.com/jrochkind/ensure_valid_encoding
- stormbrew 13y agoWish I'd known about this when I was pointing out in another HN thread how utf-16 is a terrible encoding for, among other reasons, pushing the corner case where you find out your encoding/decoding is broken to the very edge of likelihood. It's ridiculous that v8 doesn't properly support utf16, but it's to be expected I suppose. UTF-8 does not have this problem. That's the way we should be moving.
- sillysaurus2 13y agoThis. Why doesn't everybody use UTF-8? Nobody seems to have any problems with UTF-8. It seems to work almost perfectly, and it's efficient.
- est 13y agoBecause some of us are pissed that some BMP characters takes 3 bytes in UTF8, that's 50% more waste of storage space and 50% more time to read/write. I like the design of Python 3.3 encoding. ASCII takes 1 byte, BMP takes 2 bytes, everything else 4 bytes. http://www.python.org/dev/peps/pep-0393/ http://www.python.org/dev/peps/pep-0393/
- stormbrew 13y agoThat's a reasonable replacement for ucs-4 for an internal representation, but it's not actually a character encoding like utf-8 and utf-16 are. It's just a tagged union of several encodings. As for the inflation issue, 50% is just the absolute worst case. Many kinds of textual data include large amounts of code units that fit in one byte in utf-8 and 2 bytes in utf-16. It tends to even out somewhat. And if you really want your data to be small, gzip will do a better job than either.
- est 13y ago> 50% is just the absolute worst case. Many kinds of textual data include large amounts of code units that fit in one byte in utf-8 For latin alphabets, yes. For CJK, it's really bad. Things get worse if you dealt with non-BMP before, like iOS emoji, which force you to upgrade MySQL to support utf8mb4, which is totally bullshit. (why the hell do people even presume utf8 is max 3 bytes?)
- ChrisAntaki 13y agoI can't believe NodeJS doesn't support Dragon symbols. This is a dealbreaker.
- scott_karana 13y agoIs it just me, or is the two-column layout a bit tricky for readability? (1440x900)
- justin_vanw 13y agoMan, I'm starting to think there is a cult around JSON. If you need to accept arbitrary binary data, JSON is a profoundly bad choice. At a minimum, you would expect them to base64 encode the data and put that into a JSON string. If you are looking at error reports, how is it even remotely acceptable to have them silently modified to include invalid unicode replacement characters? The lesson here isn't some crappy hack workaround they found, it's a case study in the lengths you'll have to go to when you insist on making technology choices without considering the problem you want to solve.
- derefr 13y agoAny wire-serialization format that wants to send arbitrary data should really have a "raw binary payload" type. XML has CDATA. ASN.1 has bitstrings. BERT has Binaries. But JSON doesn't really have anything like that. I wonder... at some point, Javascript could get a convenient literal syntax for creating pre-filled ArrayBuffers, which would basically be the format JSON would want to adopt. But would it? Are changes to Javascript literal syntax folded into JSON, or is JSON now its own thing that doesn't track JS any more?
- Dylan16807 13y agoCDATA disallows null bytes, so it's even worse than non-support: illusory support XML doesn't even allow escaped null bytes, so you're basically forced to use base64 or weird custom app-internal escapes. JSON never tracked javascript. It has one version, period. But you could get people to adopt a superset with a new data type, if you kept it simple.
- ygra 13y agoIsn't CDATA character data anyway and thus not even binary but in the document's character set? Which makes it a poor choice for binary data even without taking into account that XML forbids certain characters. As for binary data in web services ... isn't it easier to just use Content-Type for that and use the appropriate type for the payload? That wouldn't require a textual data format that can contain arbitrary binary data.
- scoopr 13y agoThis same problem manifests with Java as well, where some methods that claim to return UTF-8 on closer inspection actually return “modified UTF-8”, which is broken the same way. Notably I ran across this in with JNI function GetStringUTFChars, but may come across in DataOutputStream's writeUTF etc.
- bsaul 13y agoReminds me of a previous discussion about Go being more "mature" than node.js, where i said having someone like Pike on board gives you more than 30 years of "maturity". I'm pretty sure you wouldn't find those leaky UTF encoding handling in Go.
- pjscott 13y agoSince Rob Pike and Ken Thompson are the guys who came up with UTF-8, you'd expect them to write decent Unicode encoding for Go. It would be surprising if they didn't.
- ygra 13y agoWell, Node builds atop an established language, while Go is a new development. It's probably easier to build sane Unicode semantics into a new language than to change the JS spec.