21 ms·

I couldn't debug the code because of my name

258 points by mikasjp 5y ago

mikasjp 5y ago
I think the whole problem is keeping the character encoding consistent in the applications and their dependencies. Programmers often forget this because they avoid non-ASCII characters in their code.
xwdv 5y ago
What’s wrong with just writing it as Mikolaj? It’s not like it’s a kanji or something.
- wbsss4412 5y ago
  So the solution is for the user to change their entire windows account name, rather than handling common characters in your code?
  - toast0 5y ago
    For a user, changing their account (probably creating a new user, since rename apparently doesn't change the directory), is something they can do. Changing all software to respect their perfectly valid name isn't something they can do. They shouldn't need to change their name, but if they do, they can ignore all the broken software and go about their day. This particular user is more capable than most, and found a workaround for this particular problem, which is good... But this is not likely to be the last of the problems.
  - dahfizz 5y ago
    Of course it would be better if all code was bug free. But that's impossible. As a user, avoiding unicode is a pretty easy way to avoid bugs like this - its the rational thing to do.
  - Jensson 5y ago
    When you have non-standard characters in your name you quickly learn to never use them in computers since even though most systems works fine, some don't. And you can't fix all the thousands of systems your name has to interact with. I even had trouble booking flight tickets since their security system couldn't parse my name, and then had to go through some special security check due to it returning errors. After that, never again. Not sure how they managed to do it but they had some basic rules that they used to say "no real name can look like this, this is a fake person!" and just kicked it out.
    - q3k 5y ago
      > When you have non-standard characters in your name 'standard' by what measure? Ł is more standard than X or Q in the polish alphabet. ~ Sincerely, a person whose name contains „ń” and therefore had to deal with this bullshit his entire life.
      - Jensson 5y ago
        From a programmers perspective. The characters in my name are standard where I come from, but they are not standard to the international air travel security systems likely developed by Americans. Edit: You know how aircraft travel security always transforms your name into letters from the English alphabet to parse? Yeah, it transformed my name and then the resulting string looked so bad that the system rejected that. The original name doesn't look bad, but after transformations it did...
    - wbsss4412 5y ago
      I totally understand what you’re saying, but it’s also a sad state of affairs when we can’t handle “non standard characters”. Standard characters (ie english) are only used by a small subset (maybe 5-10%) of the global population.
    - yuliyp 5y ago
      They're not non-standard characters. They're just as much a part of the Polish alphabet as 'a' and 'b' are.
- needle0 5y ago
  Then there are the people whose names ARE in Kanji, thankyouverymuch. Ah, no big deal, there's only around 1.6 billion of us.
- miloignis 5y ago
  From the article: The first idea was to change the username to one that does not contain Polish characters. It turned out that Windows does not rename the user’s folder when changing the username. Manually renaming the folder was not an option. This way I could corrupt my profile in the system. The end of the article is about how to change the directory where the temporary files go to one not under the user folder.
- sophacles 5y ago
  Because that's not their name?
- dahfizz 5y ago
  Their URL is even mikolaj-kaminski.com . I get its annoying, but I would never use non-ascii chars in a username / file path.
  - jerf 5y ago
    So, what does 阿部明仁 do in this case? Polish may be close enough that an approximation is available in English, but there's an awful lot of languages that don't have a large overlap with English characters. In the Asian case above, if someone with that name did try to "convert to English" they are ironically just as likely to end up with ＡｋｉｈｉｔｏＡｂｅ as the ASCII, which will be just as broken!
    - dahfizz 5y ago
      They could use a different name as their windows name (Do people use their real names as their usernames? I never do). Or, they would have to go through the pain of finding a real solution, like the author did. Considering JetBrains seems unwilling to fix this bug, maybe the best solution of all is to switch to an IDE that works.
    - numpad0 5y ago
      Assuming that hypothetical guy is an average Japanese male(somewhat leaning right), he'd just turn IME off. Japanese input on desktop is consist of three following states: - IME On state. IME capture and interpret keypresses as engraved and generate corresponding Kana-Kanji texts. - IME Off state. IME passes through keypresses as engraved on keytops. - Direct Input state. IME becomes dormant. In IME Off state, the keyboard behaves as a plain jp106(or ANSI if it is) keyboard, like I'm doing right now. The cases where you would use conversion with IME on for an English word is when you have reasons for the word to be in "full width"(usually for typesetting reasons).
      - jerf 5y ago
        I don't think it's something that people should 'just know' that when Windows asks them their name during install time, they ought to use 7-bit clean ASCII for everything, no matter where they are in the world or how much they know about other languages. When Windows says "What is your name?", they ought to be able to use their name without things breaking. I'm sure a computer savvy speaker of a fully-non-Latin language may still guess this is a good idea, but "computer savvy" doesn't cover everyone... and they shouldn't have to. "Just use 7-bit-clean ASCII English" is not a solution to this problem.
- ludamad 5y ago
  For the record, it's a stark pronunciation difference as ł has drifted to a very different "w" sound
  - MadeThisToReply 5y ago
    Yep. For example, the name of the third-largest city in Poland is "Łódź", which might look like it's pronounced "lods", but is actually pronounced more like "wootch".
    - garaetjjte 5y ago
      Sometimes you end up with parcel addressed to city "??d?". Shipping systems cannot cope with non-ASCII chars more often than I would expect...
      - greenshackle2 5y ago
        I've seen shipping labels with HTML encoded characters, like é and è. I'm not sure if that's better or worse: &Lstrok;ód&zacute;
        qwerty456127 5y ago
        This is a pretty frequent thing to encounter actually. Just some years ago many websites actually preferred to use ISO/Windows codepages to save space on multibyte Unicode symbols adding HTML entities to represent everything which is not in it the basic ASCII and their primary language alphabet. Fun fact: I was looking for an e-mail solution for a small company about a decade ago and found Zarafa. It seemed nice and I deployed it happily. Just to find out it only supports the Western European ISO codepage which was hardcoded. I hope they have switched to UTF-8 since then.
      - qwerty456127 5y ago
        Always omit diacritics when giving your address/name to foreign parties which are going to have to ship you anything or issue any documents for you.
- ssivark 5y ago
  That’s about as aggravating as asking Ryan to change name to Pyan — because the encoding doesn’t support “R” and “P” looks very similar.
  - ufo 5y ago
    Interestingly, this particular change would work in cyrillic :)
    - ordu 5y ago
      Руап? No, "Яуап" is better, I think.
- no_time 5y ago
  Because it's not his name. Imagine you are John but you had to make do with Yohn because the people designing you software didn't need the letter J...
  - zanderwohl 5y ago
    It's not strange to localize your name. In ASL for example, you could sign your English name letter-by-letter, but it's much more common to have a totally new sign for your name - usually a word combined with the first letter of your name. Taking part in a different system often means taking on another name.
    - q3k 5y ago
      It seems that you're implying computers are universally american and therefore people are expected to speak/use/adapt to american.
      - deleted 5y ago
        [deleted]
      - thereddaikon 5y ago
        That's the harsh way to put it. A more diplomatic way is that computing is not unique in having deeply ingrained artifacts of the language and culture that birthed it and developed many of the paradigms. Take anything having to do with seamanship. There are many terms that date back to early modern English that simply don't make sense anymore yet are accepted and universal because the British Empire had a large and enduring influence on maritime matters and happened to be at the forefront of most modern developments until about 70 years ago. In some cases this is actually built into laws and industry practice. Pilots speak English. That's the rules. Don't like it? Invent the time machine and beat Wilbur and Orville. For much the same reason, science speaks Latin. This technical debt is difficult if not impossible to overcome, especially in regards to computers because we still haven't cracked general purpose AI. Software will only accommodate what it was written to accommodate. Recognizing the problem and working to fix it is all well and good. But its wise to understand that this wont be solved any time soon so in the meantime it is pragmatic to operate in such a way to maximize compatibility. After all, I still have to call it a Foc'sle even if I think that's dumb or isn't inclusive of my culture.
f311a 5y ago
That's a pretty common problem, especially for cyrillic names. People just use ASCII names.
deleted 5y ago
[deleted]
dmingod666 5y ago
The domain name to the website is all ascii..
- zamalek 5y ago
  If you use a Microsoft account to set up windows then you have no control over the local username.
  - dmingod666 5y ago
    That sucks.. always hated the idea of an online account to access your local system..
    - zamalek 5y ago
      I'm not saying it's a good idea (even though I stupidly do it), I'm merely pointing out that there are reasons that U0080+ may end up in a username; for reasons other than intentionally putting it in there. As for the benefits, which is completely off-topic, Windows Store is actually pretty awesome if you completely avoid search (and you need to do the Microsoft account thing for it AFAIK). Windows has needed a system to update 3rd-party software, to compete with Linux package managers, and the store is a really good effort (there are still annoying warts that Aur, Deb, RPM do not have). If you're willing to be a bit dumb, there is convenience.
  - moonchrome 5y ago
    This is exactly why I don't do that initially - I don't mind my account being linked - but I've been bitten by the home path bugs multiple times, I unplug my pc during setup
spicybright 5y ago
So frustrating how this still happens. It's too latin centric.
bagswatchesus 5y ago
Not sure how they managed to do it but they had some basic rules that they used to say "no real name can look like this, this is a fake person!" and just kicked it out. https://www.thelvbags.co/louis-vuitton-wallets-and-purses.html https://www.thelvbags.co/louis-vuitton-wallets-and-purses.ht...
tazjin 5y ago
The amount of random encoding problems that still exist are so bizarre. I recently left a UK job after already leaving the country more than a year ago, and in their attempt to mail P45 form to my new address (in Moscow) the only bits that survived are the string "c/o" and the postal code.
m_kos 5y ago
Isn't it bizarre that we have self-driving cars, the ISS, and phones with 50 megapixel cameras but still struggle with character encoding?
- zakius 5y ago
  for self driving cars, ISS and digital cameras everything you do is blurry in a sense, "good enough" approximation is actually good enough while character encoding and transformations have to be done perfectly and precisely and have surprisingly big number of edge cases
- tetha 5y ago
  Character encoding is in a special class of problems. Like time handling. If you pick up a halfway non-ancient framework in a somewhat common language with a somewhat non-terrible persistence like postgres, you just don't have problems. Just don't care, and it just works. But it's super easy to derail that fragile correctness with something like MySQLs utf8-ish handling, or some OS's path handling, or 'efficiency', or a user or frontend dev submitting data in a wrong encoding. And then it gets mangled. And then the user is unhappy. At that point, it becomes very hard to argue why one of the two things is wrong, and the other is not. While the user argues the other way around. Because both look correct, if you look from the right angle. And the only reason why I am right is because of some standard, while the customer is right because of money. And yes, it is very 'surprising' why our software now functions correctly for russian or greek customers.
  - ctdonath 5y ago
    That it's a special class of problems doesn't mean it shouldn't be solved by now. Time handling should be solved too; amazing that an iOS app can't get current correct GMT.
- quadrifoliate 5y ago
  It's not bizarre at all. Character encodings are a sort of language in themselves, and end up with all the problems that regular old languages have – there's a lot of variety, people can't agree on one particular solution, and there's not a lot of money in taking care of the edge cases. It would be bizarre if we were at the point where we had perfect translations for everything, but still struggled with character encodings specifically.
amarshall 5y ago
For a list of strings that often cause problems to, e.g., add to a test suite, see https://github.com/minimaxir/big-list-of-naughty-strings https://github.com/minimaxir/big-list-of-naughty-strings
- OskarS 5y ago
  An enormously useful list, I’ve used it several times, and it can often dig up some real nastiness if you haven’t been super careful. This entry, by the way, is a fantastic little easter egg in the list: https://github.com/minimaxir/big-list-of-naughty-strings/blob/db33ec7b1d5d9616a88c76394b7d0897bd0b97eb/blns.txt#L712 https://github.com/minimaxir/big-list-of-naughty-strings/blo...
  - vertis 5y ago
    No, seriously, wake up
    - deleted 5y ago
      [deleted]
    - jamiek88 5y ago
      That was creepy!
- tomaslaureano 5y ago
  Great resource! I usually use pangrams (holoalphabetic sentences like "The quick brown fox jumps over the lazy dog") to ensure that my code can handle all the alphabet characters for the languages that should be supported at the very minimum.
- ryanianian 5y ago
  Very handy. My previous simple test-case was simply a selection from this well-known text-file which is simply a collection of somewhat uncommon unicode characters, usually used for rendering tests. https://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-demo.txt https://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-demo.txt But this set of strings is specifically designed to cause edge-case errors. Also don't forget Spolsky's seminal "The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)". https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
mkotowski 5y ago
I, too, have the Ł letter in my name, and yes, it is a sick joke that so many things even in a supposedly modern systems make an assumption that the world runs on ASCII. In the case of the Windows operating system, the worst fact is that every single part of it behaves differently. Some parts display the path with a wrong encoding, but handle it correctly. A third-party app can display it correctly, but fails while trying to access any file. From what I remember, even the built-in PATH variable editor/manager goes through some arcane steps to display the letters in a wrong way, but getting them to work sometimes. I can only imagine how much more pain it is for someone using any of the less widely-used writing systems or those with more advanced features compared to ASCII (Hebrew’s RTL, Arabic scripts mid- and final forms, etcetera).
- gerdesj 5y ago
  Can Ł have an alternative representation? For example the German ß => ss. Also I think ö can be written as oe. In English we simply shake the big bag of letters, pick a few at random and then throw them at the page until a few stick.
  - q3k 5y ago
    > Can Ł have an alternative representation? Nope. Neither can ź, ć, ś, ą or ę. You can, and people do write them as z, c, s, a and e when writing in a restriced character set, but that is not 'correct' and is not a bijection, ie. „półka” and „polka” mean two different things. There's also the case of technically-same-sounding-especially-recently ż/rz and ó/u (whose replacement would let you get rid of two 'non standard' characters), but for historical reasons these are not interchangeable.
    - gerdesj 5y ago
      I do find this sort of stuff fascinating and also faintly frustrating but of course my mother tongue is (in)famous for being a bit loose at first sight. According to one of my employees (Polish) Ł sounds roughly like w as in win or water but not as in what. A quick read of this: https://en.wikipedia.org/wiki/%C5%81 https://en.wikipedia.org/wiki/%C5%81 doesn't help too much. Does enforcing Ł instead of say w cause your written language to fail in some way? I don't want to cause offense, I want to understand the causes of difference.
rcxdude 5y ago
Sadly there is even still software which fails to build or even fails to run when there is a space in a filename (as is super common on windows file paths, as well as autogenerated CI build folders). It's ridiculous to no end that software cannot handle paths correctly.
tediousdemise 5y ago
The solution to this is extremely simple: don't validate usernames, period. The rationale is from an article someone linked here ("Falsehoods Programmer's Believe About Names"): > Anything someone tells you is their name is—by definition—an appropriate identifier for them. If you try to validate by checking for profanity, knowing full well that people can have names that contain profane substrings, I have a tongue-in-check message for you—you are a fucking asshole.
anotheraccount9 5y ago
When ł visiłed his page, my browser crashed.
lukaszkups 5y ago
Ah it's so common for me that I've totally abandoned using Latin letters in my first/last name long time ago (and I recommend the same for you ;))
tyteen4a03 5y ago
Fun fact: If you have the exclamation mark (!) in your Windows username, Java will think it's the jar separator and `getResourceAsStream` will refuse to work. This broke many people's Minecraft installation over the years. The bug in question [0] was reported in 2001 and remains unsolved 20 years later. [0] https://bugs.java.com/bugdatabase/view_bug.do?bug_id=4523159 https://bugs.java.com/bugdatabase/view_bug.do?bug_id=4523159
- alisonkisk 5y ago
  The only practical fix would be to ban ! in usernames.
- kspacewalk2 5y ago
  I find the very idea of putting an exclamation mark on one's username and not expecting eventual problems to be quite curious.
  - edaemon 5y ago
    It can be used to indicate a click sound in some languages and the IPA. Probably not the most common reason to come across this bug but it's a case worth considering. https://en.wikipedia.org/wiki/Exclamation_mark#Phonetics https://en.wikipedia.org/wiki/Exclamation_mark#Phonetics
  - jmopp 5y ago
    The Nama language has a letter that is often replaced with an exclamation mark in common typography. The lead actor in the film The God's Must be Crazy was named N!xau ǂToma
  - vincnetas 5y ago
    i recommend reading this "Falsehoods Programmers Believe About Names". world is crazy place if you expect some system in how people name each other and are writing it down. https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/ https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...
  - 5y ago
jonnycomputer 5y ago
>When I found out that the bug was in the Rider itself, I reported it to technical support. I also found a similar report for PyCharm. Unfortunately, things haven’t moved forward since then. Unfortunately, typical with Jetbrains.
trinovantes 5y ago
In CS, most algorithms assume an ASCII character set. I wonder if there's any string-related algorithms that completely break (functionally or complexity wise) when given UTF-16 or UTF-8 character sets
- deathanatos 5y ago
  > In CS, most algorithms assume an ASCII character set. They most certainly do not. E.g., a Turing machine assumes an alphabet Γ which is a set of some characters and is defined no further, as any exact definition is meaningless to the theory. (I.e., the algorithm is generic over any alphabet.) The alphabet need not even be text; e.g., for a Turing machine, the set of all octets suffices. Even for something like Levenshtein distance, the only real requirement of the algorithm is that the abstract "characters" implement equality testing. For Unicode text, I'd start with graphemes, and then look for counter examples.
  - trinovantes 5y ago
    I guess they won't break correctness but I do remember many algorithms (e.g. tries) assume you have constant random access to characters which AFAIK is not possible in UTF-8
- a1369209993 5y ago
  Asymtotic complexity can't change based on the character set, since you can just reuse the same algorithm with larger opaque datums. (Exception being algorithms with O(n^8) or O(n^256) complexity, but noone uses those anyway.) A variable width encoding can cause issues in principle, but useful algorithms already have to deal with strings that have variable-length physical represention anyway (eg "yes" vs "no"), so it tends not to be a problem in practice.
askvictor 5y ago
Somewhat surprising that this is an issue with JetBrains, given that they are based in Eastern Europe, and would probably have more direct experience of these sorts of problems than US or UK based companies. OTOH maybe it's just a scale thing - bigger companies have more resources to handle these sort of cases, regardless where they're based (not that they always do...)
PrivateButts 5y ago
Similar to this, Node and NPM get very temperamental when you have a User folder with a space in it. I gave up on the community workarounds and just created a new account and copied my files over to fix it.
godmode2019 5y ago
I have a set of names I give to different providers. Advertisers always assume a name is constant and email addresses can change. I got a name saying 'Hi John I just want to xyz' I can skip this email as they used a fake name. Works better than other methods I have found.
auggierose 5y ago
Haha, that's why something like Cosmopolitan Identifiers would be a good idea: https://doi.org/10.47757/obua.cosmo-id.3 https://doi.org/10.47757/obua.cosmo-id.3
souptonuts 5y ago
Idk changing your stupid fucking name could be a fix too
shantnutiwari 5y ago
the bug was fixed one hour ago-- looks like HN customer service worked again
supernes 5y ago
It's somewhat common to see videogames issue a patch shortly after release where they fix crashes due to non-ASCII Windows usernames or non-English locales. I'm not sure what the root cause of the confusion is, other than text strings being hard in general.
- jerf 5y ago
  It's easy to think the answer is "just UTF-8 everything" but unfortunately the long and twisty history of filesystems means that's not the correct answer, and the "correct answer" is really hard to write down quickly. If you never display the filename, the answer is to treat existing filenames as bags of bytes, but that breaks down as soon as you need to display them, or if you need to manipulate them by appending unicode to them, in which case you have to decide on an encoding. Unicode encodings tend to mangle non-Unicode values because they're specified to replace whatever they can't understand with a particular Unicode character, usually represented as a diamond with an inverted ? inside of it. There's some obscure solutions to this problem, like https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ (which includes discussion of the 16 bit encodings you need for Windows), but I haven't seen broad support for them. You need a deliberately "noncompliant" encoding/decoding system that doesn't replace unknown characters with replacement characters. Fortunately, compliant systems are becoming more and more popular and available. Unfortunately, that can make file name handling harder than when you had a non-Unicode-compliant handling system for your strings.
  - nyanpasu64 5y ago
    Rust uses WTF-8 on Windows for OsStr[ing] and Path[Buf]. It's zero-overhead to cast from &str to &OsStr/&Path to &[u8] (though converting WTF-8 to UTF-16 costs an extra operation when performing a Win32 function call). However this doesn't solve the inability to round-trip "possibly-valid UTF-8/16" to "Unicode text" and back (though Python's surrogateescape might be one viable approach). Other libraries handle this even worse than Rust. On Linux (filenames are bytes), Qt is unable to open files with invalid UTF-8 names, while GTK can open them (but shows an "invalid encoding" message instead of the original filename), which I think is a good-enough approach.
simonblack 5y ago
Isn't this one of those "100 things Programmers don't know about People's Names" things? Like the poor, it will be with us always.
- xdfgh1112 5y ago
  I don't know, it's just a Unicode character? Not even a newer one, it's just 2 utf8 bytes. Pretty much everything should support that in 2021. When I think of 100 things I think of stuff like "some people spell their name in all lowercase and get really funny if you change it"
  - numpad0 5y ago
    Yeah so double byte characters costs extra. I don’t know, a checkbox or something default off. Always did still does. Double width costs even more.
    - horsawlarway 5y ago
      you're getting downvoted, but between tchar hiding wchar vs char... this literally could be someone toggling off the "UNICODE" checkbox in visual studio somewhere.
  - hprotagonist 5y ago
    windows probably defaults to latin-1
    - bryanrasmussen 5y ago
      the default windows encoding is UTF-16, a long time ago it was Windows-1252 https://en.wikipedia.org/wiki/Windows-1252 https://en.wikipedia.org/wiki/Windows-1252
      - hprotagonist 5y ago
        or CP-1251, in some locations.
jasonpeacock 5y ago
And yet it's one of the simplest things to add non-ASCII chars to your tests to validate their handling. It's like not testing if your calculate application can handle negative numbers or decimals.
- nradov 5y ago
  In fact it's trivial to generate a text file of all valid Unicode code points and use that as input to unit tests.
  - yakubin 5y ago
    It may be faster to generate them on the fly. Iterating over ranges of integers is a lot faster than reading files from disk.
  - Someone 5y ago
    I would have to do research on whether the list of valid code points depends on the Unicode version. For example, can regional indicator code points (https://en.wikipedia.org/wiki/Regional_indicator_symbol https://en.wikipedia.org/wiki/Regional_indicator_symbol) appear in isolation? If not, is that different in Unicode < 6, where those code points weren’t assigned yet? Similarly, what about tags (https://en.wikipedia.org/wiki/Tags_(Unicode_block) https://en.wikipedia.org/wiki/Tags_(Unicode_block) )? Do these require an U+E007F CANCEL TAG? The 66 noncharacters certainly need consideration. http://www.unicode.org/faq/private_use.html http://www.unicode.org/faq/private_use.html says: “Because of this complicated history and confusing changes of wording in the standard over the years regarding what are now known as noncharacters, there is still considerable disagreement about their use and whether they should be considered "illegal" or "invalid" in various contexts” Edit: also, testing all code points likely is overkill and using code points in isolation likely isn’t enough. Most tests are better of with something like the big list of naughty strings (https://github.com/minimaxir/big-list-of-naughty-strings https://github.com/minimaxir/big-list-of-naughty-strings)
umvi 5y ago
Using non-ascii characters in file paths, toolchain config files, and other non-display contexts is just asking for trouble, even if it is your name...
- fluxem 5y ago
  Also spaces. I spent half an hour debugging why cmake cuda build was failing.
  - munk-a 5y ago
    A lack of support for spaces at this point is unacceptable. I, personally, despise spaces in paths but on windows a whole bunch of default system paths already have spaces embedded in them in major ways... and let's not forget parens as well - thanks "Program Files (x86)"
- b112 5y ago
  This wouldn't have happened if using rust!
  - nightfly 5y ago
    Can you knock it off??? This is even more annoying that out-of-place rust evangelism
    - GoblinSlayer 5y ago
      You know he's right. Look at all the rust converts preaching their dogmas here.
  - burnished 5y ago
    Some of the other attempts are a little subtle, this one is a pretty blatant attempt to rile up the folks that are already angry about rust for whatever reason. Please stop.
- jasonpeacock 5y ago
numpad0 5y ago
Oh, it’s not a common knowledge that you should not UTF-8 in Windows username? That had been the case since 95 days. Only recently it had supposedly improved after Microsoft Account login become semi mandatory.
- progval 5y ago
  On the contrary, the first bug happens because docker-compose tries to decode the path as UTF-8, but it is not UTF-8-encoded. ("'utf-8' codec can't decode byte")
- chris_overseas 5y ago
  I don't think this bug is anything to do with Windows, rather it is due to the way the paths are handled in the IDE's codebase. Presumably the same problem exists when using these IDEs in conjunction with a path containing non-ascii characters in the Linux or macOS world.
  - numpad0 5y ago
    Isn't it some compilation option issue in native part? I thought it's a line on .sln or include library in a C++ source or something that has to be explicitly specified when building a Win32 binary.
    - GoblinSlayer 5y ago
      InteliJ has native part?
  - account42 5y ago
    > Presumably the same problem exists when using these IDEs in conjunction with a path containing non-ascii characters in the Linux or macOS world. Why would you presume that when the problem seems to be that one tool uses the systems native 8-bit encoding while another tool expects UTF-8 - under sane systems these are the same.
- Fordec 5y ago
  A lot of adults today weren't even alive in 95. Also, the assumption that people are familiar with windows vs other operating systems is becoming less and less valid. And as the world gets more globalised and remote, it's no longer to be assumed that all technical people are of a Anglo American culture.
sschueller 5y ago
Many years ago I could not access the apple developer panel because of the umlaut in my last name. It was eventually fixed but I was quite surprised that such a large company would run into such a basic issue.
- deleted 5y ago
  [deleted]
- devrand 5y ago
  My last name has an apostrophe in it which Apple apparently loves to embed directly into their JavaScript unescaped. For a long time neither I nor Apple could look up AppleCare status on my stuff as they were all linked to my Apple ID. The portal would thus require me to login, but then would just show a partially rendered page as my last name was causing an JS syntax error.
  - doubled112 5y ago
    You'd think the apostrophe would be common enough they'd know it could happen, but no. I love to enter it and see what each vendor and website's backend does with it. The Staples Canada website, for example, returns it as ' (HTML escaped) A couple times I've logged in, it seems to escape a new character. I'm currently up to &amp;#39;
    - devrand 5y ago
      Haha yeah I'm fairly used to seeing HTML escaping in my name. The weirdest case I've had with that is the Six Flags mobile app. To add a season pass you need to provide your card number and last name. For the life of me I couldn't get it to validate, but I saw they showed the HTML escaped version in their e-mails to me. Turns out I had to type out "'" into their input box for my last name as that's apparently what they put in their database.
  - nneonneo 5y ago
    Hmm, it sure sounds like John <script>alert(1);</script>Doe (Bobby Tables' distant cousin) should sign up for an Apple account. An XSS attack which could target the AppleCare reps' machines could be catastrophically bad...
pledess 5y ago
The article offers a solution of idea.system.path=${root.dir}/JetBrains/Rider/system but doesn't mention the C:\JetBrains directory permissions. Directory permissions under %LOCALAPPDATA% (the location that works for people without a Polish character) should restrict write access to one user. With the Windows default behavior, creating C:\JetBrains would inherit permissions from C:\ - and wouldn't restrict write access to one user. Maybe 99% of the time this is irrelevant (i.e., there's no realistic threat from malicious actors who control unprivileged user accounts on your own development machine). Still, it's a potential downside of the solution, and more motivation for the vendor to fix their code so that Polish characters can be used under %LOCALAPPDATA%.
- Kwpolska 5y ago
  If you are on a multi-user system, the path "C:\JetBrains" isn’t really ideal (what if other users also need Rider and have non-ASCII usernames?). That said, you can easily change file permissions on Windows if the default ones don’t work for you.
mrweasel 5y ago
It’s a pretty good test case. Similarly we found a number of bugs in a Django application and path handling, because I happend to be using Windows for six months, while the rest of the team was on Linux and Mac.
xlii 5y ago
Very similar problem to one described started my exodus from Google services. I also have non-latin characters in my name however I knew it was always an issue so I never used it in paths etc. At some point, long time ago, I was tasked to do some maintance with Google Cloud service (can't remember the name of the service now) which was doable only through Python CLI utility and it failed with very similar Python error. What I found out rather quickly is that utility took my name from Google+ profile, which did include those non-latin characters. No biggie - I thought and fired e-mail to support (yeah it was those times it was still that easy). Few hours passed and I received information that this won't be fixed anytime soon and the best course of action would be to change my name. Of course, support person probably meant to remove the diacriticals from my Google+ profiles, but still it left unplesant aftertaste for years to come.
- perl4ever 5y ago
  A Polish relative of mine used to just gave an arbitrary substitute name (e.g. "Dave Smith") for restaurant reservations, because even if they could write his last name, they wouldn't be able to pronounce it. My sibling has a name that has an accent, and just enters it with the plain letter most of the time. The name was once rare and "ethnic", but became popular a generation later so people know how to pronounce it regardless. Our parents gave us two middle names, wanting to preserve our grandmothers' surnames, but also in the spirit of "Bobby Tables", having ambivalent feelings about the computerization of society tending towards inflexibility. According to: https://en.wikipedia.org/wiki/Naming_customs_of_Hispanic_America https://en.wikipedia.org/wiki/Naming_customs_of_Hispanic_Ame... ...misunderstanding of naming customs in the US has actually led to significant consequences due to last names not matching on legal documents. I remember reading a story about how there are people in China whose name incorporates a character that is obscure enough, the authorities are trying to eliminate it and get them to change their name. If I recall correctly, Chinese has a particular problem with characters that are part of names that have been around forever, but are no longer used for ordinary writing.
  - em-bee 5y ago
    a friend of mine has that problem. his company is owned by his wife because his name can't appear in legal documents and he refuses to change it. his approach was that he petitioned unicode to include that character. i don't think the authorities are actively trying to eliminate those characters but simply don't want to go through the effort to track down and have those characters added to the standard. the process takes years and in the mean time you have to live with the inconvenience. also most of the people faced with the problem would not even know how.
ygra 5y ago
One way of working arrive such issues is to use subst. That way the application thinks your project directory is actually located on P:\ or something like that.
Dannymetconan 5y ago
I can very much relate to this but also have very little sympathy here. I have a special character in my name, an apostrophe, and it causes trouble regularly online and with tooling. A number of years ago I decided just to never use it when it came to anything to do with technical work be it email, logins or usernames. Unicode characters are a pain to deal with and I have suffered from it first hand trying to handle it. At the end of the day it is much easier just to not use the special characters and move on with your life rather then be battling the constant frustration. I'm sure these tools have lots of issues opening and you would be surprised at the amount of time, effort and testing it would be required to provide fully Unicode support. Most people would see it as a very small positive and not worth the effort. I find it hard to disagree.
- jltsiren 5y ago
  My legal last name is "Sirén". When I was younger, I almost always used "Siren", because it was easier to type. Then, ~15 years ago, I started noticing that American websites sometimes rejected it, because they considered it inappropriate. Sometimes "Sirén" would work, sometimes it worked but caused minor annoyances, and sometimes it would not work for technical reasons. Both versions work most of the time these days, but I still run into trouble once in a while no matter which name I use.
  - Dannymetconan 5y ago
    Totally agree with the sentiment. It has gotten a lot better in the last 10 years. Very frustrating to have your name blacklisted by that. It does seem most system have a very US focused design. I still find it funny that even in my home country you can't use a lot of local special characters in names. Also most airlines won't accept it so technically I'm not giving them my true name!
  - 10000truths 5y ago
    Why would Siren be an inappropriate name?
    - 5y ago
david422 5y ago
There's also this article: falsehoods-programmers-believe-about-names: https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/ https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-... Certainly informative if you haven't seen it before. My takeaway from it was that design your system to try to accommodate as much as possible, but it would basically be impossible to accommodate them all, so aim for your target audience.
darkhorn 5y ago
I think it is a Java related issue. Relevant issue occurs in Jaspersoft Report. You cannot install Jaspersoft Report on Turkish Windows no matter what.
SergeAx 5y ago
When I first installed Windows 7 like ten years ago, I entered my Russian name in Cyrillic. When I saw that the system created a directory with exactly that name under `C:\Users\` I immediately scanned the internet for a way to rename it and done just that. I don't want to know how much mess like that in a story I thus had successfully escaped. NB: the method is still the same, it's a second (not accepted) answer here: https://superuser.com/questions/890812/how-to-rename-the-user-folder-in-windows-10 https://superuser.com/questions/890812/how-to-rename-the-use... (about ProfileImagePath registry value).
- vertis 5y ago
  This is sad though. You shouldn't have to change who you are for a computer program.
  - GoblinSlayer 5y ago
    Is "vertis" who you are? There's more to a human, than a name.
  - SergeAx 5y ago
    I have this lower ASCII handle since about 1990, I beleive. That was the time when you just can't do literally anything without one.
ddeyar 5y ago
Some years ago I used the + feature in my gmail address. e.g. myname+ycombinator@gmail.com to track down which service is giving away my email address. It happened more than once that I could not log in anymore at some point because they started to disallow the + character in email addresses. I also got phone calls from some companies complaining that i misspelled my email address because there was their company name in it.
- cgufus 5y ago
  hehe, did the same, although not with +, but using a catch-all feature of the provider. I still get a lot of spam and phishing attempts on my „dropbox@<mydomain>“ address. I faintly remember they (dropbox) had a breach some time in the past.
asimjalis 5y ago
This is like Kafka’s story in which the protagonist wakes up to find out he’s a (software) bug.
Ansil849 5y ago
Sometimes even "regular" ASCII surnames cause problems. When written in the Latin alphabet, my surname is one letter. I've had an amazing amount of problems with this not just due to technical limitations (like various forms marking the entry as invalid), but--much more aggravatingly--human limitations. One particularly infuriating anecdote: at a past job many years ago, the email structure was lastname@company.com. I dutifully sent the IT person in charge of creating emails my desired email. The IT person wrote back an amazingly condescending email that as per the policy, emails had to be last names, not individual letters. I then had to go find a bunch of random websites which explained single-letter names and forwarded them to the IT person. They then obliged, but did not apologize for insulting me. That is not right that I had to put up with that.
- account42 5y ago
  > One particularly infuriating anecdote: at a past job many years ago, the email structure was lastname@company.com. I dutifully sent the IT person in charge of creating emails my desired email. The IT person wrote back an amazingly condescending email that as per the policy, emails had to be last names, not individual letters. I then had to go find a bunch of random websites which explained single-letter names and forwarded them to the IT person. They then obliged, but did not apologize for insulting me. That is not right that I had to put up with that. Except single letter last names are less common than people not following policy and/or abbriviating the name. It could simply be an honest mistake and the email is just their standard response since they have other things to get to. Did you try simply pointing out that that the letter was in fact your last name instead of getting passive-agressive?
deleted 5y ago
[deleted]
Natfan 5y ago
I've also had issues putting in my full name as my username. Lots of programs do not expect spaces in the path, and I experience a lot of errors which are resolved by changing the path to not contain a space. [1]: https://github.com/microsoft/WSL/issues/2577#issuecomment-901815008 https://github.com/microsoft/WSL/issues/2577#issuecomment-90...
itsrajju 5y ago
As of 2 hours before me writing this comment, JetBrains claims to have fixed the underlying issue [0]. Maybe they saw this post? :D [0]: https://youtrack.jetbrains.com/issue/IDEA-264563 https://youtrack.jetbrains.com/issue/IDEA-264563
deepsun 5y ago
> My username contains a "ł" character and because of it, this file cannot be processed properly. What is so curious there? Some names contain all non-latin characters, and some softwares don't work with non-ASCII symbols. I just cannot understand why is it interesting.
Svoka 5y ago
Did you know that Android still won't build on Windows if you have Cyrillic letter in user name?

258 points by mikasjp 5y ago

I think the whole problem is keeping the character encoding consistent in the applications and their dependencies. Programmers often forget this because they avoid non-ASCII characters in their code.

xwdv 5y ago

What’s wrong with just writing it as Mikolaj? It’s not like it’s a kanji or something.

wbsss4412 5y ago
So the solution is for the user to change their entire windows account name, rather than handling common characters in your code?
- toast0 5y ago
  For a user, changing their account (probably creating a new user, since rename apparently doesn't change the directory), is something they can do. Changing all software to respect their perfectly valid name isn't something they can do. They shouldn't need to change their name, but if they do, they can ignore all the broken software and go about their day. This particular user is more capable than most, and found a workaround for this particular problem, which is good... But this is not likely to be the last of the problems.
- dahfizz 5y ago
  Of course it would be better if all code was bug free. But that's impossible. As a user, avoiding unicode is a pretty easy way to avoid bugs like this - its the rational thing to do.
- Jensson 5y ago
  When you have non-standard characters in your name you quickly learn to never use them in computers since even though most systems works fine, some don't. And you can't fix all the thousands of systems your name has to interact with. I even had trouble booking flight tickets since their security system couldn't parse my name, and then had to go through some special security check due to it returning errors. After that, never again. Not sure how they managed to do it but they had some basic rules that they used to say "no real name can look like this, this is a fake person!" and just kicked it out.
  - q3k 5y ago
    > When you have non-standard characters in your name 'standard' by what measure? Ł is more standard than X or Q in the polish alphabet. ~ Sincerely, a person whose name contains „ń” and therefore had to deal with this bullshit his entire life.
    - Jensson 5y ago
      From a programmers perspective. The characters in my name are standard where I come from, but they are not standard to the international air travel security systems likely developed by Americans. Edit: You know how aircraft travel security always transforms your name into letters from the English alphabet to parse? Yeah, it transformed my name and then the resulting string looked so bad that the system rejected that. The original name doesn't look bad, but after transformations it did...
  - wbsss4412 5y ago
    I totally understand what you’re saying, but it’s also a sad state of affairs when we can’t handle “non standard characters”. Standard characters (ie english) are only used by a small subset (maybe 5-10%) of the global population.
  - yuliyp 5y ago
    They're not non-standard characters. They're just as much a part of the Polish alphabet as 'a' and 'b' are.
needle0 5y ago
Then there are the people whose names ARE in Kanji, thankyouverymuch. Ah, no big deal, there's only around 1.6 billion of us.
miloignis 5y ago
From the article: The first idea was to change the username to one that does not contain Polish characters. It turned out that Windows does not rename the user’s folder when changing the username. Manually renaming the folder was not an option. This way I could corrupt my profile in the system. The end of the article is about how to change the directory where the temporary files go to one not under the user folder.
sophacles 5y ago
Because that's not their name?
dahfizz 5y ago
Their URL is even mikolaj-kaminski.com . I get its annoying, but I would never use non-ascii chars in a username / file path.
- jerf 5y ago
  So, what does 阿部明仁 do in this case? Polish may be close enough that an approximation is available in English, but there's an awful lot of languages that don't have a large overlap with English characters. In the Asian case above, if someone with that name did try to "convert to English" they are ironically just as likely to end up with ＡｋｉｈｉｔｏＡｂｅ as the ASCII, which will be just as broken!
  - dahfizz 5y ago
    They could use a different name as their windows name (Do people use their real names as their usernames? I never do). Or, they would have to go through the pain of finding a real solution, like the author did. Considering JetBrains seems unwilling to fix this bug, maybe the best solution of all is to switch to an IDE that works.
  - numpad0 5y ago
    Assuming that hypothetical guy is an average Japanese male(somewhat leaning right), he'd just turn IME off. Japanese input on desktop is consist of three following states: - IME On state. IME capture and interpret keypresses as engraved and generate corresponding Kana-Kanji texts. - IME Off state. IME passes through keypresses as engraved on keytops. - Direct Input state. IME becomes dormant. In IME Off state, the keyboard behaves as a plain jp106(or ANSI if it is) keyboard, like I'm doing right now. The cases where you would use conversion with IME on for an English word is when you have reasons for the word to be in "full width"(usually for typesetting reasons).
    - jerf 5y ago
      I don't think it's something that people should 'just know' that when Windows asks them their name during install time, they ought to use 7-bit clean ASCII for everything, no matter where they are in the world or how much they know about other languages. When Windows says "What is your name?", they ought to be able to use their name without things breaking. I'm sure a computer savvy speaker of a fully-non-Latin language may still guess this is a good idea, but "computer savvy" doesn't cover everyone... and they shouldn't have to. "Just use 7-bit-clean ASCII English" is not a solution to this problem.
ludamad 5y ago
For the record, it's a stark pronunciation difference as ł has drifted to a very different "w" sound
- MadeThisToReply 5y ago
  Yep. For example, the name of the third-largest city in Poland is "Łódź", which might look like it's pronounced "lods", but is actually pronounced more like "wootch".
  - garaetjjte 5y ago
    Sometimes you end up with parcel addressed to city "??d?". Shipping systems cannot cope with non-ASCII chars more often than I would expect...
    - greenshackle2 5y ago
      I've seen shipping labels with HTML encoded characters, like é and è. I'm not sure if that's better or worse: &Lstrok;ód&zacute;
      - qwerty456127 5y ago
        This is a pretty frequent thing to encounter actually. Just some years ago many websites actually preferred to use ISO/Windows codepages to save space on multibyte Unicode symbols adding HTML entities to represent everything which is not in it the basic ASCII and their primary language alphabet. Fun fact: I was looking for an e-mail solution for a small company about a decade ago and found Zarafa. It seemed nice and I deployed it happily. Just to find out it only supports the Western European ISO codepage which was hardcoded. I hope they have switched to UTF-8 since then.
    - qwerty456127 5y ago
      Always omit diacritics when giving your address/name to foreign parties which are going to have to ship you anything or issue any documents for you.
ssivark 5y ago
That’s about as aggravating as asking Ryan to change name to Pyan — because the encoding doesn’t support “R” and “P” looks very similar.
- ufo 5y ago
  Interestingly, this particular change would work in cyrillic :)
  - ordu 5y ago
    Руап? No, "Яуап" is better, I think.
no_time 5y ago
Because it's not his name. Imagine you are John but you had to make do with Yohn because the people designing you software didn't need the letter J...
- zanderwohl 5y ago
  It's not strange to localize your name. In ASL for example, you could sign your English name letter-by-letter, but it's much more common to have a totally new sign for your name - usually a word combined with the first letter of your name. Taking part in a different system often means taking on another name.
  - q3k 5y ago
    It seems that you're implying computers are universally american and therefore people are expected to speak/use/adapt to american.
    - deleted 5y ago
      [deleted]
    - thereddaikon 5y ago
      That's the harsh way to put it. A more diplomatic way is that computing is not unique in having deeply ingrained artifacts of the language and culture that birthed it and developed many of the paradigms. Take anything having to do with seamanship. There are many terms that date back to early modern English that simply don't make sense anymore yet are accepted and universal because the British Empire had a large and enduring influence on maritime matters and happened to be at the forefront of most modern developments until about 70 years ago. In some cases this is actually built into laws and industry practice. Pilots speak English. That's the rules. Don't like it? Invent the time machine and beat Wilbur and Orville. For much the same reason, science speaks Latin. This technical debt is difficult if not impossible to overcome, especially in regards to computers because we still haven't cracked general purpose AI. Software will only accommodate what it was written to accommodate. Recognizing the problem and working to fix it is all well and good. But its wise to understand that this wont be solved any time soon so in the meantime it is pragmatic to operate in such a way to maximize compatibility. After all, I still have to call it a Foc'sle even if I think that's dumb or isn't inclusive of my culture.

f311a 5y ago

That's a pretty common problem, especially for cyrillic names. People just use ASCII names.

deleted 5y ago

[deleted]

dmingod666 5y ago

The domain name to the website is all ascii..

zamalek 5y ago
If you use a Microsoft account to set up windows then you have no control over the local username.
- dmingod666 5y ago
  That sucks.. always hated the idea of an online account to access your local system..
  - zamalek 5y ago
    I'm not saying it's a good idea (even though I stupidly do it), I'm merely pointing out that there are reasons that U0080+ may end up in a username; for reasons other than intentionally putting it in there. As for the benefits, which is completely off-topic, Windows Store is actually pretty awesome if you completely avoid search (and you need to do the Microsoft account thing for it AFAIK). Windows has needed a system to update 3rd-party software, to compete with Linux package managers, and the store is a really good effort (there are still annoying warts that Aur, Deb, RPM do not have). If you're willing to be a bit dumb, there is convenience.
- moonchrome 5y ago
  This is exactly why I don't do that initially - I don't mind my account being linked - but I've been bitten by the home path bugs multiple times, I unplug my pc during setup

spicybright 5y ago

So frustrating how this still happens. It's too latin centric.

bagswatchesus 5y ago

Not sure how they managed to do it but they had some basic rules that they used to say "no real name can look like this, this is a fake person!" and just kicked it out. https://www.thelvbags.co/louis-vuitton-wallets-and-purses.html https://www.thelvbags.co/louis-vuitton-wallets-and-purses.ht...

tazjin 5y ago

The amount of random encoding problems that still exist are so bizarre. I recently left a UK job after already leaving the country more than a year ago, and in their attempt to mail P45 form to my new address (in Moscow) the only bits that survived are the string "c/o" and the postal code.

m_kos 5y ago

Isn't it bizarre that we have self-driving cars, the ISS, and phones with 50 megapixel cameras but still struggle with character encoding?

zakius 5y ago
for self driving cars, ISS and digital cameras everything you do is blurry in a sense, "good enough" approximation is actually good enough while character encoding and transformations have to be done perfectly and precisely and have surprisingly big number of edge cases
tetha 5y ago
Character encoding is in a special class of problems. Like time handling. If you pick up a halfway non-ancient framework in a somewhat common language with a somewhat non-terrible persistence like postgres, you just don't have problems. Just don't care, and it just works. But it's super easy to derail that fragile correctness with something like MySQLs utf8-ish handling, or some OS's path handling, or 'efficiency', or a user or frontend dev submitting data in a wrong encoding. And then it gets mangled. And then the user is unhappy. At that point, it becomes very hard to argue why one of the two things is wrong, and the other is not. While the user argues the other way around. Because both look correct, if you look from the right angle. And the only reason why I am right is because of some standard, while the customer is right because of money. And yes, it is very 'surprising' why our software now functions correctly for russian or greek customers.
- ctdonath 5y ago
  That it's a special class of problems doesn't mean it shouldn't be solved by now. Time handling should be solved too; amazing that an iOS app can't get current correct GMT.
quadrifoliate 5y ago
It's not bizarre at all. Character encodings are a sort of language in themselves, and end up with all the problems that regular old languages have – there's a lot of variety, people can't agree on one particular solution, and there's not a lot of money in taking care of the edge cases. It would be bizarre if we were at the point where we had perfect translations for everything, but still struggled with character encodings specifically.

amarshall 5y ago

For a list of strings that often cause problems to, e.g., add to a test suite, see https://github.com/minimaxir/big-list-of-naughty-strings https://github.com/minimaxir/big-list-of-naughty-strings

OskarS 5y ago
An enormously useful list, I’ve used it several times, and it can often dig up some real nastiness if you haven’t been super careful. This entry, by the way, is a fantastic little easter egg in the list: https://github.com/minimaxir/big-list-of-naughty-strings/blob/db33ec7b1d5d9616a88c76394b7d0897bd0b97eb/blns.txt#L712 https://github.com/minimaxir/big-list-of-naughty-strings/blo...
- vertis 5y ago
  No, seriously, wake up
  - deleted 5y ago
    [deleted]
  - jamiek88 5y ago
    That was creepy!
tomaslaureano 5y ago
Great resource! I usually use pangrams (holoalphabetic sentences like "The quick brown fox jumps over the lazy dog") to ensure that my code can handle all the alphabet characters for the languages that should be supported at the very minimum.
ryanianian 5y ago
Very handy. My previous simple test-case was simply a selection from this well-known text-file which is simply a collection of somewhat uncommon unicode characters, usually used for rendering tests. https://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-demo.txt https://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-demo.txt But this set of strings is specifically designed to cause edge-case errors. Also don't forget Spolsky's seminal "The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)". https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...

mkotowski 5y ago

I, too, have the Ł letter in my name, and yes, it is a sick joke that so many things even in a supposedly modern systems make an assumption that the world runs on ASCII. In the case of the Windows operating system, the worst fact is that every single part of it behaves differently. Some parts display the path with a wrong encoding, but handle it correctly. A third-party app can display it correctly, but fails while trying to access any file. From what I remember, even the built-in PATH variable editor/manager goes through some arcane steps to display the letters in a wrong way, but getting them to work sometimes. I can only imagine how much more pain it is for someone using any of the less widely-used writing systems or those with more advanced features compared to ASCII (Hebrew’s RTL, Arabic scripts mid- and final forms, etcetera).

gerdesj 5y ago
Can Ł have an alternative representation? For example the German ß => ss. Also I think ö can be written as oe. In English we simply shake the big bag of letters, pick a few at random and then throw them at the page until a few stick.
- q3k 5y ago
  > Can Ł have an alternative representation? Nope. Neither can ź, ć, ś, ą or ę. You can, and people do write them as z, c, s, a and e when writing in a restriced character set, but that is not 'correct' and is not a bijection, ie. „półka” and „polka” mean two different things. There's also the case of technically-same-sounding-especially-recently ż/rz and ó/u (whose replacement would let you get rid of two 'non standard' characters), but for historical reasons these are not interchangeable.
  - gerdesj 5y ago
    I do find this sort of stuff fascinating and also faintly frustrating but of course my mother tongue is (in)famous for being a bit loose at first sight. According to one of my employees (Polish) Ł sounds roughly like w as in win or water but not as in what. A quick read of this: https://en.wikipedia.org/wiki/%C5%81 https://en.wikipedia.org/wiki/%C5%81 doesn't help too much. Does enforcing Ł instead of say w cause your written language to fail in some way? I don't want to cause offense, I want to understand the causes of difference.

rcxdude 5y ago

Sadly there is even still software which fails to build or even fails to run when there is a space in a filename (as is super common on windows file paths, as well as autogenerated CI build folders). It's ridiculous to no end that software cannot handle paths correctly.

tediousdemise 5y ago

The solution to this is extremely simple: don't validate usernames, period. The rationale is from an article someone linked here ("Falsehoods Programmer's Believe About Names"): > Anything someone tells you is their name is—by definition—an appropriate identifier for them. If you try to validate by checking for profanity, knowing full well that people can have names that contain profane substrings, I have a tongue-in-check message for you—you are a fucking asshole.

anotheraccount9 5y ago

When ł visiłed his page, my browser crashed.

lukaszkups 5y ago

Ah it's so common for me that I've totally abandoned using Latin letters in my first/last name long time ago (and I recommend the same for you ;))

tyteen4a03 5y ago

Fun fact: If you have the exclamation mark (!) in your Windows username, Java will think it's the jar separator and `getResourceAsStream` will refuse to work. This broke many people's Minecraft installation over the years. The bug in question [0] was reported in 2001 and remains unsolved 20 years later. [0] https://bugs.java.com/bugdatabase/view_bug.do?bug_id=4523159 https://bugs.java.com/bugdatabase/view_bug.do?bug_id=4523159

alisonkisk 5y ago
The only practical fix would be to ban ! in usernames.
kspacewalk2 5y ago
I find the very idea of putting an exclamation mark on one's username and not expecting eventual problems to be quite curious.
- edaemon 5y ago
  It can be used to indicate a click sound in some languages and the IPA. Probably not the most common reason to come across this bug but it's a case worth considering. https://en.wikipedia.org/wiki/Exclamation_mark#Phonetics https://en.wikipedia.org/wiki/Exclamation_mark#Phonetics
- jmopp 5y ago
  The Nama language has a letter that is often replaced with an exclamation mark in common typography. The lead actor in the film The God's Must be Crazy was named N!xau ǂToma
- vincnetas 5y ago
  i recommend reading this "Falsehoods Programmers Believe About Names". world is crazy place if you expect some system in how people name each other and are writing it down. https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/ https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...
- 5y ago

jonnycomputer 5y ago

>When I found out that the bug was in the Rider itself, I reported it to technical support. I also found a similar report for PyCharm. Unfortunately, things haven’t moved forward since then. Unfortunately, typical with Jetbrains.

trinovantes 5y ago

In CS, most algorithms assume an ASCII character set. I wonder if there's any string-related algorithms that completely break (functionally or complexity wise) when given UTF-16 or UTF-8 character sets

deathanatos 5y ago
> In CS, most algorithms assume an ASCII character set. They most certainly do not. E.g., a Turing machine assumes an alphabet Γ which is a set of some characters and is defined no further, as any exact definition is meaningless to the theory. (I.e., the algorithm is generic over any alphabet.) The alphabet need not even be text; e.g., for a Turing machine, the set of all octets suffices. Even for something like Levenshtein distance, the only real requirement of the algorithm is that the abstract "characters" implement equality testing. For Unicode text, I'd start with graphemes, and then look for counter examples.
- trinovantes 5y ago
  I guess they won't break correctness but I do remember many algorithms (e.g. tries) assume you have constant random access to characters which AFAIK is not possible in UTF-8
a1369209993 5y ago
Asymtotic complexity can't change based on the character set, since you can just reuse the same algorithm with larger opaque datums. (Exception being algorithms with O(n^8) or O(n^256) complexity, but noone uses those anyway.) A variable width encoding can cause issues in principle, but useful algorithms already have to deal with strings that have variable-length physical represention anyway (eg "yes" vs "no"), so it tends not to be a problem in practice.

askvictor 5y ago

Somewhat surprising that this is an issue with JetBrains, given that they are based in Eastern Europe, and would probably have more direct experience of these sorts of problems than US or UK based companies. OTOH maybe it's just a scale thing - bigger companies have more resources to handle these sort of cases, regardless where they're based (not that they always do...)

PrivateButts 5y ago

Similar to this, Node and NPM get very temperamental when you have a User folder with a space in it. I gave up on the community workarounds and just created a new account and copied my files over to fix it.

godmode2019 5y ago

I have a set of names I give to different providers. Advertisers always assume a name is constant and email addresses can change. I got a name saying 'Hi John I just want to xyz' I can skip this email as they used a fake name. Works better than other methods I have found.

auggierose 5y ago

Haha, that's why something like Cosmopolitan Identifiers would be a good idea: https://doi.org/10.47757/obua.cosmo-id.3 https://doi.org/10.47757/obua.cosmo-id.3

souptonuts 5y ago

Idk changing your stupid fucking name could be a fix too

shantnutiwari 5y ago

the bug was fixed one hour ago-- looks like HN customer service worked again

supernes 5y ago

It's somewhat common to see videogames issue a patch shortly after release where they fix crashes due to non-ASCII Windows usernames or non-English locales. I'm not sure what the root cause of the confusion is, other than text strings being hard in general.

jerf 5y ago
It's easy to think the answer is "just UTF-8 everything" but unfortunately the long and twisty history of filesystems means that's not the correct answer, and the "correct answer" is really hard to write down quickly. If you never display the filename, the answer is to treat existing filenames as bags of bytes, but that breaks down as soon as you need to display them, or if you need to manipulate them by appending unicode to them, in which case you have to decide on an encoding. Unicode encodings tend to mangle non-Unicode values because they're specified to replace whatever they can't understand with a particular Unicode character, usually represented as a diamond with an inverted ? inside of it. There's some obscure solutions to this problem, like https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ (which includes discussion of the 16 bit encodings you need for Windows), but I haven't seen broad support for them. You need a deliberately "noncompliant" encoding/decoding system that doesn't replace unknown characters with replacement characters. Fortunately, compliant systems are becoming more and more popular and available. Unfortunately, that can make file name handling harder than when you had a non-Unicode-compliant handling system for your strings.
- nyanpasu64 5y ago
  Rust uses WTF-8 on Windows for OsStr[ing] and Path[Buf]. It's zero-overhead to cast from &str to &OsStr/&Path to &[u8] (though converting WTF-8 to UTF-16 costs an extra operation when performing a Win32 function call). However this doesn't solve the inability to round-trip "possibly-valid UTF-8/16" to "Unicode text" and back (though Python's surrogateescape might be one viable approach). Other libraries handle this even worse than Rust. On Linux (filenames are bytes), Qt is unable to open files with invalid UTF-8 names, while GTK can open them (but shows an "invalid encoding" message instead of the original filename), which I think is a good-enough approach.

simonblack 5y ago

Isn't this one of those "100 things Programmers don't know about People's Names" things? Like the poor, it will be with us always.

xdfgh1112 5y ago
I don't know, it's just a Unicode character? Not even a newer one, it's just 2 utf8 bytes. Pretty much everything should support that in 2021. When I think of 100 things I think of stuff like "some people spell their name in all lowercase and get really funny if you change it"
- numpad0 5y ago
  Yeah so double byte characters costs extra. I don’t know, a checkbox or something default off. Always did still does. Double width costs even more.
  - horsawlarway 5y ago
    you're getting downvoted, but between tchar hiding wchar vs char... this literally could be someone toggling off the "UNICODE" checkbox in visual studio somewhere.
- hprotagonist 5y ago
  windows probably defaults to latin-1
  - bryanrasmussen 5y ago
    the default windows encoding is UTF-16, a long time ago it was Windows-1252 https://en.wikipedia.org/wiki/Windows-1252 https://en.wikipedia.org/wiki/Windows-1252
    - hprotagonist 5y ago
      or CP-1251, in some locations.

jasonpeacock 5y ago

And yet it's one of the simplest things to add non-ASCII chars to your tests to validate their handling. It's like not testing if your calculate application can handle negative numbers or decimals.

nradov 5y ago
In fact it's trivial to generate a text file of all valid Unicode code points and use that as input to unit tests.
- yakubin 5y ago
  It may be faster to generate them on the fly. Iterating over ranges of integers is a lot faster than reading files from disk.
- Someone 5y ago
  I would have to do research on whether the list of valid code points depends on the Unicode version. For example, can regional indicator code points (https://en.wikipedia.org/wiki/Regional_indicator_symbol https://en.wikipedia.org/wiki/Regional_indicator_symbol) appear in isolation? If not, is that different in Unicode < 6, where those code points weren’t assigned yet? Similarly, what about tags (https://en.wikipedia.org/wiki/Tags_(Unicode_block) https://en.wikipedia.org/wiki/Tags_(Unicode_block) )? Do these require an U+E007F CANCEL TAG? The 66 noncharacters certainly need consideration. http://www.unicode.org/faq/private_use.html http://www.unicode.org/faq/private_use.html says: “Because of this complicated history and confusing changes of wording in the standard over the years regarding what are now known as noncharacters, there is still considerable disagreement about their use and whether they should be considered "illegal" or "invalid" in various contexts” Edit: also, testing all code points likely is overkill and using code points in isolation likely isn’t enough. Most tests are better of with something like the big list of naughty strings (https://github.com/minimaxir/big-list-of-naughty-strings https://github.com/minimaxir/big-list-of-naughty-strings)

umvi 5y ago

Using non-ascii characters in file paths, toolchain config files, and other non-display contexts is just asking for trouble, even if it is your name...

fluxem 5y ago
Also spaces. I spent half an hour debugging why cmake cuda build was failing.
- munk-a 5y ago
  A lack of support for spaces at this point is unacceptable. I, personally, despise spaces in paths but on windows a whole bunch of default system paths already have spaces embedded in them in major ways... and let's not forget parens as well - thanks "Program Files (x86)"
b112 5y ago
This wouldn't have happened if using rust!
- nightfly 5y ago
  Can you knock it off??? This is even more annoying that out-of-place rust evangelism
  - GoblinSlayer 5y ago
    You know he's right. Look at all the rust converts preaching their dogmas here.
- burnished 5y ago
  Some of the other attempts are a little subtle, this one is a pretty blatant attempt to rile up the folks that are already angry about rust for whatever reason. Please stop.
jasonpeacock 5y ago

numpad0 5y ago

Oh, it’s not a common knowledge that you should not UTF-8 in Windows username? That had been the case since 95 days. Only recently it had supposedly improved after Microsoft Account login become semi mandatory.

progval 5y ago
On the contrary, the first bug happens because docker-compose tries to decode the path as UTF-8, but it is not UTF-8-encoded. ("'utf-8' codec can't decode byte")
chris_overseas 5y ago
I don't think this bug is anything to do with Windows, rather it is due to the way the paths are handled in the IDE's codebase. Presumably the same problem exists when using these IDEs in conjunction with a path containing non-ascii characters in the Linux or macOS world.
- numpad0 5y ago
  Isn't it some compilation option issue in native part? I thought it's a line on .sln or include library in a C++ source or something that has to be explicitly specified when building a Win32 binary.
  - GoblinSlayer 5y ago
    InteliJ has native part?
- account42 5y ago
  > Presumably the same problem exists when using these IDEs in conjunction with a path containing non-ascii characters in the Linux or macOS world. Why would you presume that when the problem seems to be that one tool uses the systems native 8-bit encoding while another tool expects UTF-8 - under sane systems these are the same.
Fordec 5y ago
A lot of adults today weren't even alive in 95. Also, the assumption that people are familiar with windows vs other operating systems is becoming less and less valid. And as the world gets more globalised and remote, it's no longer to be assumed that all technical people are of a Anglo American culture.

sschueller 5y ago

Many years ago I could not access the apple developer panel because of the umlaut in my last name. It was eventually fixed but I was quite surprised that such a large company would run into such a basic issue.

deleted 5y ago
[deleted]
devrand 5y ago
My last name has an apostrophe in it which Apple apparently loves to embed directly into their JavaScript unescaped. For a long time neither I nor Apple could look up AppleCare status on my stuff as they were all linked to my Apple ID. The portal would thus require me to login, but then would just show a partially rendered page as my last name was causing an JS syntax error.
- doubled112 5y ago
  You'd think the apostrophe would be common enough they'd know it could happen, but no. I love to enter it and see what each vendor and website's backend does with it. The Staples Canada website, for example, returns it as ' (HTML escaped) A couple times I've logged in, it seems to escape a new character. I'm currently up to &amp;#39;
  - devrand 5y ago
    Haha yeah I'm fairly used to seeing HTML escaping in my name. The weirdest case I've had with that is the Six Flags mobile app. To add a season pass you need to provide your card number and last name. For the life of me I couldn't get it to validate, but I saw they showed the HTML escaped version in their e-mails to me. Turns out I had to type out "'" into their input box for my last name as that's apparently what they put in their database.
- nneonneo 5y ago
  Hmm, it sure sounds like John <script>alert(1);</script>Doe (Bobby Tables' distant cousin) should sign up for an Apple account. An XSS attack which could target the AppleCare reps' machines could be catastrophically bad...

pledess 5y ago

The article offers a solution of idea.system.path=${root.dir}/JetBrains/Rider/system but doesn't mention the C:\JetBrains directory permissions. Directory permissions under %LOCALAPPDATA% (the location that works for people without a Polish character) should restrict write access to one user. With the Windows default behavior, creating C:\JetBrains would inherit permissions from C:\ - and wouldn't restrict write access to one user. Maybe 99% of the time this is irrelevant (i.e., there's no realistic threat from malicious actors who control unprivileged user accounts on your own development machine). Still, it's a potential downside of the solution, and more motivation for the vendor to fix their code so that Polish characters can be used under %LOCALAPPDATA%.

Kwpolska 5y ago
If you are on a multi-user system, the path "C:\JetBrains" isn’t really ideal (what if other users also need Rider and have non-ASCII usernames?). That said, you can easily change file permissions on Windows if the default ones don’t work for you.

mrweasel 5y ago

It’s a pretty good test case. Similarly we found a number of bugs in a Django application and path handling, because I happend to be using Windows for six months, while the rest of the team was on Linux and Mac.

xlii 5y ago

Very similar problem to one described started my exodus from Google services. I also have non-latin characters in my name however I knew it was always an issue so I never used it in paths etc. At some point, long time ago, I was tasked to do some maintance with Google Cloud service (can't remember the name of the service now) which was doable only through Python CLI utility and it failed with very similar Python error. What I found out rather quickly is that utility took my name from Google+ profile, which did include those non-latin characters. No biggie - I thought and fired e-mail to support (yeah it was those times it was still that easy). Few hours passed and I received information that this won't be fixed anytime soon and the best course of action would be to change my name. Of course, support person probably meant to remove the diacriticals from my Google+ profiles, but still it left unplesant aftertaste for years to come.

perl4ever 5y ago
A Polish relative of mine used to just gave an arbitrary substitute name (e.g. "Dave Smith") for restaurant reservations, because even if they could write his last name, they wouldn't be able to pronounce it. My sibling has a name that has an accent, and just enters it with the plain letter most of the time. The name was once rare and "ethnic", but became popular a generation later so people know how to pronounce it regardless. Our parents gave us two middle names, wanting to preserve our grandmothers' surnames, but also in the spirit of "Bobby Tables", having ambivalent feelings about the computerization of society tending towards inflexibility. According to: https://en.wikipedia.org/wiki/Naming_customs_of_Hispanic_America https://en.wikipedia.org/wiki/Naming_customs_of_Hispanic_Ame... ...misunderstanding of naming customs in the US has actually led to significant consequences due to last names not matching on legal documents. I remember reading a story about how there are people in China whose name incorporates a character that is obscure enough, the authorities are trying to eliminate it and get them to change their name. If I recall correctly, Chinese has a particular problem with characters that are part of names that have been around forever, but are no longer used for ordinary writing.
- em-bee 5y ago
  a friend of mine has that problem. his company is owned by his wife because his name can't appear in legal documents and he refuses to change it. his approach was that he petitioned unicode to include that character. i don't think the authorities are actively trying to eliminate those characters but simply don't want to go through the effort to track down and have those characters added to the standard. the process takes years and in the mean time you have to live with the inconvenience. also most of the people faced with the problem would not even know how.

ygra 5y ago

One way of working arrive such issues is to use subst. That way the application thinks your project directory is actually located on P:\ or something like that.

Dannymetconan 5y ago

I can very much relate to this but also have very little sympathy here. I have a special character in my name, an apostrophe, and it causes trouble regularly online and with tooling. A number of years ago I decided just to never use it when it came to anything to do with technical work be it email, logins or usernames. Unicode characters are a pain to deal with and I have suffered from it first hand trying to handle it. At the end of the day it is much easier just to not use the special characters and move on with your life rather then be battling the constant frustration. I'm sure these tools have lots of issues opening and you would be surprised at the amount of time, effort and testing it would be required to provide fully Unicode support. Most people would see it as a very small positive and not worth the effort. I find it hard to disagree.

jltsiren 5y ago
My legal last name is "Sirén". When I was younger, I almost always used "Siren", because it was easier to type. Then, ~15 years ago, I started noticing that American websites sometimes rejected it, because they considered it inappropriate. Sometimes "Sirén" would work, sometimes it worked but caused minor annoyances, and sometimes it would not work for technical reasons. Both versions work most of the time these days, but I still run into trouble once in a while no matter which name I use.
- Dannymetconan 5y ago
  Totally agree with the sentiment. It has gotten a lot better in the last 10 years. Very frustrating to have your name blacklisted by that. It does seem most system have a very US focused design. I still find it funny that even in my home country you can't use a lot of local special characters in names. Also most airlines won't accept it so technically I'm not giving them my true name!
- 10000truths 5y ago
  Why would Siren be an inappropriate name?
  - 5y ago

david422 5y ago

There's also this article: falsehoods-programmers-believe-about-names: https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/ https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-... Certainly informative if you haven't seen it before. My takeaway from it was that design your system to try to accommodate as much as possible, but it would basically be impossible to accommodate them all, so aim for your target audience.

darkhorn 5y ago

I think it is a Java related issue. Relevant issue occurs in Jaspersoft Report. You cannot install Jaspersoft Report on Turkish Windows no matter what.

SergeAx 5y ago

When I first installed Windows 7 like ten years ago, I entered my Russian name in Cyrillic. When I saw that the system created a directory with exactly that name under `C:\Users\` I immediately scanned the internet for a way to rename it and done just that. I don't want to know how much mess like that in a story I thus had successfully escaped. NB: the method is still the same, it's a second (not accepted) answer here: https://superuser.com/questions/890812/how-to-rename-the-user-folder-in-windows-10 https://superuser.com/questions/890812/how-to-rename-the-use... (about ProfileImagePath registry value).

vertis 5y ago
This is sad though. You shouldn't have to change who you are for a computer program.
- GoblinSlayer 5y ago
  Is "vertis" who you are? There's more to a human, than a name.
- SergeAx 5y ago
  I have this lower ASCII handle since about 1990, I beleive. That was the time when you just can't do literally anything without one.

ddeyar 5y ago

Some years ago I used the + feature in my gmail address. e.g. myname+ycombinator@gmail.com to track down which service is giving away my email address. It happened more than once that I could not log in anymore at some point because they started to disallow the + character in email addresses. I also got phone calls from some companies complaining that i misspelled my email address because there was their company name in it.

cgufus 5y ago
hehe, did the same, although not with +, but using a catch-all feature of the provider. I still get a lot of spam and phishing attempts on my „dropbox@<mydomain>“ address. I faintly remember they (dropbox) had a breach some time in the past.

asimjalis 5y ago

This is like Kafka’s story in which the protagonist wakes up to find out he’s a (software) bug.

Ansil849 5y ago

Sometimes even "regular" ASCII surnames cause problems. When written in the Latin alphabet, my surname is one letter. I've had an amazing amount of problems with this not just due to technical limitations (like various forms marking the entry as invalid), but--much more aggravatingly--human limitations. One particularly infuriating anecdote: at a past job many years ago, the email structure was lastname@company.com. I dutifully sent the IT person in charge of creating emails my desired email. The IT person wrote back an amazingly condescending email that as per the policy, emails had to be last names, not individual letters. I then had to go find a bunch of random websites which explained single-letter names and forwarded them to the IT person. They then obliged, but did not apologize for insulting me. That is not right that I had to put up with that.

account42 5y ago
> One particularly infuriating anecdote: at a past job many years ago, the email structure was lastname@company.com. I dutifully sent the IT person in charge of creating emails my desired email. The IT person wrote back an amazingly condescending email that as per the policy, emails had to be last names, not individual letters. I then had to go find a bunch of random websites which explained single-letter names and forwarded them to the IT person. They then obliged, but did not apologize for insulting me. That is not right that I had to put up with that. Except single letter last names are less common than people not following policy and/or abbriviating the name. It could simply be an honest mistake and the email is just their standard response since they have other things to get to. Did you try simply pointing out that that the letter was in fact your last name instead of getting passive-agressive?

deleted 5y ago

[deleted]

Natfan 5y ago

I've also had issues putting in my full name as my username. Lots of programs do not expect spaces in the path, and I experience a lot of errors which are resolved by changing the path to not contain a space. [1]: https://github.com/microsoft/WSL/issues/2577#issuecomment-901815008 https://github.com/microsoft/WSL/issues/2577#issuecomment-90...

itsrajju 5y ago

As of 2 hours before me writing this comment, JetBrains claims to have fixed the underlying issue [0]. Maybe they saw this post? :D [0]: https://youtrack.jetbrains.com/issue/IDEA-264563 https://youtrack.jetbrains.com/issue/IDEA-264563

deepsun 5y ago

> My username contains a "ł" character and because of it, this file cannot be processed properly. What is so curious there? Some names contain all non-latin characters, and some softwares don't work with non-ASCII symbols. I just cannot understand why is it interesting.

Svoka 5y ago

Did you know that Android still won't build on Windows if you have Cyrillic letter in user name?