5 ms·
Because human language is hard to boil down to a simple computing model and the problem is underdefined, based on naive assumptions. Or perhaps I should say na
by BoringTimesGang 2y ago
Because human language is hard to boil down to a simple computing model and the problem is underdefined, based on naive assumptions.
Or perhaps I should say naïve.
- cm2187 2y agoWell pretty much every other more recent language solved that problem.
- kccqzy 2y agoAlmost no programming language, perhaps other than Swift, solved that problem. Just use the article's examples as test cases. It's just as wrong as the C++ version in the article, except it's wrong with nicer syntax.
- zahlman 2y agoPython's strings have uppercase, lowercase and case-folding methods that don't choke on this. They don't use UTF-16 internally (they can use UCS-2 for strings whose code points will fit in that range; while a string might store code points from the surrogate-pair range, they're never interpreted as surrogate pairs, but instead as an error encoding so that e.g. invalid UTF-8 can be round-tripped) so they're never worried about surrogate pairs, and it knows a few things about localized text casing: >>> 'ß'.upper() 'SS' >>> 'ß'.lower() 'ß' >>> 'ß'.casefold() 'ss' There are a lot of really complicated tasks for Unicode strings. String casing isn't really one of them. (No, Python can't turn 'SS' back into 'ß'. But doing that requires metadata about language that a string simply doesn't represent.)
- kccqzy 2y agoStill breaks on, for example, Turkish i vs İ. It's impossible to do correctly without language information. > (No, Python can't turn 'SS' back into 'ß'. But doing that requires metadata about language that a string simply doesn't represent.) Yes that's my point. Because in typical languages strings don't store language metadata, this is impossible to do correctly in general.
- zahlman 2y agoI'm not seeing anything in the Swift documentation about strings carrying language metadata, either, though?
- kccqzy 2y agoThis lowercase function takes a locale argument https://developer.apple.com/documentation/foundation/nsstring/1417298-lowercased https://developer.apple.com/documentation/foundation/nsstrin... It looks like an old NSString method that's available in both Obj-C and Swift. The casefold function is even older than that. https://developer.apple.com/documentation/foundation/nsstring/1413779-folding https://developer.apple.com/documentation/foundation/nsstrin... Its documentation specifically includes a discussion of the Turkish İ/I issue.
- tedunangst 2y agoBut that's wrong. The upper case for ß is ẞ.
- cm2187 2y agoC#'s "ToUpper" takes an optional CultureInfo argument if you want to play around with how to treat different languages. Again, solved problem decades ago.
- account42 2y agoThis is not a locale issue, it's a Unicode version issue. Which hightlights another problem with adding this to the base standard library.
- IncreasePosts 2y agoThat was only adopted in Germany like 7 years ago!
- kccqzy 2y agoWell languages and conventions change. The € sign was added not that long ago and it was somewhat painful. The Chinese language uses a single character to refer to chemical elements so when IUPAC names new elements they will invent new characters. Etc.
- extraduder_ire 2y agoDoes unicode have space set aside for those new symbols to slot into? I know it's very rare, but it could get messy.
- account42 2y agoUnicode is already messy. Chinese characters especially so due to han unificiation.
- 2y ago
- crote 2y agoBut that's wrong. The uppercase for "in Maßen" ("in moderate amounts") is not "IN MASSEN" ("in Massen", meaning "in massive amounts").
- tialaramex 2y agoRust will cheerfully: assert_eq!("ὀδυσσεύς", "ὈΔΥΣΣΕΎΣ".to_lowercase()); [Notice that this is in fact entirely impossible with the naive strategy since Greek cares about position of symbols] Some of the latter examples aren't cases where a programming language or library should just "do the right thing" but cases of ambiguity where you need locale information to decide what's appropriate, which isn't "just as wrong as the C++ version" it's a whole other problem. It isn't wrong to capitalise A-acute as a capital A-acute, it's just not always appropriate depending on the locale.
- account42 2y agoIs this assert_eq!("\u1F41δυσσεύς", "ὈΔΥΣΣΕΎΣ".to_lowercase()); or assert_eq!("\u03BF\u0314δυσσεύς", "ὈΔΥΣΣΕΎΣ".to_lowercase()); For display it doesn't matter but most other applications really want some kind of normalizatin which does much much more so having a convenient to_lowercase() doesn't buy you as much as you think and can be actively misleading.
- MBCook 2y agoSo what? That doesn’t prevent adding a new function that converts an entire string to upper or lowercase in a Unicode aware way. What would be wrong with adding new correct functions to the standard library to make this easy? There are already namespaces in C++ so you don’t even have to worry about collisions. That’s the problem I see. It’s fine if you have a history of stuff that’s not that great in hindsight. But what’s wrong with having a better standard library going forward? It’s not like this is an esoteric thing.
- wakawaka28 2y agoThe reason that wasn't done is because Unicode is not really in older C++ standards. I think it may have been added to C++23 but I am not familiar with that. There are many partial solutions in older C++ but if you want to do it well then you need to get a library for it from somewhere, or else (possibly) wait for a new standard. Unicode and character encodings are pretty esoteric. So are fonts. The stuff is technically everywhere and fundamental, but there are many encodings, technical details, etc. And most programmers only care about one language, or else only use UTF-8 with the most basic chars (the ones that agree with ASCII). That isn't terrible. You only need what you actually need. Most programs don't strictly have to be built for multiple random languages, and there is kind of a standard methodology to learn before you can do that.