6 ms·
There’s some interesting information in there. Unfortunately the person or LLM writing this got pretty confused right in the introduction already. > “Suppose [
by lukasgelbmann 1mo ago
There’s some interesting information in there. Unfortunately the person or LLM writing this got pretty confused right in the introduction already.
> “Suppose […] they type straße and you’ve stored STRASSE. To make these count as matches, you need […]”
Really bad example, because as the article says later on, this casefold crate won’t match those two strings because the ß → ss conversion isn’t done.
> “[str::to_lowercase and case folding] diverge on real characters—ß, İ, final sigma”
The main point is true (case folding is different from lowercasing), but two of the three examples are wrong. The casefold operation that they use maps ß to itself, as does str::to_lowercase. The casefold operation maps İ to U+0069 U+0307 regardless of locale, as does str::to_lowercase.
When I’m reading an article, these kind of mistakes in the introduction make me doubt the accuracy of the whole article. Which is a shame, because again, it’s an interesting write-up. The mistakes also make the article harder to follow, since the examples imply ß is folded to ss.
- lukasgelbmann 1mo agoToo late to edit, but the situation with İ is more complicated, and I got the mapping wrong for this specific casefold implementation. What I should have said is that in the case of İ/i/I/ı, using str::to_lowercase for string matching wouldn’t be any less correct than using their locale-independent casefold. The third character in the list, final sigma, is a good example that illustrates why using str::to_lowercase for string matching isn’t good.
- wongarsu 1mo agoIf anything, the word STRASSE is a great example why case-folding followed by string comparison is not the same as a case-independent string comparison Or if you really want to do it, your case-folding needs to map certain lowercase characters to to other lowercase characters (lowercase ß to ss, lowercase ı to i, etc), losing some of the meaning
- 7bit 1mo agoIn some cases your're also lost ng all of the meaning. For example: aß/Ass, Maß/Mass, Buße/Busse, Floß/floss
- mananaysiempre 1mo ago> If anything, the word STRASSE is a great example why case-folding followed by string comparison is not the same as a case-independent string comparison s/case-folding/lowercasing/ Proper Unicode case-folding absolutely does map ß to ss, ς to σ, etc. Moreover, there are some scripts (IIRC Georgian) where for historical reasons case-folding yields uppercase letters, not lowercase ones. The case-folding mapping is specifically designed in concert with the comparison rules to yield the same result, that’s why it’s a separate operation from lowercasing. (I believe the thing described in TFA is supposed to be proper case-folding in that sense, but given TFA is AI-written I wouldn’t trust its descriptions either way.) That said, if you want to match the sort order customary in a specific language, you need to use language-specific rules for producing collation keys rather than generic case-folding. There’s no way out of this because different users of e.g. the Latin alphabet want contradictory results. And if you think you do want generic casefolding, then you probably actually want NFKC_Casefold instead unless your input is pre-normalized.