4 ms·

> The fix was to create new exceptions so that str.lower() would behave as if it was using Unicode 3.2.0 for only particular function. So, we go through each Un

by ummonk 22d ago

> The fix was to create new exceptions so that str.lower() would behave as if it was using Unicode 3.2.0 for only particular function. So, we go through each Unicode codepoint and record when the behavior of str.lower() is different when comparing the Unicode version shipped with Python and Unicode 3.2.0 This sounds like a really hacky solution compared to implementing a separate frozen Unicode 3.2.0 lower.

jwilk 22d ago
The first sentence sounds as if they modified the implementemention of str.lower(). That would be bonkers, but that's not what they did. https://github.com/python/cpython/commit/7e109d084d55e7eb https://github.com/python/cpython/commit/7e109d084d55e7eb The important part is: # B.3 is mostly Python's .lower, except for a number # of special cases, e.g. considering canonical forms. +# To enforce Unicode 3.2.0 behavior of .lower instead of +# whatever Unicode version is included with Python we +# add unassigned or newly case-folding codepoints to +# the exception map, too. b3_exceptions = {} for k,v in table_b2.items(): if list(map(ord, chr(k).lower())) != v: b3_exceptions[k] = "".join(map(chr,v)) +for cp in range(0x110000): + ch = chr(cp) + # Assigned in current Unicode version + # and supports case folding, but not + # explicitly in B.2 or B.3 tables. + if (unicodedata_current.category(ch) != "Cn" + and ch.lower() != ch + and cp not in table_b2 + and cp not in table_b3): + b3_exceptions[cp] = ch # Identity.
- quietbritishjim 22d ago
  That fragment doesn't mean much in isolation. You've just said that they didn't modify str.lower (because "that would be bonkers") but you've posted a fragment which, for all we know, is part of the str.lower implementation.
philipwhiuk 22d ago
To be clear (because the snippet is non-explanatory). For encode("idna") what they did is use lower() except where it would produce a result different to 3.2.0 and then instead use the result from 3.2.0 instead. Essentially they've frozen the IDNA encoding to be based on 3.2.0 by overriding any changes.
- ryukoposting 22d ago
  Yeah. I can understand the confusion, though. The title claims the issue was in lower(). Though the problem was actually in encode('idna')'s usage of lower(). The article would probably get far fewer clicks if it were named "when encode('idna') is a security vulnerability"
- ummonk 22d ago
  Yeah, that's what I figured, but my worry upon seeing this is "what happens if lower() changes again and people forget to update the list of exceptions?". Unless they have unit testing on the entire Unicode code space to ensure what they're doing is always identical to 3.2.0.

by ummonk 22d ago

jwilk 22d ago

The first sentence sounds as if they modified the implementemention of str.lower(). That would be bonkers, but that's not what they did. https://github.com/python/cpython/commit/7e109d084d55e7eb https://github.com/python/cpython/commit/7e109d084d55e7eb The important part is: # B.3 is mostly Python's .lower, except for a number # of special cases, e.g. considering canonical forms. +# To enforce Unicode 3.2.0 behavior of .lower instead of +# whatever Unicode version is included with Python we +# add unassigned or newly case-folding codepoints to +# the exception map, too. b3_exceptions = {} for k,v in table_b2.items(): if list(map(ord, chr(k).lower())) != v: b3_exceptions[k] = "".join(map(chr,v)) +for cp in range(0x110000): + ch = chr(cp) + # Assigned in current Unicode version + # and supports case folding, but not + # explicitly in B.2 or B.3 tables. + if (unicodedata_current.category(ch) != "Cn" + and ch.lower() != ch + and cp not in table_b2 + and cp not in table_b3): + b3_exceptions[cp] = ch # Identity.

quietbritishjim 22d ago
That fragment doesn't mean much in isolation. You've just said that they didn't modify str.lower (because "that would be bonkers") but you've posted a fragment which, for all we know, is part of the str.lower implementation.

philipwhiuk 22d ago

To be clear (because the snippet is non-explanatory). For encode("idna") what they did is use lower() except where it would produce a result different to 3.2.0 and then instead use the result from 3.2.0 instead. Essentially they've frozen the IDNA encoding to be based on 3.2.0 by overriding any changes.

ryukoposting 22d ago
Yeah. I can understand the confusion, though. The title claims the issue was in lower(). Though the problem was actually in encode('idna')'s usage of lower(). The article would probably get far fewer clicks if it were named "when encode('idna') is a security vulnerability"
ummonk 22d ago
Yeah, that's what I figured, but my worry upon seeing this is "what happens if lower() changes again and people forget to update the list of exceptions?". Unless they have unit testing on the entire Unicode code space to ensure what they're doing is always identical to 3.2.0.