7 ms·
> It's also pretty telling that every article that tries to explain how to safely copy or concat strings in C, like this one, only ever works with ASCII, no att
by _gabe_ 2y ago
> It's also pretty telling that every article that tries to explain how to safely copy or concat strings in C, like this one, only ever works with ASCII, no attempt whatsoever to handle UTF-8 and keep code points together, let alone grapheme clusters.
Can you expand on this? Why does it matter to keep code points and grapheme clusters together in the case of truncation? If you’re already truncating the string, then you can just copy as many bytes as possible. Then, later when you interpret that string, you’ll hit a malformed codepoint and ignore it. I guess what you might be getting at is that if you have a codepoint sequence, then you shouldn’t copy the bytes if they can’t all fit in the truncated string?
I feel like this is an edge case and not “the reason almost all C software has problems with non-English strings”. 99% of the time, copying up until the null byte is fine, whether or not the string is UTF-8 or ASCII. The reason most C software doesn’t work with non-English strings is because the developer never added support. The bytes are still there, they just need to be interpreted correctly in the UI portions of the code.
- Retr0id 2y agoTruncating mid-codepoint produces an invalid utf8 sequence, which some decoders will silently ignore or replace with <?> glyphs, and others will fail loudly. Truncating mid-grapheme-cluster can change semantics in unexpected ways. For example, the "family" emoji(s) can be encoded as a set of conjoined codepoints for each family member: https://stackoverflow.com/questions/49958287/printing-family-emoji-with-u200d-zero-width-joiner-directly-vs-via-list https://stackoverflow.com/questions/49958287/printing-family.... Depending on where you truncate, you might just get the father, which probably won't cause you any serious issues but it might cause confusion depending on context. (IMHO, if you're truncating a way where this would matter, you should probably be truncating in screen-space, i.e., way higher up the stack)
- _gabe_ 2y agoYep, this all makes sense and is what I was thinking as well. I also think you’re right about truncation being a better idea in the rendering state rather than in the bytes themselves. Either way, truncation of strings being displayed to the user requires extra care no matter what language you’re using. As far as this being the reason most C code doesn’t handle UTF-8 though, I’m still skeptical.
- deleted 2y ago[deleted]