7 ms·

One thing I don't like about Rust is how taking a slice of a string can cause a runtime panic if the start or end of the slice ends up intersecting a multi-byte

One thing I don't like about Rust is how taking a slice of a string can cause a runtime panic if the start or end of the slice ends up intersecting a multi-byte UTF-8 char. I would prefer it if this feature didn't exist at all rather than cause runtime panics. https://play.rust-lang.org/?gist=e02ce5e9aacfee3a2b4917d5624b9ec1&version=stable https://play.rust-lang.org/?gist=e02ce5e9aacfee3a2b4917d5624...

dbaupp 8y ago
String slicing using byte indices has to exist in some form, since it is the only thing that is efficient (O(1)). But, I guess it could have used syntax other than somestring[...].
- throw0u1t 8y ago
  It could slice on bytes and return a slice of bytes since the String type is a wrapper over Vec<u8>.
  - dbaupp 8y ago
    That means one loses all the conveniences and guarantees of the string types and, in many cases, forces an immediate revalidation the byte slice as UTF-8 to get back to &str, which is O(n). Furthermore, this is also rather clunky. I suppose one could have it return StrWithInvalidSurrounds, where just the first (at most) 3 and last (at most) 3 bytes might be invalid, which would then allow for O(1) revalidation to a &str, and even other operations like continuing to slice... But this is even more clunky for actual use! I think a moderately less clunky API might have been to not use integers for byte indexing, but instead some ByteIndex wrapper type that string operations return, meaning one can't just write `s[..5]` in an attempt to get the first 5 characters of the string. (Also, there's str::get that returns an Option: https://doc.rust-lang.org/std/primitive.str.html#method.get https://doc.rust-lang.org/std/primitive.str.html#method.get )
  - int_19h 8y ago
    If you want to just slice on bytes without any String semantics, why not use Vec<u8> then? String implies that it is, well, a string.
pmarreck 8y ago
Does this bug exist because it would be too expensive to check every string before slicing? (Being Rust-ignorant), can you not type a binary as UTF-8? Are there 2 versions of string functions, fast ones that assume ASCII and slow ones that assume UTF-8?
- steveklabnik 8y ago
  Every string is checked. But UTF8 is a multi-byte encoding, and slicing works per bytes, so you if you slice in the middle of a multi-byte character, you may get nonsense. The error happens because of this checking, not in spite of it. String always assumes full UTF-8. You could make an AsciiString type if you wanted, but it's not provided by the standard library.
  - da_chicken 8y ago
    The obvious follow up question would be: so why is slicing a string a byte-wise operation and not a character-wise operation? If a string is an array of characters, why does it let me refer to individual bytes without explicitly casting it to a byte array? How often comparatively do you want the nth byte compared to the nth character? I would suspect that's pretty rare.
    - steveklabnik 8y ago
      As stated below, indexing is an O(1) operation, and that is a O(n) operation. > If a string is an array of characters It is not, it is an array (technically vector) of bytes.
      - da_chicken 8y ago
        Who cares if it's O(1) if it causes a panic? What good is high performance if it doesn't complete or isn't safe? At the very least, shouldn't there be an O(n) method to do character-wise slicing?
        8y ago
gamegoblin 8y ago
It is a common pattern in Rust to use [] for things that cannot fail and will panic otherwise and a method for things that can fail and return Option or Result. e.g. my_hashmap["foo"] will panic at runtime if the key "foo" is not present, or return the associated value if it is. But my_hashmap.get("foo") will return None if "foo" is not present and Some(value) if it is.
- throw0u1t 8y ago
  TIL! I'm still learning Rust so it's good to learn this now! Thanks!
  - test9753 8y ago
    One approach to solve the slicing issue: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=8127145d60b52d9c29f1518418845bc3 https://play.rust-lang.org/?version=stable&mode=debug&editio...
- sephoric 8y ago
  What's the point of the [] version then? It seems inherently more dangerous, and Rust emphasizes safety. I know it wants to be pragmatic as well as safe, but this seems like a strange default.
  - steveklabnik 8y ago
    There's a few things that come into play here: First of all, panics are perfectly safe. None of this has to do with safety guarantees. Second, the [] syntax is controlled by the Index trait, which returns an &T, not an Option<&T>. It does this due to Rust's error handling philosophy. There's two kinds of errors: recoverable and unrecoverable errors. When something shouldn't fail, unless there's a bug, you shouldn't be using Option/Result, you should panic. When something may normally fail, and you want to be able to handle that explicitly, you should use Option/Result. If [] always returned an Option, you'd be seeing tons and tons and tons of unwraps. It's not the right default here. However, that's why the .get method also exists: If you do think that this may fail, but not due to a bug, then you should use .get instead, which does give you an option. TL;DR: everything is tradeoffs, and we picked a specific set of them, and that's how they all play out together. Personal commentary: this is the kind of thing that's largely concerning until you actually use the language more, IMHO. Dealing with Options all the time here would feel really bad. Consider the other sub-thread about floats; it often feels like boilerplate for no good reason. That would introduce this for every single time you want to index something, which is a very common operation.
jzelinskie 8y ago
Go indexes bytes on strings, even though there's a builtin type called Rune which delimits utf-8 codepoints. This is yet another footgun. Is there a language that doesn't handle this poorly? https://play.golang.org/p/CkBp0w8T621 https://play.golang.org/p/CkBp0w8T621
- Skunkleton 8y ago
  UTF-8 is at odds with efficient array indexing. I like pythons approach where bytes and strings are distinct types, though I have no idea what it is doing under the hood.
  - colatkinson 8y ago
    I actually had to work with Python strings at the C level recently, and their approach is pretty clever. IIRC, the runtime can take any common form of Unicode, and will store it. When you access that string, the accessor requests a specific encoding, and the runtime will convert if need be, and then store it in the string object. So it handles the (very) common case of needing the same encoding multiple times (e.g. for all file paths on Windows), while not introducing too much overhead in memory or speed. I could be mistaken on exact details though, especially since I recall there being multiple implementations even within py3.x.
    - Skunkleton 8y ago
      Any idea how it handles indexing? Does it convert everything to 32 bit chars and ignore graphemes?
  - int_19h 8y ago
    Modern Python uses whatever representation is sufficient to ensure 1-unit-per-codepoint for a given string (which it can do on creation, since strings are immutable). So you get ASCII, UTF-16 sans surrogate pairs, or UTF-32. This is great for high-level code, but painful to work with from native code, because it usually needs some specific encoding to call into other libraries, and it's usually UTF-8 - so you need to re-encode all the time.
pornel 8y ago
It's not a problem in practice, because you'd use something like `.char_indices()` iterator, or result from a substring search, etc. to get correct offsets in the first place. It's not useful to blindly read at random offsets in UTF-8 strings. If it didn't panic, you'd get garbage. If offsets were automatically moved to skip over garbage, you wouldn't know what you're getting, and your overall algorithm would likely end up with nonsense (duplicated or skipped chars). For algorithms that don't care about characters or UTF-8 validity, there's zero-cost `.as_bytes()`.
- StavrosK 8y ago
  What does zero-cost mean in this context? It must cost something to run, no? Or is it basically a compiler hint instructing the next function to treat the data as pure bytes?
  - burntsushi 8y ago
    In this particular context, you can think of going from a `&str` to a `&[u8]` via `string.as_bytes()` as a safe cast. The in-memory representation remains the same, and the function call will almost certainly be inlined because its implementation is trivial.
- KajMagnus 8y ago
  Couldn't syntax like `a_string[..3]` be made to result in compilation errors in Rust? Since that'd almost always be a bug? (right?) And in the rare cases, when it's not a bug, then one can just use `as_bytes` which would be good to do in any case, to indicate to other humans that this is not a bug. B.t.w. I love the error message `[..3]` generates: "thread 'main' panicked at 'byte index 3 is not a char boundary; it is inside '早' (bytes 2..5) of `ab早`'" — I've never seen such easy to understand error messages in any language (except for in a few cases in Scala).
  - steveklabnik 8y ago
    We could have never implemented Index for String, sure. We have though, so removing it would be a breaking change.
hinkley 8y ago
This seems specious to me. The only way to get an invalid index in a string in any language is that you either have an array index arithmetic error or you are blindly operating on a string you haven't validated. If you want all the data after a : character, you slice on the index of the :. The character after it is going to be the beginning of a UTF-8 character. You do not under any circumstances guess that the colon is at position 6 in the string. That's not safe. Why are you going cowboy in a language that is so obsessed with safety?
- v_lisivka 8y ago
  I just realized that I have bug in my GPS driver. It operates on ASCII data, so [] operator is safe, BUT data can be corrupted (low chance, but non-zero), so it can form valid multibyte character, so my code will panic on it, trying to parse and validate NMEA message.
  - UncleEntity 8y ago
    Panicking on parsing corrupted data seem like a feature to me... It's like the default rule in a lexer, if it ever gets to it then it's an unrecognized character and lexing stops so error handling can proceed. --edit-- Which I now realize was probably your point.
- kd5bjo 8y ago
  Truncating a string to fit in a fixed-size storage field is probably the most common reason to split at a particular byte position. If you’re throwing data away anyway, you probably don’t care too much about the little bit of corruption. Granted, this is certainly incorrect but has little to do with safety, especially if the downstream code has to revalidate everything anyway.

dbaupp 8y ago

String slicing using byte indices has to exist in some form, since it is the only thing that is efficient (O(1)). But, I guess it could have used syntax other than somestring[...].

throw0u1t 8y ago
It could slice on bytes and return a slice of bytes since the String type is a wrapper over Vec<u8>.
- dbaupp 8y ago
  That means one loses all the conveniences and guarantees of the string types and, in many cases, forces an immediate revalidation the byte slice as UTF-8 to get back to &str, which is O(n). Furthermore, this is also rather clunky. I suppose one could have it return StrWithInvalidSurrounds, where just the first (at most) 3 and last (at most) 3 bytes might be invalid, which would then allow for O(1) revalidation to a &str, and even other operations like continuing to slice... But this is even more clunky for actual use! I think a moderately less clunky API might have been to not use integers for byte indexing, but instead some ByteIndex wrapper type that string operations return, meaning one can't just write `s[..5]` in an attempt to get the first 5 characters of the string. (Also, there's str::get that returns an Option: https://doc.rust-lang.org/std/primitive.str.html#method.get https://doc.rust-lang.org/std/primitive.str.html#method.get )
- int_19h 8y ago
  If you want to just slice on bytes without any String semantics, why not use Vec<u8> then? String implies that it is, well, a string.

pmarreck 8y ago

Does this bug exist because it would be too expensive to check every string before slicing? (Being Rust-ignorant), can you not type a binary as UTF-8? Are there 2 versions of string functions, fast ones that assume ASCII and slow ones that assume UTF-8?

steveklabnik 8y ago
Every string is checked. But UTF8 is a multi-byte encoding, and slicing works per bytes, so you if you slice in the middle of a multi-byte character, you may get nonsense. The error happens because of this checking, not in spite of it. String always assumes full UTF-8. You could make an AsciiString type if you wanted, but it's not provided by the standard library.
- da_chicken 8y ago
  The obvious follow up question would be: so why is slicing a string a byte-wise operation and not a character-wise operation? If a string is an array of characters, why does it let me refer to individual bytes without explicitly casting it to a byte array? How often comparatively do you want the nth byte compared to the nth character? I would suspect that's pretty rare.
  - steveklabnik 8y ago
    As stated below, indexing is an O(1) operation, and that is a O(n) operation. > If a string is an array of characters It is not, it is an array (technically vector) of bytes.
    - da_chicken 8y ago
      Who cares if it's O(1) if it causes a panic? What good is high performance if it doesn't complete or isn't safe? At the very least, shouldn't there be an O(n) method to do character-wise slicing?
      - 8y ago

gamegoblin 8y ago

It is a common pattern in Rust to use [] for things that cannot fail and will panic otherwise and a method for things that can fail and return Option or Result. e.g. my_hashmap["foo"] will panic at runtime if the key "foo" is not present, or return the associated value if it is. But my_hashmap.get("foo") will return None if "foo" is not present and Some(value) if it is.

throw0u1t 8y ago
TIL! I'm still learning Rust so it's good to learn this now! Thanks!
- test9753 8y ago
  One approach to solve the slicing issue: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=8127145d60b52d9c29f1518418845bc3 https://play.rust-lang.org/?version=stable&mode=debug&editio...
sephoric 8y ago
What's the point of the [] version then? It seems inherently more dangerous, and Rust emphasizes safety. I know it wants to be pragmatic as well as safe, but this seems like a strange default.
- steveklabnik 8y ago
  There's a few things that come into play here: First of all, panics are perfectly safe. None of this has to do with safety guarantees. Second, the [] syntax is controlled by the Index trait, which returns an &T, not an Option<&T>. It does this due to Rust's error handling philosophy. There's two kinds of errors: recoverable and unrecoverable errors. When something shouldn't fail, unless there's a bug, you shouldn't be using Option/Result, you should panic. When something may normally fail, and you want to be able to handle that explicitly, you should use Option/Result. If [] always returned an Option, you'd be seeing tons and tons and tons of unwraps. It's not the right default here. However, that's why the .get method also exists: If you do think that this may fail, but not due to a bug, then you should use .get instead, which does give you an option. TL;DR: everything is tradeoffs, and we picked a specific set of them, and that's how they all play out together. Personal commentary: this is the kind of thing that's largely concerning until you actually use the language more, IMHO. Dealing with Options all the time here would feel really bad. Consider the other sub-thread about floats; it often feels like boilerplate for no good reason. That would introduce this for every single time you want to index something, which is a very common operation.

jzelinskie 8y ago

Go indexes bytes on strings, even though there's a builtin type called Rune which delimits utf-8 codepoints. This is yet another footgun. Is there a language that doesn't handle this poorly? https://play.golang.org/p/CkBp0w8T621 https://play.golang.org/p/CkBp0w8T621

Skunkleton 8y ago
UTF-8 is at odds with efficient array indexing. I like pythons approach where bytes and strings are distinct types, though I have no idea what it is doing under the hood.
- colatkinson 8y ago
  I actually had to work with Python strings at the C level recently, and their approach is pretty clever. IIRC, the runtime can take any common form of Unicode, and will store it. When you access that string, the accessor requests a specific encoding, and the runtime will convert if need be, and then store it in the string object. So it handles the (very) common case of needing the same encoding multiple times (e.g. for all file paths on Windows), while not introducing too much overhead in memory or speed. I could be mistaken on exact details though, especially since I recall there being multiple implementations even within py3.x.
  - Skunkleton 8y ago
    Any idea how it handles indexing? Does it convert everything to 32 bit chars and ignore graphemes?
- int_19h 8y ago
  Modern Python uses whatever representation is sufficient to ensure 1-unit-per-codepoint for a given string (which it can do on creation, since strings are immutable). So you get ASCII, UTF-16 sans surrogate pairs, or UTF-32. This is great for high-level code, but painful to work with from native code, because it usually needs some specific encoding to call into other libraries, and it's usually UTF-8 - so you need to re-encode all the time.

pornel 8y ago

It's not a problem in practice, because you'd use something like `.char_indices()` iterator, or result from a substring search, etc. to get correct offsets in the first place. It's not useful to blindly read at random offsets in UTF-8 strings. If it didn't panic, you'd get garbage. If offsets were automatically moved to skip over garbage, you wouldn't know what you're getting, and your overall algorithm would likely end up with nonsense (duplicated or skipped chars). For algorithms that don't care about characters or UTF-8 validity, there's zero-cost `.as_bytes()`.

StavrosK 8y ago
What does zero-cost mean in this context? It must cost something to run, no? Or is it basically a compiler hint instructing the next function to treat the data as pure bytes?
- burntsushi 8y ago
  In this particular context, you can think of going from a `&str` to a `&[u8]` via `string.as_bytes()` as a safe cast. The in-memory representation remains the same, and the function call will almost certainly be inlined because its implementation is trivial.
KajMagnus 8y ago
Couldn't syntax like `a_string[..3]` be made to result in compilation errors in Rust? Since that'd almost always be a bug? (right?) And in the rare cases, when it's not a bug, then one can just use `as_bytes` which would be good to do in any case, to indicate to other humans that this is not a bug. B.t.w. I love the error message `[..3]` generates: "thread 'main' panicked at 'byte index 3 is not a char boundary; it is inside '早' (bytes 2..5) of `ab早`'" — I've never seen such easy to understand error messages in any language (except for in a few cases in Scala).
- steveklabnik 8y ago
  We could have never implemented Index for String, sure. We have though, so removing it would be a breaking change.

hinkley 8y ago

This seems specious to me. The only way to get an invalid index in a string in any language is that you either have an array index arithmetic error or you are blindly operating on a string you haven't validated. If you want all the data after a : character, you slice on the index of the :. The character after it is going to be the beginning of a UTF-8 character. You do not under any circumstances guess that the colon is at position 6 in the string. That's not safe. Why are you going cowboy in a language that is so obsessed with safety?

v_lisivka 8y ago
I just realized that I have bug in my GPS driver. It operates on ASCII data, so [] operator is safe, BUT data can be corrupted (low chance, but non-zero), so it can form valid multibyte character, so my code will panic on it, trying to parse and validate NMEA message.
- UncleEntity 8y ago
  Panicking on parsing corrupted data seem like a feature to me... It's like the default rule in a lexer, if it ever gets to it then it's an unrecognized character and lexing stops so error handling can proceed. --edit-- Which I now realize was probably your point.
kd5bjo 8y ago
Truncating a string to fit in a fixed-size storage field is probably the most common reason to split at a particular byte position. If you’re throwing data away anyway, you probably don’t care too much about the little bit of corruption. Granted, this is certainly incorrect but has little to do with safety, especially if the downstream code has to revalidate everything anyway.