6 ms·

In fact, Zig has pretty much the same support for strings as both Rust and Go. The primary difference which people seem to complain about, that "[]const u8" is

In fact, Zig has pretty much the same support for strings as both Rust and Go. The primary difference which people seem to complain about, that "[]const u8" is not written as "string", can be solved by writing `const string = []const u8;` at the top of your program if you like. Rust: `&str`, 'some bytes, and a length', 'Constructing a non-UTF-8 string slice is not immediate undefined behavior, but any function called on a string slice may assume that it is valid UTF-8, which means that a non-UTF-8 string slice can lead to undefined behavior down the road.' - also `String` and `OsString`. edit: apparently Rust enforces that &str cannot be created with invalid UTF8, at the additional performance cost of runtime checks, so it has stronger guarantees than other languages like Go/Java/Zig/etc (as usual) Go: `string`, 'in effect a read-only slice of bytes', 'a string holds arbitrary bytes. It is not required to hold Unicode text, UTF-8 text, or any other predefined format. As far as the content of a string is concerned, it is exactly equivalent to a slice of bytes.' Zig: `[]const u8`, 'some bytes, and a length', 'Zig has no concept of strings', 'by convention parameters that are "strings" are expected to be UTF-8 encoded slices of u8.', 'Generally, you can use UTF-8 and not worry about whether something is a string.' - and the stdlib has functions for working with unicode and strings.

Ygg2 3y ago
> Rust: `&str`, 'some bytes, and a length', 'Constructing a non-UTF-8 string slice is not immediate undefined behavior If you're constructing invalid `str` you already have severely fucked up. And dog is eating your HDD.
- DinaCoder98 3y ago
  > If you're constructing invalid `str` you already have severely fucked up. Presumably it'd be better to know this immediately on initialization of the variable rather than at some point in the program that actually expects valid utf8—god forbid you transmit or persist the data before this happens.
  - duped 3y ago
    That's why str::from_utf8 returns Result<&str, FromUtf8Error>. And if you know the string is UTF-8 encoded and don't want to pay the cost, there's `from_utf8_unchecked`, which is marked unsafe.
    - DinaCoder98 3y ago
      Ah, I thought you were referring to the benefits of such a system as compared to zig and blithely passing the responsibility of input validation on to the programmer. Rust of course demonstrates the benefit of checking validity at initialization.
josephg 3y ago
> 'Constructing a non-UTF-8 string slice is not immediate undefined behavior Rust’s string constructors (for both str and String) check that the data contains strictly valid UTF8. You can opt out of this check in performance sensitive code - but doing so is considered unsafe. In safe rust it’s impossible to construct a str / String which contains invalid utf8. Eg: https://doc.rust-lang.org/std/str/fn.from_utf8.html https://doc.rust-lang.org/std/str/fn.from_utf8.html
- slimsag 3y ago
  good to know! I just copied that quote directly from the Rust docs[0] which were not clear it is only possible in unsafe code. [0] https://doc.rust-lang.org/std/primitive.str.html#invariant https://doc.rust-lang.org/std/primitive.str.html#invariant
- _nalply 3y ago
  You shouldn't misuse unsafe Rust to create `&str` or `String` with invalid UTF-8. Doing that violates assumptions some optimizations rely on. I even would contend it's undefined behavior. At least I am convinced that you should never have invalid UTF-8 in `&str` or `String`. If you need that, there's the [bstr][0] crate for byte strings which may or may not be valid UTF-8. Essentially these are just byte slices `&[u8]` and the owned version byte vectors `Vec<u8>` and traits and methods around them. Very useful for text data which is not reliably UTF-8. [0]: https://crates.io/crates/bstr https://crates.io/crates/bstr The idea of using unsafety here is to have a way to tell the Rust compiler: "Trust me, I promise that I only fill the strings with valid UTF-8 data. You don't need to insert code to check UTF-8 validity." And the Rust compiler answers: "Okay. I accept that but if something goes wrong I can't catch that." This allows avoiding repeated validations of a byte slice for UTF-8 validity, for example.
  - josephg 3y ago
    > You shouldn't misuse unsafe Rust to create `&str` or `String` with invalid UTF-8. Sorry - I think my comment wasn't clear. The point of from_utf8_unchecked isn't to create strings with invalid UTF8. The point is that if you know that your string has already been validated, you can skip re-validating it.
pgwhalen 3y ago
I am definitely not a rust expert, but as far as I can tell this is just not true? The quote you copied is from the documentation is in the context of creating a string in an unsafe manner. If you're going to include unsafe behavior, you may as well say that nothing in rust means anything at all.
duped 3y ago
> also `String` and `OsString` Because these are different types with different semantics. Why should there be one string type? Languages have to interface with the real world, where string data could be anything, but programs still need to be written for the common cases... hence, &str and String.
bobbylarrybobby 3y ago
String is utf8, OsString is not (necessarily). Having two separate types mean you can guarantee utf8 when you want to, but don't have to when you can't, such as when dealing with “strings” that an OS produces, which are basically just sequences of bytes (perhaps with zero bytes disallowed). When you have one string type that has to do both jobs, you lose out on the guarantees of the more restrictive type.
thayne 3y ago
> and the stdlib has functions for working with unicode and strings. Last time I looked I don't think it did. It looks like there is a unicode namespace now, but it is very primative. It doesn't have a simple way to iterate over codepoints, much less graphemes. Nor does it have anything to inspect the category a codepoint belongs to.
- slimsag 3y ago
  `std.unicode` supports validation, codepoint iteration, conversion from UTF8 <-> other encodings, etc. It doesn't have graphemes (neither does Go or Rust), or other advanced unicode support. Instead that is provided by ziglyph[0] [0] https://codeberg.org/dude_the_builder/ziglyph https://codeberg.org/dude_the_builder/ziglyph

Ygg2 3y ago

> Rust: `&str`, 'some bytes, and a length', 'Constructing a non-UTF-8 string slice is not immediate undefined behavior If you're constructing invalid `str` you already have severely fucked up. And dog is eating your HDD.

DinaCoder98 3y ago
> If you're constructing invalid `str` you already have severely fucked up. Presumably it'd be better to know this immediately on initialization of the variable rather than at some point in the program that actually expects valid utf8—god forbid you transmit or persist the data before this happens.
- duped 3y ago
  That's why str::from_utf8 returns Result<&str, FromUtf8Error>. And if you know the string is UTF-8 encoded and don't want to pay the cost, there's `from_utf8_unchecked`, which is marked unsafe.
  - DinaCoder98 3y ago
    Ah, I thought you were referring to the benefits of such a system as compared to zig and blithely passing the responsibility of input validation on to the programmer. Rust of course demonstrates the benefit of checking validity at initialization.

josephg 3y ago

> 'Constructing a non-UTF-8 string slice is not immediate undefined behavior Rust’s string constructors (for both str and String) check that the data contains strictly valid UTF8. You can opt out of this check in performance sensitive code - but doing so is considered unsafe. In safe rust it’s impossible to construct a str / String which contains invalid utf8. Eg: https://doc.rust-lang.org/std/str/fn.from_utf8.html https://doc.rust-lang.org/std/str/fn.from_utf8.html

slimsag 3y ago
good to know! I just copied that quote directly from the Rust docs[0] which were not clear it is only possible in unsafe code. [0] https://doc.rust-lang.org/std/primitive.str.html#invariant https://doc.rust-lang.org/std/primitive.str.html#invariant
_nalply 3y ago
You shouldn't misuse unsafe Rust to create `&str` or `String` with invalid UTF-8. Doing that violates assumptions some optimizations rely on. I even would contend it's undefined behavior. At least I am convinced that you should never have invalid UTF-8 in `&str` or `String`. If you need that, there's the [bstr][0] crate for byte strings which may or may not be valid UTF-8. Essentially these are just byte slices `&[u8]` and the owned version byte vectors `Vec<u8>` and traits and methods around them. Very useful for text data which is not reliably UTF-8. [0]: https://crates.io/crates/bstr https://crates.io/crates/bstr The idea of using unsafety here is to have a way to tell the Rust compiler: "Trust me, I promise that I only fill the strings with valid UTF-8 data. You don't need to insert code to check UTF-8 validity." And the Rust compiler answers: "Okay. I accept that but if something goes wrong I can't catch that." This allows avoiding repeated validations of a byte slice for UTF-8 validity, for example.
- josephg 3y ago
  > You shouldn't misuse unsafe Rust to create `&str` or `String` with invalid UTF-8. Sorry - I think my comment wasn't clear. The point of from_utf8_unchecked isn't to create strings with invalid UTF8. The point is that if you know that your string has already been validated, you can skip re-validating it.

pgwhalen 3y ago

I am definitely not a rust expert, but as far as I can tell this is just not true? The quote you copied is from the documentation is in the context of creating a string in an unsafe manner. If you're going to include unsafe behavior, you may as well say that nothing in rust means anything at all.

duped 3y ago

> also `String` and `OsString` Because these are different types with different semantics. Why should there be one string type? Languages have to interface with the real world, where string data could be anything, but programs still need to be written for the common cases... hence, &str and String.

bobbylarrybobby 3y ago

String is utf8, OsString is not (necessarily). Having two separate types mean you can guarantee utf8 when you want to, but don't have to when you can't, such as when dealing with “strings” that an OS produces, which are basically just sequences of bytes (perhaps with zero bytes disallowed). When you have one string type that has to do both jobs, you lose out on the guarantees of the more restrictive type.

thayne 3y ago

> and the stdlib has functions for working with unicode and strings. Last time I looked I don't think it did. It looks like there is a unicode namespace now, but it is very primative. It doesn't have a simple way to iterate over codepoints, much less graphemes. Nor does it have anything to inspect the category a codepoint belongs to.

slimsag 3y ago
`std.unicode` supports validation, codepoint iteration, conversion from UTF8 <-> other encodings, etc. It doesn't have graphemes (neither does Go or Rust), or other advanced unicode support. Instead that is provided by ziglyph[0] [0] https://codeberg.org/dude_the_builder/ziglyph https://codeberg.org/dude_the_builder/ziglyph