6 ms·

It's a nice feature but I can't help feeling like, if you need a stable pointer to an item in a collection, ArrayList is the wrong data structure to use? Maybe

by _bohm 17d ago

It's a nice feature but I can't help feeling like, if you need a stable pointer to an item in a collection, ArrayList is the wrong data structure to use? Maybe someone can chime in and give me an example of when you'd do this instead of, e.g., just storing an index. Alternatively, you could use an Unrolled Linked List (FKA SegmentedList in Zig before it was removed in 0.16, not sure why).

nvme0n1p1 17d ago
SegmentedList had a weird API, especially the way you control the list growth factor by the size of an inline array. And it hadn't kept up with stdlib norms in recent versions. I do hope it comes back eventually with an improved API. At least we got Deque in exchange. I use that far more often than I used SegmentedList.
- _bohm 17d ago
  Aha, gotcha. To tell you the truth, I don't think I ever used it. I have a custom implementation I wrote because I didn't realize at the time that SegmentedList was an unrolled linked list :p
beepbooptheory 17d ago
Maybe one use case is if you are interfacing with external C library and you're stuck with pointers?
the__alchemist 17d ago
My mental model of Zig is that it is explicitly the language for developers who prefer using pointers in business logic (instead of just in MMIMO, and are looking for something with improvements over C); i.e. exactly this class of abstraction.
- _bohm 17d ago
  Having written a bunch of Zig, I wouldn't say that the language design or culture explicitly encourages the use of pointers over indices in such situations. I would say it's more a language which trusts the programmer to make correct decisions about which constructs are appropriate in any given circumstance.
  - svachalek 17d ago
    Hasn't the past 30 years of the Internet age taught us that given such trust, programmers will make the incorrect decision with horrifying predictability? The most trivial level of software security requires that pointer safety needs to be mathematically proven not up to human (or LLM) judgment.
    - yefol 17d ago
      [flagged]
      - insanitybit 17d ago
        Who cares what mojo handles? This is about Zig and memory safety.
    - insanitybit 17d ago
      Yes, undoubtedly. Anyone in denial of this should be legally barred from programming.
  - the__alchemist 17d ago
    I appreciate the explanation! I'm a noob at this sort of thing. (My experience with pointers is limited to MMIO and C FFI). Regarding trusting the programmer: is that in the context of pointers, or more broadly? I'm betting I'm missing more, e.g. maybe mutation control too across threads/cores etc? Or as a broad design principle? (I have written some Zig as a learning dive, but don't Grok it, as am waiting for some core functionality like HALs, GUI libs, 3D libs etc. Was also a bit miffed by the 'operator overloading will never be allowed' as the use cases where I use low level languages have a near total overlap with the ones where I use vectors, quaternions, and matrices. Stated another way: I learned with higher level langs first, which I believe biased my mental model. Example: Say I want to use a C library in my rust program/lib. (Example: CMSIS DSP). One of the actions I take in the wrapper is convert the pointer to an array ref; my mental model is the param is a list of items; it is divorced from memory. If I want to read/write a reg, or access FLASH, that's where I look to pointers. I e I think zig vs others is about if you want to conflate or divorce collections and memory.
bvrmn 17d ago
ArrayList is a very generic (pun not intended) structure and could be stretched quite freely in any direction with useful property of owning underlying slice. Like readonly preallocated ArrayList is a thing.
- _bohm 17d ago
  That's a good point. Using lockPointers would be a good way to enforce at runtime that your ArrayList is truly read-only.
hansvm 17d ago
I use it in a lot of places where I know the max capacity ahead of time -- ensureCapacity() followed by a lot of *AssumeCapacity()-styled commands. It's convenient for all of the ... convenience ... methods (append() requires some bookkeeping somewhere, appendSlice() requires more, and so on). In those usages, it's basically syntactic sugar over a slice. That's not a perfect solution, but it's reasonably good often enough that I keep doing it. The proposed change doesn't do much for me personally (memory safety is ensured in other ways, and if it weren't I wouldn't be annoyed debugging the allocator-observed errors), but I could see myself using it at some other point in time for the same class of usages, or I could see other people relying on it when they choose that class of coding.
dataflow 17d ago
I don't know Zig, but conceptually: a direct pointer is the fastest way to access an object. An arraylist is the fastest dynamic sequence of objects (fattest in access, not in growth). You use these when you need the performance. It's not often but it certainly happens. The most trivial example is a string that you append to but still need to pass to a C API in between that expects it to be contagious, but it's far more useful than just for storing characters.
- _bohm 17d ago
  Sure, in cases where you need elements to be contiguous in memory then certainly an unrolled linked list is not appropriate. There's usually not a meaningful performance difference between a pointer deref and an indexed array access, however.
- kllrnohj 17d ago
  Indexing into an array is direct pointer access, there's just an addition in front of it but it's hard to imagine that showing up at all in even the tightest of benchmark loops
  - dataflow 17d ago
    I can't speak for your imagination, but this absolutely does come up if you're writing high-performance code. Also note that being able to access arbitrary objects (as opposed to objects in the same array) requires storing two pieces of information: an index and a pointer to the beginning of the array. So it can use twice as much memory, which affects your cache etc., though you don't even need that to see the effect.
    - pocksuppet 6d ago
      The addition is much cheaper than the memory access. The register pressure is worse than the addition. The size reduction of the index (versus the pointer) and resulting cache pressure reduction can outweigh the register pressure and the addition if you are storing many indices.
ivanjermakov 17d ago
I count this as a "rookie at system programming" mistake alongside returning a reference to a local variable. Rust is great at this because borrow checker can catch those at compile time and it can _teach_ devs to not do that.

by _bohm 17d ago

nvme0n1p1 17d ago

SegmentedList had a weird API, especially the way you control the list growth factor by the size of an inline array. And it hadn't kept up with stdlib norms in recent versions. I do hope it comes back eventually with an improved API. At least we got Deque in exchange. I use that far more often than I used SegmentedList.

_bohm 17d ago
Aha, gotcha. To tell you the truth, I don't think I ever used it. I have a custom implementation I wrote because I didn't realize at the time that SegmentedList was an unrolled linked list :p

beepbooptheory 17d ago

Maybe one use case is if you are interfacing with external C library and you're stuck with pointers?

the__alchemist 17d ago

My mental model of Zig is that it is explicitly the language for developers who prefer using pointers in business logic (instead of just in MMIMO, and are looking for something with improvements over C); i.e. exactly this class of abstraction.

_bohm 17d ago
Having written a bunch of Zig, I wouldn't say that the language design or culture explicitly encourages the use of pointers over indices in such situations. I would say it's more a language which trusts the programmer to make correct decisions about which constructs are appropriate in any given circumstance.
- svachalek 17d ago
  Hasn't the past 30 years of the Internet age taught us that given such trust, programmers will make the incorrect decision with horrifying predictability? The most trivial level of software security requires that pointer safety needs to be mathematically proven not up to human (or LLM) judgment.
  - yefol 17d ago
    [flagged]
    - insanitybit 17d ago
      Who cares what mojo handles? This is about Zig and memory safety.
  - insanitybit 17d ago
    Yes, undoubtedly. Anyone in denial of this should be legally barred from programming.
- the__alchemist 17d ago
  I appreciate the explanation! I'm a noob at this sort of thing. (My experience with pointers is limited to MMIO and C FFI). Regarding trusting the programmer: is that in the context of pointers, or more broadly? I'm betting I'm missing more, e.g. maybe mutation control too across threads/cores etc? Or as a broad design principle? (I have written some Zig as a learning dive, but don't Grok it, as am waiting for some core functionality like HALs, GUI libs, 3D libs etc. Was also a bit miffed by the 'operator overloading will never be allowed' as the use cases where I use low level languages have a near total overlap with the ones where I use vectors, quaternions, and matrices. Stated another way: I learned with higher level langs first, which I believe biased my mental model. Example: Say I want to use a C library in my rust program/lib. (Example: CMSIS DSP). One of the actions I take in the wrapper is convert the pointer to an array ref; my mental model is the param is a list of items; it is divorced from memory. If I want to read/write a reg, or access FLASH, that's where I look to pointers. I e I think zig vs others is about if you want to conflate or divorce collections and memory.

bvrmn 17d ago

ArrayList is a very generic (pun not intended) structure and could be stretched quite freely in any direction with useful property of owning underlying slice. Like readonly preallocated ArrayList is a thing.

_bohm 17d ago
That's a good point. Using lockPointers would be a good way to enforce at runtime that your ArrayList is truly read-only.

hansvm 17d ago

I use it in a lot of places where I know the max capacity ahead of time -- ensureCapacity() followed by a lot of *AssumeCapacity()-styled commands. It's convenient for all of the ... convenience ... methods (append() requires some bookkeeping somewhere, appendSlice() requires more, and so on). In those usages, it's basically syntactic sugar over a slice. That's not a perfect solution, but it's reasonably good often enough that I keep doing it. The proposed change doesn't do much for me personally (memory safety is ensured in other ways, and if it weren't I wouldn't be annoyed debugging the allocator-observed errors), but I could see myself using it at some other point in time for the same class of usages, or I could see other people relying on it when they choose that class of coding.

dataflow 17d ago

I don't know Zig, but conceptually: a direct pointer is the fastest way to access an object. An arraylist is the fastest dynamic sequence of objects (fattest in access, not in growth). You use these when you need the performance. It's not often but it certainly happens. The most trivial example is a string that you append to but still need to pass to a C API in between that expects it to be contagious, but it's far more useful than just for storing characters.

_bohm 17d ago
Sure, in cases where you need elements to be contiguous in memory then certainly an unrolled linked list is not appropriate. There's usually not a meaningful performance difference between a pointer deref and an indexed array access, however.
kllrnohj 17d ago
Indexing into an array is direct pointer access, there's just an addition in front of it but it's hard to imagine that showing up at all in even the tightest of benchmark loops
- dataflow 17d ago
  I can't speak for your imagination, but this absolutely does come up if you're writing high-performance code. Also note that being able to access arbitrary objects (as opposed to objects in the same array) requires storing two pieces of information: an index and a pointer to the beginning of the array. So it can use twice as much memory, which affects your cache etc., though you don't even need that to see the effect.
  - pocksuppet 6d ago
    The addition is much cheaper than the memory access. The register pressure is worse than the addition. The size reduction of the index (versus the pointer) and resulting cache pressure reduction can outweigh the register pressure and the addition if you are storing many indices.

ivanjermakov 17d ago

I count this as a "rookie at system programming" mistake alongside returning a reference to a local variable. Rust is great at this because borrow checker can catch those at compile time and it can _teach_ devs to not do that.