6 ms·

Managing mutable data in Elixir with Rust

131 points by clarkema 3y ago

doctor_phil 3y ago
Nice. I thought that Zig would be a nice language for writing NIFs - but of course Rust would be good too. Cool!
- rubin55 3y ago
  Zigler! https://github.com/E-xyza/zigler https://github.com/E-xyza/zigler
- deleted 3y ago
  [deleted]
- impulser_ 3y ago
  Rust perfect for this because Rust code can be very reliable which is needed for NIFs in Erlang because a NIF can crash the whole VM. So using C and Zig libraries without fully understanding them can be a death trap while in Rust as long as it doesn't use unsafe code you can feel pretty good about using it.
  - cybrox 3y ago
    This has nothing to do with Rust itself. While the compiler does prevent a lot of common pitfalls, you can still write erroneous code with it. It's entirely the rustler project's effort (and goal) to wrap any kind of Rust program so that it will not bring down the BEAM under any circumstance, which they have done a great job achieving.
    - filmor 3y ago
      It's still to a large degree Rust itself. It's the language design which makes it possible to wrap the NIF C API in a safe fashion (e.g. using lifetimes, phantom data, etc.). The only additional safety feature we use is catch_unwind (https://doc.rust-lang.org/std/panic/fn.catch_unwind.html https://doc.rust-lang.org/std/panic/fn.catch_unwind.html) to prevent panics from unwinding into the BEAM (and killing it).
deleted 3y ago
[deleted]
atonse 3y ago
Getting rustler up and running for us was very easy. Thank you to the team for making this excellent library. We had some inconsistent build results (ours is an umbrella app) but apart from forcing a compilation and losing the ability to cache the rust builds, everything else has worked so well so we’re happy to get access to the massive rust ecosystem.
AlchemistCamp 3y ago
It’s exactly this use case that nudged me (primarily an Elixir dev) to start learning Rust a few years back. Unfortunately, I haven’t had a project where I’ve needed to use Rustler yet, though.
elbasti 3y ago
Cool writeup. A little ironic, since Erlang's `digraphs` are also mutable!
- Miner49er 3y ago
  Erlang's digraphs are stored in an ETS table, so aren't they only mutable in the same way that ETS tables are mutable? I don't normally see people consider (D)ETS tables as mutable, however.
  - h0l0cube 3y ago
    Yeah. I think even though the article doesn’t use it as an example, what’s really desirable about escape hatching to a systems language is the ability to in-place mutate lots of data. Specifically sparse mutations of a large chunk of data where a copy penalty would be wasteful. ETS is basically just swapping pointers (which I hope is mutation under the hood)
  - filmor 3y ago
    ETS tables are absolutely mutable, they even have specific functions to iterate over them while being mutated (https://www.erlang.org/doc/man/ets#safe_fixtable-2 https://www.erlang.org/doc/man/ets#safe_fixtable-2). I use them extensively to share data in a "lock-free" fashion with other processes (a `gen_server` that gets all messages and aggregates data in ETS tables, retrieval via direct reads on a known table name instead of gen_server:call). Mnesia is also (usually) ETS down below.
wredue 3y ago
Immutable data is not a “foundation of scalability and robustness”.
- WolfeReader 3y ago
  I'm not sure Joe Armstrong would agree with your comment.
  - parthdesai 3y ago
    After a certain scale, it actually does start to slow you down https://discord.com/blog/using-rust-to-scale-elixir-for-11-million-concurrent-users https://discord.com/blog/using-rust-to-scale-elixir-for-11-m...
    - dpflan 3y ago
      This is a great engineering blogpost. Other good ones from Discord on scaling (elixir, rust, cassandraDB vs. scyllaDB: - https://discord.com/blog/how-discord-scaled-elixir-to-5-000-000-concurrent-users https://discord.com/blog/how-discord-scaled-elixir-to-5-000-... — continually improving elixir - https://discord.com/blog/how-discord-stores-trillions-of-messages https://discord.com/blog/how-discord-stores-trillions-of-mes... — moving to scyllaDB
    - jimbokun 3y ago
      That was an excellent write up. And great case study in how to iteratively approach a performance problem. First address the algorithmic issues. Then once the algorithm is no longer the bottleneck, look into dropping into a higher performance, lower overhead language like Rust. Jumping directly to Rust with the original algorithm, would not have helped much. And in the general case, getting the algorithm right might make dropping down into Rust unnecessary. Very nice work.
  - wredue 3y ago
    I don’t really care what people making claims say when they make claims without evidence. ”Who” makes a claim has no bearing on its truth. Immutability is a tool, not a rule, and I am free to reject any assertion otherwise when those assertions provide no evidence, or shitty anecdotes. Prove your claims. Certainly, immutability is a foundation for performance problems. Another provable rule in computing is that more lines of code = more bugs. Immutability uses more lines of code. Another demonstrable fact is that Haskell based programs have just as many bugs as any other programming language whether you have immutability or not. Therefore, immutability is not a bastion of robustness. You’re going to have significant difficulty proving to me that immutability = scalability and robustness when both are demonstrably not true just by taking measurements of thing you expect to improve out of those foundations. Immutability is not a silver bullet. It is a tool that is sometimes useful, but has significant drawbacks, including shitty performance, and significantly limiting how your data can be managed (without that limitation paying off in any significant way)
NiklasBegley 3y ago
I also want to give a shout out to the Rustler folks for creating a great library! We use Rustler quite extensively at Doctave, and have written about our experiences with Rustler before [0] (though our architecture has advanced quite a bit since the article was written). Integrating Elixir and Rust has been delightfully straightforward and is a great choice for calling into libraries not available in Elixir, or offloading CPU intensive tasks. [0]: https://www.doctave.com/blog/2021/08/19/using-rust-with-elixir-for-code-reuse-and-performance https://www.doctave.com/blog/2021/08/19/using-rust-with-elix...
mgdev 3y ago
Rustler is great. Though this gets me thinking about how you can maintain as many Elixir invariants and conventions as possible, even while escaping them under the covers. Being able to call FeGraph.set/2 and have db actually be mutated violates Elixir's common call patterns, even if it's technically allowed. For example: I wonder if it wouldn't be more "erlangy"/"elixiry" to model the mutable ops behind a genserver that you send messages to. In the Elixir world it's perfectly normal to make GenServer.call/3 and expect the target PID to change its internal state in a non-deterministic way. It's one of the only APIs that explicitly blesses this. The ETS API is another. Alternatively, you could have the ref store both a DB sequence and a ref ID (set to the last DB sequence), and compare them on operations. If you call FeGraph.set/2 with the same db ref two times, you compare the ref ID to the sequence and panic if they aren't equal. They always need to operate with the latest ref. Then at last the local semantics are maintained. Maybe this is less relevant for the FeGraph example, since Elixir libs dealing with data are more willing to treat the DB as a mutable thing (ETS, Digraph). But the it's not universal. Postgrex, for example, follows the DB-as-PID convention. Defaulting to an Elixiry pattern by default for Rustler implementation is probably a good practice.
- clarkema 3y ago
  That's an interesting point that I should perhaps have covered in the original article. The real code that this is based on is in fact hidden behind a GenServer for this exact reason -- to maintain the expectations of other Elixir code that has to interact with it. The advantage of the escape hatch, as another commenter mentions, is allowing efficient sparse mutations of a large chunk of data, without having to pay a copy penalty every time. I definitely wouldn't recommend sharing the db handle widely.
  - lvass 3y ago
    Have you measured performance? If mutating from Elixir like this can bring serious benefits, maybe there's a place for mutable versions of libraries like Explorer and Nx.
    - clarkema 3y ago
hpeter 3y ago
This is super cool. I learn something new every day.

131 points by clarkema 3y ago

doctor_phil 3y ago

Nice. I thought that Zig would be a nice language for writing NIFs - but of course Rust would be good too. Cool!

rubin55 3y ago
Zigler! https://github.com/E-xyza/zigler https://github.com/E-xyza/zigler
deleted 3y ago
[deleted]
impulser_ 3y ago
Rust perfect for this because Rust code can be very reliable which is needed for NIFs in Erlang because a NIF can crash the whole VM. So using C and Zig libraries without fully understanding them can be a death trap while in Rust as long as it doesn't use unsafe code you can feel pretty good about using it.
- cybrox 3y ago
  This has nothing to do with Rust itself. While the compiler does prevent a lot of common pitfalls, you can still write erroneous code with it. It's entirely the rustler project's effort (and goal) to wrap any kind of Rust program so that it will not bring down the BEAM under any circumstance, which they have done a great job achieving.
  - filmor 3y ago
    It's still to a large degree Rust itself. It's the language design which makes it possible to wrap the NIF C API in a safe fashion (e.g. using lifetimes, phantom data, etc.). The only additional safety feature we use is catch_unwind (https://doc.rust-lang.org/std/panic/fn.catch_unwind.html https://doc.rust-lang.org/std/panic/fn.catch_unwind.html) to prevent panics from unwinding into the BEAM (and killing it).

deleted 3y ago

[deleted]

atonse 3y ago

Getting rustler up and running for us was very easy. Thank you to the team for making this excellent library. We had some inconsistent build results (ours is an umbrella app) but apart from forcing a compilation and losing the ability to cache the rust builds, everything else has worked so well so we’re happy to get access to the massive rust ecosystem.

AlchemistCamp 3y ago

It’s exactly this use case that nudged me (primarily an Elixir dev) to start learning Rust a few years back. Unfortunately, I haven’t had a project where I’ve needed to use Rustler yet, though.

elbasti 3y ago

Cool writeup. A little ironic, since Erlang's `digraphs` are also mutable!

Miner49er 3y ago
Erlang's digraphs are stored in an ETS table, so aren't they only mutable in the same way that ETS tables are mutable? I don't normally see people consider (D)ETS tables as mutable, however.
- h0l0cube 3y ago
  Yeah. I think even though the article doesn’t use it as an example, what’s really desirable about escape hatching to a systems language is the ability to in-place mutate lots of data. Specifically sparse mutations of a large chunk of data where a copy penalty would be wasteful. ETS is basically just swapping pointers (which I hope is mutation under the hood)
- filmor 3y ago
  ETS tables are absolutely mutable, they even have specific functions to iterate over them while being mutated (https://www.erlang.org/doc/man/ets#safe_fixtable-2 https://www.erlang.org/doc/man/ets#safe_fixtable-2). I use them extensively to share data in a "lock-free" fashion with other processes (a `gen_server` that gets all messages and aggregates data in ETS tables, retrieval via direct reads on a known table name instead of gen_server:call). Mnesia is also (usually) ETS down below.

wredue 3y ago

Immutable data is not a “foundation of scalability and robustness”.

WolfeReader 3y ago
I'm not sure Joe Armstrong would agree with your comment.
- parthdesai 3y ago
  After a certain scale, it actually does start to slow you down https://discord.com/blog/using-rust-to-scale-elixir-for-11-million-concurrent-users https://discord.com/blog/using-rust-to-scale-elixir-for-11-m...
  - dpflan 3y ago
    This is a great engineering blogpost. Other good ones from Discord on scaling (elixir, rust, cassandraDB vs. scyllaDB: - https://discord.com/blog/how-discord-scaled-elixir-to-5-000-000-concurrent-users https://discord.com/blog/how-discord-scaled-elixir-to-5-000-... — continually improving elixir - https://discord.com/blog/how-discord-stores-trillions-of-messages https://discord.com/blog/how-discord-stores-trillions-of-mes... — moving to scyllaDB
  - jimbokun 3y ago
    That was an excellent write up. And great case study in how to iteratively approach a performance problem. First address the algorithmic issues. Then once the algorithm is no longer the bottleneck, look into dropping into a higher performance, lower overhead language like Rust. Jumping directly to Rust with the original algorithm, would not have helped much. And in the general case, getting the algorithm right might make dropping down into Rust unnecessary. Very nice work.
- wredue 3y ago
  I don’t really care what people making claims say when they make claims without evidence. ”Who” makes a claim has no bearing on its truth. Immutability is a tool, not a rule, and I am free to reject any assertion otherwise when those assertions provide no evidence, or shitty anecdotes. Prove your claims. Certainly, immutability is a foundation for performance problems. Another provable rule in computing is that more lines of code = more bugs. Immutability uses more lines of code. Another demonstrable fact is that Haskell based programs have just as many bugs as any other programming language whether you have immutability or not. Therefore, immutability is not a bastion of robustness. You’re going to have significant difficulty proving to me that immutability = scalability and robustness when both are demonstrably not true just by taking measurements of thing you expect to improve out of those foundations. Immutability is not a silver bullet. It is a tool that is sometimes useful, but has significant drawbacks, including shitty performance, and significantly limiting how your data can be managed (without that limitation paying off in any significant way)

NiklasBegley 3y ago

I also want to give a shout out to the Rustler folks for creating a great library! We use Rustler quite extensively at Doctave, and have written about our experiences with Rustler before [0] (though our architecture has advanced quite a bit since the article was written). Integrating Elixir and Rust has been delightfully straightforward and is a great choice for calling into libraries not available in Elixir, or offloading CPU intensive tasks. [0]: https://www.doctave.com/blog/2021/08/19/using-rust-with-elixir-for-code-reuse-and-performance https://www.doctave.com/blog/2021/08/19/using-rust-with-elix...

mgdev 3y ago

Rustler is great. Though this gets me thinking about how you can maintain as many Elixir invariants and conventions as possible, even while escaping them under the covers. Being able to call FeGraph.set/2 and have db actually be mutated violates Elixir's common call patterns, even if it's technically allowed. For example: I wonder if it wouldn't be more "erlangy"/"elixiry" to model the mutable ops behind a genserver that you send messages to. In the Elixir world it's perfectly normal to make GenServer.call/3 and expect the target PID to change its internal state in a non-deterministic way. It's one of the only APIs that explicitly blesses this. The ETS API is another. Alternatively, you could have the ref store both a DB sequence and a ref ID (set to the last DB sequence), and compare them on operations. If you call FeGraph.set/2 with the same db ref two times, you compare the ref ID to the sequence and panic if they aren't equal. They always need to operate with the latest ref. Then at last the local semantics are maintained. Maybe this is less relevant for the FeGraph example, since Elixir libs dealing with data are more willing to treat the DB as a mutable thing (ETS, Digraph). But the it's not universal. Postgrex, for example, follows the DB-as-PID convention. Defaulting to an Elixiry pattern by default for Rustler implementation is probably a good practice.

clarkema 3y ago
That's an interesting point that I should perhaps have covered in the original article. The real code that this is based on is in fact hidden behind a GenServer for this exact reason -- to maintain the expectations of other Elixir code that has to interact with it. The advantage of the escape hatch, as another commenter mentions, is allowing efficient sparse mutations of a large chunk of data, without having to pay a copy penalty every time. I definitely wouldn't recommend sharing the db handle widely.
- lvass 3y ago
  Have you measured performance? If mutating from Elixir like this can bring serious benefits, maybe there's a place for mutable versions of libraries like Explorer and Nx.
  - clarkema 3y ago

hpeter 3y ago

This is super cool. I learn something new every day.