7 ms·

Can a Rust binary use incompatible versions of the same library?

84 points by braxxox 2y ago

woodruffw 2y ago
I thought this was about loading two incompatible versions of a shared object into the same address space at first :-) The author correctly contrasts Rust (and NPM's) behavior with that of Python/pip, where only one version per package name is allowed. The Python packaging ecosystem could in theory standardize a form of package name mangling wherein multiple versions could be imported simultaneously (akin to what's currently possible with multiple vendored versions), but that would likely be a significant undertaking given that a lot of applications probably - accidentally - break the indirect relationship and directly import their transitive dependencies. (The more I work in Python, the more I think that Python's approach is actually a good one: preventing multiple versions of the same package prevents dependency graph spaghetti when every subdependency depends on a slightly different version, and provides a strong incentive to keep public API surfaces small and flexible. But I don't think that was the intention, more of an accidental perk of an otherwise informal approach to packaging.)
- stouset 2y ago
  > dependency graph spaghetti The worst spaghetti comes from hard dependencies on minor versions and revisions. I will die on the hill that you should only ever specify dependencies on “at least this major-minor (and optionally and rarely revision for a bugfix)” in whatever the syntax is for your preferred language. Excepting of course a known incompatibility with a specific version or range of versions, and/or developers who refuse to get on the semver bandwagon who should collectively be rounded up and yelled at. In Rust, Cargo makes this super easy: “x.y.z” means “>= x.y.z, < (x+1).0.0”. It’s fine to ship a generated lock file that locks everything to a fixed, known-good version of all your dependencies. But you should be able to trivially run an update that will bring everything to the latest minor and revision (and alert on newer major versions).
  - joshka 2y ago
    There's a subtle point there though. When you rely on something that was introduced in x.y.z, stating that your version requirement is x.y.0 is an error that can easily cause downstream breakage.
    - ComputerGuru 2y ago
      I’m confused. If you rely on a feature introduced in X.y.z why would you specify X.y.0 to begin with (and not just X.y.z)? In practice, usual rust projects that have not put a ton of work into their dependencies encode X.y.z in Cargo.toml matching the current release at the time they developed the system. So you get at worst an unnecessarily higher version requirement but never a lower one. Moreover, rust semver would normally imply that new features should only be introduced in X.y releases, so this doesn’t really happen in practice!
      - gmueckl 2y ago
        It's easy to accidentally ship a minimum version requirement that is out of date when you also consistently use lock files pinned to newer versions. The code may silently depend on something introduced in a newer version pulled in by the lock file.
        sanxiyn 2y ago
        You can have a CI builder using direct-minimal-versions to check this.
        ComputerGuru 2y ago
        Point releases are often bugfix releases, i.e. not api changes but runtime changes. CI won’t help without very specific accompanying tests.
        stouset 2y ago
- rtpg 2y ago
  Another thing I appreciate about this in the Python world is it avoids an issue I've seen in node a lot, which is people being too clever by a half and pre-emptively adding major version bounds to their library. So foo depends on "bar<9", despite bar 9, 10, 11, 12, 13, and 14 all working with foo's usage of bar. The end result of this is that you end up with some random library in your stack (4 transitive layers deep because of course it is) holding back stuff like chokadir in a huge chunk of your dep tree for... no real good reason. So you now have several copies of a huge library. Of course new major versions might break your usage! Minor versions might as well! Patch versions too sometimes! Upper bounds pre-emptively set help mainly in one thing, and that's reducing the number of people who would help "beta-test" new major versions because they don't care enough to pin their own dependencies.
- josephg 2y ago
  > (The more I work in Python, the more I think that Python's approach is actually a good one ...) I've come to the opposite conclusion. I've "git cloned" several programs in both python and ruby (which has the same behaviour) only to discover that I can't actually install the project's dependencies. The larger your gemfile / requirements.txt is, the more likely this is to happen. All it takes is a couple packages in your tree to update their own dependencies out of sync with one another and you can run into this problem. A build that worked yesterday doesn't work today. Not because anyone made a mistake - but just because you got unlucky. Ugh. Its a completely unnecessary landmine. Worse yet, new developers (or new teammembers) are very likely to run into this problem as it shows up when you're getting your dev environment setup. This problem is entirely unnecessary. In (almost) every way, software should treat foo-1.x.x as a totally distinct package from foo-2.x.x. They're mutually incompatible anyway, and semantically the only thing they share is their name. There's no reason both packages can't be loaded into the package namespace at the same time. No reason but the mistakes of shortsighted package management systems. RAM is cheap. My attention is expensive. Print a warning if you must, and I'll fix it when I feel like it.
  - woodruffw 2y ago
    I'm not saying this hasn't happened to you, but I'm curious: are you working with scientific Python codebases or similar? I've done Python development off and on for the last ~10 years, and I think I can count the number of times I've had transitive conflicts on a single hand. But I almost never touch scientific/statistical/etc. Python codebases, so I'm curious is this is a discipline/practice concern in different subsets of the ecosystem. (One of the ways I have seen this happen in this past is people attempting to use multiple requirements sources without synchronizing them or resolving them simultaneously. That's indeed a highway to pain city, and it's why modern Python packaging emphasizes either using a single standard metadata file like pyproject.toml or a fully locked environment specification like a frozen requirements file.)
    - jmillikin 2y ago
      I've encountered the same problem with Python codebases in the LLM / machine learning space. The requirements.txt files for those projects are full of unversioned dependencies, including Git repositories at some floating ref (such as master/HEAD). In the easy cases, digging through the PyPI version history to identify the latest version as of some date is enough to get a working install (as far as I can tell -- maybe it's half-broken and I only use the working half?). In the hard cases, it may take an entire day to locate a CI log or contemporary bug report or something that lists out all the installed package versions. It doesn't help that every Python-based project seems to have its own bespoke packaging system. It's never just pip + requirements.txt, it'll have a Dockerfile with `apt update`, or some weird meta-packaging thing like Conda that adds it own layers of non-determinism. Overall the feeling is that it was only barely holding together on the author's original machine, and getting it to build anywhere else is pure luck. For example: https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/v1.10.1/requirements.txt https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob... (with some discussion at https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/1373 https://github.com/AUTOMATIC1111/stable-diffusion-webui/disc...)
gorgoiler 2y ago
For fun, you could add this to Python and I think it would it cover a lot of edge cases? You would need: A function v_tree_install(spec) which installs a versioned pypi package like “foo=3.2” and all its dependencies in its own tree, rather than in site-packages. Another pair of functions v_import and v_from_import to wrap importlib with a name, version, and symbols. These functions know how to find the versioned package in its special tree and push that tree to sys.path before starting the import. To cover the case for when the imported code has dynamic imports you could also wrap any callable code (functions, classes) with a wrapper that also does the sys.push/pop before/after each call. You then replace third party imports in your code with calls assigning to symbols in your module: # import foo foo = v_import(“foo==3.2”) # from foo import bar, baz as q bar, q = v_from_import( “foo>=3.3”, “bar”, “baz”, ) Finally, provide a function (or CLI tool) to statically scan your code looking for v_import and calling v_tree_install ahead of time. Or just let v_import do it. Edit: …and you’d need to edit the sys.modules cache too, or purge it after each “clever” import?
- ahupp 2y ago
  You might be able to do this transparently with a [MetaPathFinder](https://docs.python.org/3/library/importlib.html#importlib.abc.MetaPathFinder https://docs.python.org/3/library/importlib.html#importlib.a...), the only trickyness would be replacing the lookup in sys.modules which I don't think has has an official interface.
- dathinab 2y ago
  I have though about this a bunch (and have been annoyed by it a bunch). But the main issue here is somewhat designed around a "scripts and folder of scripts from a package" design principle while such a loading system would fundamentally need to always work in terms of packages. E.g. you wouldn't execute `main.py` but `package:main`. (Through this is already the direction a bunch of tooling moved to, e.g. poetry scripts, some of the WSGI and especially more modern ASGI implementations etc.) Another issue is that rust can reliable detect type collisions of the same type of two different versions and force you to fix them. With a lot of struct type annotations on python and tooling like mypy this might be possible (with many limitations) but as of today it in practice likely will not be caught. Sometimes that is what you want (ducktyping happens to work). But for any of the reflection/inspection heavy python library this is a recipe for quite obscure errors somewhere in not so obvious inspection/auto generation/metaclass related magic code. Python can't, escept it can Anyway technically it's possible, you can put a version into __qualname__, and mess with the import system enough to allow imports to be contextual based on the manifest of the module they come from. (Through you probably would not be fully standard conform python, but we are speaking about dynamic patching pythons import system, there is nothing standard about it)
hsfzxjy 2y ago
So both versions of log crate manage their own internal states within the same process? Would this lead to surprising results?
- dwattttt 2y ago
  Their internal states in Rust are also namespaced, so two incompatible crates in the same process won't observe each others symbols. If they access external resources that are not namespaced though, that could be a problem.
  - tick_tock_tick 2y ago
    Such as stdout or stderr that a log crate would be using?
    - dwattttt 2y ago
      That's not a very clear example. You don't need to be using multiple versions of the same dependency to contend on access to stderr/out, just having a println in your code along with logging code will have the same effect. I haven't ever observed a problem of concurrent access to stdout/err though, I expect because the methods for accessing stdout/err lock them for the duration of their printing. If you Google for "Rust print console slow", you'll probably find advice to explicitly lock it, to avoid individual printlns from each acquiring the lock.
  - amelius 2y ago
    How would that work for malloc? How can you have two different functions manage the same heap?
    - anonymoushn 2y ago
      "The same heap" isn't a coherent concept here. Your malloc-implementing memory allocator has some global state, and that state has some pointers to some addresses it got from mmap and some metadata about how long those spans of memory are, which parts are unused, and how long the values it has returned from malloc previously are. If you managed to use two of these, they would each contain data referring to different non-overlapping sets of memory mappings. If you accidentally used a pointer from one with the other, you would go instantly to C UB land: The free() function frees the memory space pointed to by ptr, which must have been returned by a previous call to malloc(), calloc() or realloc(). Otherwise, or if free(ptr) has already been called before, undefined behavior occurs. If ptr is NULL, no operation is performed.
btilly 2y ago
This is great for avoiding conflicts when you try to get your project running. It sucks when there is a vulnerability in a particular library, and you're trying to track all of the ways in which that vulnerable code is being pulled into your project. My preference is to force the conflict up front by saying that you can't import conflicting versions. This creates a constant stream of small problems, but avoids really big ones later. However I absolutely understand why a lot of people prefer it the other way around.
- oefrha 2y ago
  Go got this right: you want an incompatible version, you have to use a different import path. Then you can only pick one version (which is deterministically the lowest possible version) for a certain import path, not a hundred different versions. Also forces people to actually take backwards compatibility seriously.
  - btilly 2y ago
    I'm not surprised. Go's design is heavily informed by what does and does not cause cascading design problems in software engineering at scale. These practical concerns are very different from the kinds of issues that academia had been focused on. But practical solutions to practical problems is central to Go's popularity.
  - whatshisface 2y ago
    Backwards compatibility is more difficult in Rust for many reasons. For example, you can't add a new item to an enum without creating missing-case errors everywhere it is used.
    - dwattttt 2y ago
      That's a true effect, although I'd question whether it makes it harder or easier for things to be backwards compatible. I use Rust because I trust it to throw up a bunch of errors when I make changes; if I handle all the cases of an enum somewhere, and suddenly there's a new enum variant, the answer is probably that I need to handle the new variant there too.
gmueckl 2y ago
I cannot shake the feeling that this is actually a misfeature that will get people into trouble in new and puzzling ways. The isolated classloaders in Java and the assembly domains in .Net didn't turn out to be very bright ideas and from a software design perspective this is virtually identical.
- pornel 2y ago
  It's been working like that for a decade, and it's been fine. Rust/Cargo have been designed for it from the start.
pjmlp 2y ago
And this is why one gets to watch some crates being compiled from scratch multiple times in a single "make world" build.
WD-42 2y ago
This isn’t a magic bullet. Using multiple versions of the same crate can still blow up your project. For example, the compiler error in this example: note: perhaps two different versions of crate `smithay_client_toolkit` are being used? https://github.com/pop-os/launcher/issues/237 https://github.com/pop-os/launcher/issues/237
- gary_0 2y ago
  Ah, I was wondering what would happen if you're using a type from lib-v2 and an intermediary library passes you that type from lib-v1, and the type has changed internally. Good to know the Rust compiler is set up to catch that. (I've seen cases where that happens with C and C++ software, and things seem to compile and run... until everything explodes. Fun times.)
anonymoushn 2y ago
You can do this, but you can't use two semver-compatible versions of the same library in *different binaries* in the same workspace.
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
alkonaut 2y ago
How does this work? Assume that the log crate in its internal state has a lock it uses for synchronizing writing to some log endpoint. If I have two versions of log in my process then they must have two copies of their internal state. So they both point to the same log endpoint, but they have one mutex each? That means it "works" at compile time but fails at runtime? That's the worst kind of "works!" Or if I depend transitively on two versions of a library (e.g. a matrix math lib) through A and B and try to read a value from A and send it into B. Then presumably due to type namespacing that will fail at compile time? So the options when using incompatible dependencies are a) it compiles, but fails at runtime, b) it doesn't compile, or c) it compiles and works at runtime?
- yorwba 2y ago
  If the log endpoint is external to your process and two different copies of the logging crate in the same process writing to it cause problems, two identical copies of the logging crate in different processes will likely also cause problems. The solution here is global synchronzation, not just within one process. If the log endpoint is internal to your process, how did you end up with two independent mutexes guarding (or not guarding) access to the same resource? It should be wrapped in a shared mutex as soon as you create it, and before passing it to the different versions of the logging crate. And unless you use unsafe, Rust's ownership model forces you to do that, because it forbids having two overlapping mutable references at the same time.
  - alkonaut 2y ago
    Perhaps a log wasn't the best example due to how the resource (a log sink) is often external. Take some simpler example: a counter (such as sequential ID generator). It's an in memory counter doing an atomic increment that returns the next ID. Two of my projects in depend on it when they create new items. Both want to generate process wide unique IDs. But if they depend on two versions of the crate then there would be two memory locations, and thus two sequences of IDs generated, so two of the frogs in my game will risk having the same ID? There is no sharing problem here, the problem is the opposite: that there are two memory locations instead of one?
aragilar 2y ago
FYI, Python can/did support multiple versions via buildout (http://www.buildout.org/en/latest/ http://www.buildout.org/en/latest/) but it's complicated and wide-scale support has probably bit-rotted away.
richardwhiuk 2y ago
You can do this in one crate: [dependencies] foo_v1 = { package = "foo", version = "1" } foo_v2 = { package = "foo", version = "2" }
jcelerier 2y ago
How does that work if you want to export a symbol for dlopen?
- dboreham 2y ago
  Nobody knows about dynamic linking now. And most languages don't support it (looking at you: golang).
- dureuill 2y ago
  I'm not sure I understand the use case here. Are you asking if you can depend on two versions of the same crate, for a crate that exports a `#[no_mangle]` or `#[export_name]` function? I guess you could slap a `#[used]` attribute on your exported functions, and use their mangled name to call them with dlopen, but that would be unwieldy and guessing the disambiguator used by the compiler error prone to impossible. Other than that, you cannot. What you can do is define the `#[no_mangle]` or `#[export_name]` function at the top-level of your shared library. It makes sense to have a single crate bear the responsibility of exporting the interface of your shared library. I wish Rust would enforce that, but the shared library story in Rust is subpar. Fortunately it never actually comes into play, as the ecosystem relies on static linking
  - jcelerier 2y ago
    > I'm not sure I understand the use case here. Are you asking if you can depend on two versions of the same crate, for a crate that exports a `#[no_mangle]` or `#[export_name]` function? Yes, exactly. > Other than that, you cannot. so, to the question "Can a Rust binary use incompatible versions of the same library?", then the answer is definitely "no". It's not yes if it cannot cover one of the most basic use cases when making OS-native software. To be clear: no language targeting OS-native dynamic libraries can solve this, the problem is in how PE and ELF works.
    - richardwhiuk 2y ago
      Rust uses -sys crates to link to non-native dependencies, with a links key in the manifest - https://doc.rust-lang.org/cargo/reference/build-scripts.html#the-links-manifest-key https://doc.rust-lang.org/cargo/reference/build-scripts.html... This mechanism allows Cargo to prevent multiply linking to an external library.
dboreham 2y ago
Every language will re-create its own version (sic) of DLL-hell.

84 points by braxxox 2y ago

I thought this was about loading two incompatible versions of a shared object into the same address space at first :-) The author correctly contrasts Rust (and NPM's) behavior with that of Python/pip, where only one version per package name is allowed. The Python packaging ecosystem could in theory standardize a form of package name mangling wherein multiple versions could be imported simultaneously (akin to what's currently possible with multiple vendored versions), but that would likely be a significant undertaking given that a lot of applications probably - accidentally - break the indirect relationship and directly import their transitive dependencies. (The more I work in Python, the more I think that Python's approach is actually a good one: preventing multiple versions of the same package prevents dependency graph spaghetti when every subdependency depends on a slightly different version, and provides a strong incentive to keep public API surfaces small and flexible. But I don't think that was the intention, more of an accidental perk of an otherwise informal approach to packaging.)

stouset 2y ago
> dependency graph spaghetti The worst spaghetti comes from hard dependencies on minor versions and revisions. I will die on the hill that you should only ever specify dependencies on “at least this major-minor (and optionally and rarely revision for a bugfix)” in whatever the syntax is for your preferred language. Excepting of course a known incompatibility with a specific version or range of versions, and/or developers who refuse to get on the semver bandwagon who should collectively be rounded up and yelled at. In Rust, Cargo makes this super easy: “x.y.z” means “>= x.y.z, < (x+1).0.0”. It’s fine to ship a generated lock file that locks everything to a fixed, known-good version of all your dependencies. But you should be able to trivially run an update that will bring everything to the latest minor and revision (and alert on newer major versions).
- joshka 2y ago
  There's a subtle point there though. When you rely on something that was introduced in x.y.z, stating that your version requirement is x.y.0 is an error that can easily cause downstream breakage.
  - ComputerGuru 2y ago
    I’m confused. If you rely on a feature introduced in X.y.z why would you specify X.y.0 to begin with (and not just X.y.z)? In practice, usual rust projects that have not put a ton of work into their dependencies encode X.y.z in Cargo.toml matching the current release at the time they developed the system. So you get at worst an unnecessarily higher version requirement but never a lower one. Moreover, rust semver would normally imply that new features should only be introduced in X.y releases, so this doesn’t really happen in practice!
    - gmueckl 2y ago
      It's easy to accidentally ship a minimum version requirement that is out of date when you also consistently use lock files pinned to newer versions. The code may silently depend on something introduced in a newer version pulled in by the lock file.
      - sanxiyn 2y ago
        You can have a CI builder using direct-minimal-versions to check this.
        ComputerGuru 2y ago
        Point releases are often bugfix releases, i.e. not api changes but runtime changes. CI won’t help without very specific accompanying tests.
        stouset 2y ago
rtpg 2y ago
Another thing I appreciate about this in the Python world is it avoids an issue I've seen in node a lot, which is people being too clever by a half and pre-emptively adding major version bounds to their library. So foo depends on "bar<9", despite bar 9, 10, 11, 12, 13, and 14 all working with foo's usage of bar. The end result of this is that you end up with some random library in your stack (4 transitive layers deep because of course it is) holding back stuff like chokadir in a huge chunk of your dep tree for... no real good reason. So you now have several copies of a huge library. Of course new major versions might break your usage! Minor versions might as well! Patch versions too sometimes! Upper bounds pre-emptively set help mainly in one thing, and that's reducing the number of people who would help "beta-test" new major versions because they don't care enough to pin their own dependencies.
josephg 2y ago
> (The more I work in Python, the more I think that Python's approach is actually a good one ...) I've come to the opposite conclusion. I've "git cloned" several programs in both python and ruby (which has the same behaviour) only to discover that I can't actually install the project's dependencies. The larger your gemfile / requirements.txt is, the more likely this is to happen. All it takes is a couple packages in your tree to update their own dependencies out of sync with one another and you can run into this problem. A build that worked yesterday doesn't work today. Not because anyone made a mistake - but just because you got unlucky. Ugh. Its a completely unnecessary landmine. Worse yet, new developers (or new teammembers) are very likely to run into this problem as it shows up when you're getting your dev environment setup. This problem is entirely unnecessary. In (almost) every way, software should treat foo-1.x.x as a totally distinct package from foo-2.x.x. They're mutually incompatible anyway, and semantically the only thing they share is their name. There's no reason both packages can't be loaded into the package namespace at the same time. No reason but the mistakes of shortsighted package management systems. RAM is cheap. My attention is expensive. Print a warning if you must, and I'll fix it when I feel like it.
- woodruffw 2y ago
  I'm not saying this hasn't happened to you, but I'm curious: are you working with scientific Python codebases or similar? I've done Python development off and on for the last ~10 years, and I think I can count the number of times I've had transitive conflicts on a single hand. But I almost never touch scientific/statistical/etc. Python codebases, so I'm curious is this is a discipline/practice concern in different subsets of the ecosystem. (One of the ways I have seen this happen in this past is people attempting to use multiple requirements sources without synchronizing them or resolving them simultaneously. That's indeed a highway to pain city, and it's why modern Python packaging emphasizes either using a single standard metadata file like pyproject.toml or a fully locked environment specification like a frozen requirements file.)
  - jmillikin 2y ago
    I've encountered the same problem with Python codebases in the LLM / machine learning space. The requirements.txt files for those projects are full of unversioned dependencies, including Git repositories at some floating ref (such as master/HEAD). In the easy cases, digging through the PyPI version history to identify the latest version as of some date is enough to get a working install (as far as I can tell -- maybe it's half-broken and I only use the working half?). In the hard cases, it may take an entire day to locate a CI log or contemporary bug report or something that lists out all the installed package versions. It doesn't help that every Python-based project seems to have its own bespoke packaging system. It's never just pip + requirements.txt, it'll have a Dockerfile with `apt update`, or some weird meta-packaging thing like Conda that adds it own layers of non-determinism. Overall the feeling is that it was only barely holding together on the author's original machine, and getting it to build anywhere else is pure luck. For example: https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/v1.10.1/requirements.txt https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob... (with some discussion at https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/1373 https://github.com/AUTOMATIC1111/stable-diffusion-webui/disc...)

gorgoiler 2y ago

For fun, you could add this to Python and I think it would it cover a lot of edge cases? You would need: A function v_tree_install(spec) which installs a versioned pypi package like “foo=3.2” and all its dependencies in its own tree, rather than in site-packages. Another pair of functions v_import and v_from_import to wrap importlib with a name, version, and symbols. These functions know how to find the versioned package in its special tree and push that tree to sys.path before starting the import. To cover the case for when the imported code has dynamic imports you could also wrap any callable code (functions, classes) with a wrapper that also does the sys.push/pop before/after each call. You then replace third party imports in your code with calls assigning to symbols in your module: # import foo foo = v_import(“foo==3.2”) # from foo import bar, baz as q bar, q = v_from_import( “foo>=3.3”, “bar”, “baz”, ) Finally, provide a function (or CLI tool) to statically scan your code looking for v_import and calling v_tree_install ahead of time. Or just let v_import do it. Edit: …and you’d need to edit the sys.modules cache too, or purge it after each “clever” import?

ahupp 2y ago
You might be able to do this transparently with a [MetaPathFinder](https://docs.python.org/3/library/importlib.html#importlib.abc.MetaPathFinder https://docs.python.org/3/library/importlib.html#importlib.a...), the only trickyness would be replacing the lookup in sys.modules which I don't think has has an official interface.
dathinab 2y ago
I have though about this a bunch (and have been annoyed by it a bunch). But the main issue here is somewhat designed around a "scripts and folder of scripts from a package" design principle while such a loading system would fundamentally need to always work in terms of packages. E.g. you wouldn't execute `main.py` but `package:main`. (Through this is already the direction a bunch of tooling moved to, e.g. poetry scripts, some of the WSGI and especially more modern ASGI implementations etc.) Another issue is that rust can reliable detect type collisions of the same type of two different versions and force you to fix them. With a lot of struct type annotations on python and tooling like mypy this might be possible (with many limitations) but as of today it in practice likely will not be caught. Sometimes that is what you want (ducktyping happens to work). But for any of the reflection/inspection heavy python library this is a recipe for quite obscure errors somewhere in not so obvious inspection/auto generation/metaclass related magic code. Python can't, escept it can Anyway technically it's possible, you can put a version into __qualname__, and mess with the import system enough to allow imports to be contextual based on the manifest of the module they come from. (Through you probably would not be fully standard conform python, but we are speaking about dynamic patching pythons import system, there is nothing standard about it)

hsfzxjy 2y ago

So both versions of log crate manage their own internal states within the same process? Would this lead to surprising results?

dwattttt 2y ago
Their internal states in Rust are also namespaced, so two incompatible crates in the same process won't observe each others symbols. If they access external resources that are not namespaced though, that could be a problem.
- tick_tock_tick 2y ago
  Such as stdout or stderr that a log crate would be using?
  - dwattttt 2y ago
    That's not a very clear example. You don't need to be using multiple versions of the same dependency to contend on access to stderr/out, just having a println in your code along with logging code will have the same effect. I haven't ever observed a problem of concurrent access to stdout/err though, I expect because the methods for accessing stdout/err lock them for the duration of their printing. If you Google for "Rust print console slow", you'll probably find advice to explicitly lock it, to avoid individual printlns from each acquiring the lock.
- amelius 2y ago
  How would that work for malloc? How can you have two different functions manage the same heap?
  - anonymoushn 2y ago
    "The same heap" isn't a coherent concept here. Your malloc-implementing memory allocator has some global state, and that state has some pointers to some addresses it got from mmap and some metadata about how long those spans of memory are, which parts are unused, and how long the values it has returned from malloc previously are. If you managed to use two of these, they would each contain data referring to different non-overlapping sets of memory mappings. If you accidentally used a pointer from one with the other, you would go instantly to C UB land: The free() function frees the memory space pointed to by ptr, which must have been returned by a previous call to malloc(), calloc() or realloc(). Otherwise, or if free(ptr) has already been called before, undefined behavior occurs. If ptr is NULL, no operation is performed.

btilly 2y ago

This is great for avoiding conflicts when you try to get your project running. It sucks when there is a vulnerability in a particular library, and you're trying to track all of the ways in which that vulnerable code is being pulled into your project. My preference is to force the conflict up front by saying that you can't import conflicting versions. This creates a constant stream of small problems, but avoids really big ones later. However I absolutely understand why a lot of people prefer it the other way around.

oefrha 2y ago
Go got this right: you want an incompatible version, you have to use a different import path. Then you can only pick one version (which is deterministically the lowest possible version) for a certain import path, not a hundred different versions. Also forces people to actually take backwards compatibility seriously.
- btilly 2y ago
  I'm not surprised. Go's design is heavily informed by what does and does not cause cascading design problems in software engineering at scale. These practical concerns are very different from the kinds of issues that academia had been focused on. But practical solutions to practical problems is central to Go's popularity.
- whatshisface 2y ago
  Backwards compatibility is more difficult in Rust for many reasons. For example, you can't add a new item to an enum without creating missing-case errors everywhere it is used.
  - dwattttt 2y ago
    That's a true effect, although I'd question whether it makes it harder or easier for things to be backwards compatible. I use Rust because I trust it to throw up a bunch of errors when I make changes; if I handle all the cases of an enum somewhere, and suddenly there's a new enum variant, the answer is probably that I need to handle the new variant there too.

gmueckl 2y ago

I cannot shake the feeling that this is actually a misfeature that will get people into trouble in new and puzzling ways. The isolated classloaders in Java and the assembly domains in .Net didn't turn out to be very bright ideas and from a software design perspective this is virtually identical.

pornel 2y ago
It's been working like that for a decade, and it's been fine. Rust/Cargo have been designed for it from the start.

pjmlp 2y ago

And this is why one gets to watch some crates being compiled from scratch multiple times in a single "make world" build.

WD-42 2y ago

This isn’t a magic bullet. Using multiple versions of the same crate can still blow up your project. For example, the compiler error in this example: note: perhaps two different versions of crate `smithay_client_toolkit` are being used? https://github.com/pop-os/launcher/issues/237 https://github.com/pop-os/launcher/issues/237

gary_0 2y ago
Ah, I was wondering what would happen if you're using a type from lib-v2 and an intermediary library passes you that type from lib-v1, and the type has changed internally. Good to know the Rust compiler is set up to catch that. (I've seen cases where that happens with C and C++ software, and things seem to compile and run... until everything explodes. Fun times.)

anonymoushn 2y ago

You can do this, but you can't use two semver-compatible versions of the same library in *different binaries* in the same workspace.

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

alkonaut 2y ago

How does this work? Assume that the log crate in its internal state has a lock it uses for synchronizing writing to some log endpoint. If I have two versions of log in my process then they must have two copies of their internal state. So they both point to the same log endpoint, but they have one mutex each? That means it "works" at compile time but fails at runtime? That's the worst kind of "works!" Or if I depend transitively on two versions of a library (e.g. a matrix math lib) through A and B and try to read a value from A and send it into B. Then presumably due to type namespacing that will fail at compile time? So the options when using incompatible dependencies are a) it compiles, but fails at runtime, b) it doesn't compile, or c) it compiles and works at runtime?

yorwba 2y ago
If the log endpoint is external to your process and two different copies of the logging crate in the same process writing to it cause problems, two identical copies of the logging crate in different processes will likely also cause problems. The solution here is global synchronzation, not just within one process. If the log endpoint is internal to your process, how did you end up with two independent mutexes guarding (or not guarding) access to the same resource? It should be wrapped in a shared mutex as soon as you create it, and before passing it to the different versions of the logging crate. And unless you use unsafe, Rust's ownership model forces you to do that, because it forbids having two overlapping mutable references at the same time.
- alkonaut 2y ago
  Perhaps a log wasn't the best example due to how the resource (a log sink) is often external. Take some simpler example: a counter (such as sequential ID generator). It's an in memory counter doing an atomic increment that returns the next ID. Two of my projects in depend on it when they create new items. Both want to generate process wide unique IDs. But if they depend on two versions of the crate then there would be two memory locations, and thus two sequences of IDs generated, so two of the frogs in my game will risk having the same ID? There is no sharing problem here, the problem is the opposite: that there are two memory locations instead of one?

aragilar 2y ago

FYI, Python can/did support multiple versions via buildout (http://www.buildout.org/en/latest/ http://www.buildout.org/en/latest/) but it's complicated and wide-scale support has probably bit-rotted away.

richardwhiuk 2y ago

You can do this in one crate: [dependencies] foo_v1 = { package = "foo", version = "1" } foo_v2 = { package = "foo", version = "2" }

jcelerier 2y ago

How does that work if you want to export a symbol for dlopen?

dboreham 2y ago
Nobody knows about dynamic linking now. And most languages don't support it (looking at you: golang).
dureuill 2y ago
I'm not sure I understand the use case here. Are you asking if you can depend on two versions of the same crate, for a crate that exports a `#[no_mangle]` or `#[export_name]` function? I guess you could slap a `#[used]` attribute on your exported functions, and use their mangled name to call them with dlopen, but that would be unwieldy and guessing the disambiguator used by the compiler error prone to impossible. Other than that, you cannot. What you can do is define the `#[no_mangle]` or `#[export_name]` function at the top-level of your shared library. It makes sense to have a single crate bear the responsibility of exporting the interface of your shared library. I wish Rust would enforce that, but the shared library story in Rust is subpar. Fortunately it never actually comes into play, as the ecosystem relies on static linking
- jcelerier 2y ago
  > I'm not sure I understand the use case here. Are you asking if you can depend on two versions of the same crate, for a crate that exports a `#[no_mangle]` or `#[export_name]` function? Yes, exactly. > Other than that, you cannot. so, to the question "Can a Rust binary use incompatible versions of the same library?", then the answer is definitely "no". It's not yes if it cannot cover one of the most basic use cases when making OS-native software. To be clear: no language targeting OS-native dynamic libraries can solve this, the problem is in how PE and ELF works.
  - richardwhiuk 2y ago
    Rust uses -sys crates to link to non-native dependencies, with a links key in the manifest - https://doc.rust-lang.org/cargo/reference/build-scripts.html#the-links-manifest-key https://doc.rust-lang.org/cargo/reference/build-scripts.html... This mechanism allows Cargo to prevent multiply linking to an external library.

dboreham 2y ago

Every language will re-create its own version (sic) of DLL-hell.