7 ms·

This is a huge leap forward for the JavaScript community—probably more than many people will realize right away. I loved Bundler's deterministic builds but cha

by tomdale 10y ago

This is a huge leap forward for the JavaScript community—probably more than many people will realize right away. I loved Bundler's deterministic builds but chafed against the Ruby limitation of only having a single version of a dependency at once. npm solved this problem elegantly, but still struggles with non-determinism. I had resigned myself to thinking that maybe these were just fundamental tradeoffs in package management that could never be resolved. Then I had the opportunity to use Cargo, the package manager for Rust. It synthesized what, in my mind, are the best features of npm and Bundler into a single package manager, with terrific speed to boot. Yarn looks like it will be Cargo for JavaScript. After Cargo improved on many of the great ideas of npm, those ideas are returning to help improve the JavaScript ecosystem. Cross-pollination at its best. This also highlights a shrewd move on the part of npm: a clear decoupling between the npm registry and client, with a well-defined protocol between the two. The strength of Node is in the staggering size of its ecosystem; how those bits end up on disk is an implementation detail. This smart separation allows for this kind of experimentation on the client side, without causing the ecosystem fragmentation that happens when a new package manager requires a new registry. I'm also happy to see that a significant amount of work has already gone into the governance model. Despite the largest contributors being Facebook employees, it looks like they've really outdone themselves making sure this is a community-run project. A BSD license, no PATENTS file, and an Ember/Rust RFC process[1]; this has all the hallmarks of a community open source project. That's critical for open infrastructure projects like this, and they've nailed it. [1]: https://github.com/yarnpkg/rfcs https://github.com/yarnpkg/rfcs I'm very much looking forward to using Yarn in my own projects because it looks like it solves a lot of real problems I encounter every day. Thanks for all the hard work!

k__ 10y ago
I find it strange that the time isn't invested in already existing projects. But at least it's a move away from NPM. I think the most problems I had with JavaScript develompent in the last 2 years came from NPM.
- tranvu 10y ago
  Yarn isn't a replacement for npm itself. It's a client that can read/write to npm, and other registries such as Bower.
  - thoughtpalette 10y ago
    Well, it seems like it could be according to the article. What would you still need to use NPM (the cli) for, other than package hosting? According to http://blog.npmjs.org/post/151660845210/hello-yarn http://blog.npmjs.org/post/151660845210/hello-yarn, it seems it doesn't work with private packages yet, which may or may not be an issue for your project. But it seems this is a complete CLI replacement for NPM.
    - wtetzner 10y ago
      Yes, it replaces the client, but it uses the same package repository and package format.
      - dacjames 10y ago
        And thus could have been implemented as part of the existing client.
        wtetzner 10y ago
        "Could have been implemented as part of the existing client" isn't the same as "Should have been implemented as part of the existing client". I personally don't know much about either tool (don't do a ton of JS), but it's possible that fixing the existing client without either breaking backward compatibility or making it too complicated (multiple modes of operation) was too difficult or not worth it. Also, I'm having a really hard time understanding the complaint about a new client. The value is in repository of reusable code, not the client. That you can use different clients with the same repository is a feature, not a bug.
        hal9zillion 10y ago
- cygned 10y ago
  Would you mind going a bit into detail? I work with npm on a daily basis and I am pretty happy so far.
  - k__ 10y ago
    The whole version range stuff got me many times. I went to use fixed versions on my own package.json files, but the deps of my deps could still be dynamic, which is even worse, since they sit deeper in my dependency graph AND there are more indirect deps than direct deps. (~50 direct, >200 indirect) Also, npm isn't deterministic and it got even worse with v3. Sometimes you get a flat list of libs, if a lib is used with multiple versions, the first usage will get installed flat, the rest in the directory of the parent lib, etc. The npm-cli is basically a mess :\
    - rubber_duck 10y ago
      The fun of kicking off a CI build after the weekend with no commits and see stuff randomly break because some dependency of a dependency got updated and broke things in a minor version is something I've only experienced in JS - beautiful.
      - markus2012 10y ago
        Well, now you can do that in Cargo as well :-) What we do currently is we lock everything to an explicit version - even libraries. At least it's possible to get deterministic builds if you are willing to do a bit of work carefully / manually updating all of your dependencies at once.
        steveklabnik 10y ago
        You shouldn't have that happen with Cargo, given that we have a lockfile. Even when you specify version ranges, you're locked to a single, specific version.
        markus2012 10y ago
lucian1900 10y ago
Multiple versions may sound like it's useful, but it's almost always a bad idea. Cargo doesn't allow it either. The problem isn't really fundamental. Bundler makes almost all the right choices already. Its major disadvantage is that it only works for Ruby.
- dikaiosune 10y ago
  Cargo definitely allows two dependencies to rely on different versions of a transitive dependency. If those deps expose a type from that dependency in their API, you can very occasionally get weird type errors because the "same" type comes from two different libraries. But otherwise it Just Works(tm).
- steveklabnik 10y ago
  Cargo does allow multiple versions of transitive dependencies. It tries to unify them as much as possible, but if it can't, it will bring in two different versions. What it _won't_ let you do is pass a v1.0 type to something that requires a v2.0 type. This will result in a compilation error. There's been some discussion around increasing Cargo's capacity in this regard (being able to say that a dependency is purely internal vs not), we'll see.
- Siyo 10y ago
  I feel like this is especially problematic when using NPM for your frontend. Now you have to run all your code through deduplication and slow down your build times or end up with huge assets. I wonder if it's really worth the trouble.
- rictic 10y ago
  This is especially true when you're going to be serving your code over the web. It's very easy when using npm to accidentally end up shipping a lot of duplicate code. That alone has me super excited about Yarn, before even getting into the performance, security, and 'no invisible dependency' wins.
- wycats 10y ago
  As a practical matter, the npm ecosystem today relies on duplication, and no new client that made the "highlander rule" (there can be only one) mandatory could succeed. Yarn does offer a `--flat` option that enforces the highlander rule, and I'm hopeful that the existence of this option will nudge the ecosystem towards an appreciation for fewer semver-major bumps and more ecosystem-wide effort to enable whole apps to work with `--flat` mode. Plz send halp!
pfraze 10y ago
> This also highlights a shrewd move on the part of npm: a clear decoupling between the npm registry and client, with a well-defined protocol between the two. The strength of Node is in the staggering size of its ecosystem; how those bits end up on disk is an implementation detail. This smart separation allows for this kind of experimentation on the client side, without causing the ecosystem fragmentation that happens when a new package manager requires a new registry. And a shrewd move by FB: to not announce their new registry on the same day.
- emerongi 10y ago
  > Yarn pulls packages from registry.yarnpkg.com, which allows them to run experiments with the Yarn client. This is a proxy that pulls packages from the official npm registry, much like npmjs.cf.[0] Time will tell whether they only want to be proxying NPM or will allow direct pushing to their own registry. If they do, JS ecosystem might see another big shift. [0] http://blog.npmjs.org/post/151660845210/hello-yarn http://blog.npmjs.org/post/151660845210/hello-yarn
  - deleted 10y ago
    [deleted]
  - jlongster 10y ago
    That big shift will have to happen first. I don't see them ever making their own registry unless 99.99% of people are using yarn and are having a lot of problems with the current npm registry. While I see a lot of people using yarn, I'm not sure about 99.99% and I think npm's registry itself is pretty good. So I don't think interests will ever align to create a new registry. Nobody wants to do that. That would have serious consquences for the JS community and would take years to recover, in my opinion.
    - wycats 10y ago
      I more or less agree with this.
    - evv 10y ago
33degrees 10y ago
> I loved Bundler's deterministic builds but chafed against the Ruby limitation of only having a single version of a dependency at once This is due to the way that Ruby includes code, where it's available globally, vs Node where code is scoped to the including module. I'm not sure how Ruby could support multiple versions with changes to the language and/or RubyGems
- tomdale 10y ago
  Yes, that's why I said it was a Ruby limitation, not a Bundler limitation.
  - 33degrees 10y ago
    Right, my comment wasn't neccesarily directed at you, but to others who might not be familiar with both
baq 10y ago
funny (and great!) how cargo became a gold standard for package management without any advertising. well done, rustaceans.
edem 10y ago
Why do you think that everybody is inspired from javascript tech? It is probably the other way around. For example there were very good build tools long before npm which are still used today and unmeasurably better (like Maven or Gradle).

by tomdale 10y ago

k__ 10y ago

I find it strange that the time isn't invested in already existing projects. But at least it's a move away from NPM. I think the most problems I had with JavaScript develompent in the last 2 years came from NPM.

tranvu 10y ago
Yarn isn't a replacement for npm itself. It's a client that can read/write to npm, and other registries such as Bower.
- thoughtpalette 10y ago
  Well, it seems like it could be according to the article. What would you still need to use NPM (the cli) for, other than package hosting? According to http://blog.npmjs.org/post/151660845210/hello-yarn http://blog.npmjs.org/post/151660845210/hello-yarn, it seems it doesn't work with private packages yet, which may or may not be an issue for your project. But it seems this is a complete CLI replacement for NPM.
  - wtetzner 10y ago
    Yes, it replaces the client, but it uses the same package repository and package format.
    - dacjames 10y ago
      And thus could have been implemented as part of the existing client.
      - wtetzner 10y ago
        "Could have been implemented as part of the existing client" isn't the same as "Should have been implemented as part of the existing client". I personally don't know much about either tool (don't do a ton of JS), but it's possible that fixing the existing client without either breaking backward compatibility or making it too complicated (multiple modes of operation) was too difficult or not worth it. Also, I'm having a really hard time understanding the complaint about a new client. The value is in repository of reusable code, not the client. That you can use different clients with the same repository is a feature, not a bug.
        hal9zillion 10y ago
cygned 10y ago
Would you mind going a bit into detail? I work with npm on a daily basis and I am pretty happy so far.
- k__ 10y ago
  The whole version range stuff got me many times. I went to use fixed versions on my own package.json files, but the deps of my deps could still be dynamic, which is even worse, since they sit deeper in my dependency graph AND there are more indirect deps than direct deps. (~50 direct, >200 indirect) Also, npm isn't deterministic and it got even worse with v3. Sometimes you get a flat list of libs, if a lib is used with multiple versions, the first usage will get installed flat, the rest in the directory of the parent lib, etc. The npm-cli is basically a mess :\
  - rubber_duck 10y ago
    The fun of kicking off a CI build after the weekend with no commits and see stuff randomly break because some dependency of a dependency got updated and broke things in a minor version is something I've only experienced in JS - beautiful.
    - markus2012 10y ago
      Well, now you can do that in Cargo as well :-) What we do currently is we lock everything to an explicit version - even libraries. At least it's possible to get deterministic builds if you are willing to do a bit of work carefully / manually updating all of your dependencies at once.
      - steveklabnik 10y ago
        You shouldn't have that happen with Cargo, given that we have a lockfile. Even when you specify version ranges, you're locked to a single, specific version.
        markus2012 10y ago

lucian1900 10y ago

Multiple versions may sound like it's useful, but it's almost always a bad idea. Cargo doesn't allow it either. The problem isn't really fundamental. Bundler makes almost all the right choices already. Its major disadvantage is that it only works for Ruby.

dikaiosune 10y ago
Cargo definitely allows two dependencies to rely on different versions of a transitive dependency. If those deps expose a type from that dependency in their API, you can very occasionally get weird type errors because the "same" type comes from two different libraries. But otherwise it Just Works(tm).
steveklabnik 10y ago
Cargo does allow multiple versions of transitive dependencies. It tries to unify them as much as possible, but if it can't, it will bring in two different versions. What it _won't_ let you do is pass a v1.0 type to something that requires a v2.0 type. This will result in a compilation error. There's been some discussion around increasing Cargo's capacity in this regard (being able to say that a dependency is purely internal vs not), we'll see.
Siyo 10y ago
I feel like this is especially problematic when using NPM for your frontend. Now you have to run all your code through deduplication and slow down your build times or end up with huge assets. I wonder if it's really worth the trouble.
rictic 10y ago
This is especially true when you're going to be serving your code over the web. It's very easy when using npm to accidentally end up shipping a lot of duplicate code. That alone has me super excited about Yarn, before even getting into the performance, security, and 'no invisible dependency' wins.
wycats 10y ago
As a practical matter, the npm ecosystem today relies on duplication, and no new client that made the "highlander rule" (there can be only one) mandatory could succeed. Yarn does offer a `--flat` option that enforces the highlander rule, and I'm hopeful that the existence of this option will nudge the ecosystem towards an appreciation for fewer semver-major bumps and more ecosystem-wide effort to enable whole apps to work with `--flat` mode. Plz send halp!

pfraze 10y ago

> This also highlights a shrewd move on the part of npm: a clear decoupling between the npm registry and client, with a well-defined protocol between the two. The strength of Node is in the staggering size of its ecosystem; how those bits end up on disk is an implementation detail. This smart separation allows for this kind of experimentation on the client side, without causing the ecosystem fragmentation that happens when a new package manager requires a new registry. And a shrewd move by FB: to not announce their new registry on the same day.

emerongi 10y ago
> Yarn pulls packages from registry.yarnpkg.com, which allows them to run experiments with the Yarn client. This is a proxy that pulls packages from the official npm registry, much like npmjs.cf.[0] Time will tell whether they only want to be proxying NPM or will allow direct pushing to their own registry. If they do, JS ecosystem might see another big shift. [0] http://blog.npmjs.org/post/151660845210/hello-yarn http://blog.npmjs.org/post/151660845210/hello-yarn
- deleted 10y ago
  [deleted]
- jlongster 10y ago
  That big shift will have to happen first. I don't see them ever making their own registry unless 99.99% of people are using yarn and are having a lot of problems with the current npm registry. While I see a lot of people using yarn, I'm not sure about 99.99% and I think npm's registry itself is pretty good. So I don't think interests will ever align to create a new registry. Nobody wants to do that. That would have serious consquences for the JS community and would take years to recover, in my opinion.
  - wycats 10y ago
    I more or less agree with this.
  - evv 10y ago

33degrees 10y ago

> I loved Bundler's deterministic builds but chafed against the Ruby limitation of only having a single version of a dependency at once This is due to the way that Ruby includes code, where it's available globally, vs Node where code is scoped to the including module. I'm not sure how Ruby could support multiple versions with changes to the language and/or RubyGems

tomdale 10y ago
Yes, that's why I said it was a Ruby limitation, not a Bundler limitation.
- 33degrees 10y ago
  Right, my comment wasn't neccesarily directed at you, but to others who might not be familiar with both

baq 10y ago

funny (and great!) how cargo became a gold standard for package management without any advertising. well done, rustaceans.

edem 10y ago

Why do you think that everybody is inspired from javascript tech? It is probably the other way around. For example there were very good build tools long before npm which are still used today and unmeasurably better (like Maven or Gradle).