Yarn and Node

[arcanis]

This is related to #15244 and #35682, but this particular thread is focused on one specific question: should Yarn be distributed with Node.js?

There are various points typically raised against this. Because it's an hot subject with many Opinions™, I'd like to do something a bit different. I'll open multiple sub-threads in this discussion, and offer a starting opinion on them. Feel free to discuss them, but please keep discussions scoped to each sub-thread (ie don't collate them in a single post). You can create new ones if you think I forgot a particular point.

[arcanis]

"Neither Yarn nor npm should have been included by default"

I'd tend to personally agree with this assessment, but it unfortunately ceased to be relevant years ago. Npm is part of too many workflows, and we can't reasonably remove it from the classic distributions. The question isn't so much how to remove npm, but rather how to live with it in a way that's satisfying for all parties.

[mcollina]

I disagree. Bundling npm provides a great user experience, and possibly the most critical factor in Node.js success.

[Abhishek12345679]

yarn is just way faster than npm.

[arcanis]

"This will be solved by #35682, there's no need to discuss this further"

While true, even once #35682 is implemented, it'll only affect people changing their workflows to use this new distribution strategy. In other words, it'll take a lot of time for the whole ecosystem to converge and, in the meantime, Node.js developers using Yarn will still be second-class citizen.

Additionally, if Yarn is considered for being a builtin inclusion in the context of #35682 (and it's been mentioned a few times that it was), then it would make sense to bring it to the binary distributions as well, if only for feature parity.

[bnb]

Additionally, if Yarn is considered for being a builtin inclusion in the context of #35682 (and it's been mentioned a few times that it was), then it would make sense to bring it to the binary distributions as well, if only for feature parity.

@jasnell my understanding is that your goal with corepack is to entirely remove pacakge managers from being checked into core, yet @arcanis is asserting here (and here) that we should actually check in the package managers Corepack will be including. Is this assertion in line with your goal of removing package managers from being checked in at all? Or have you changed your position on that?

[arcanis]

I'm afraid you misunderstood what I said. The line you're referencing has nothing to do with Corepack or Node, and is just part of the regular Yarn 2 install instructions. I'm sure reading them (or better, running them) would give you a better idea of what I mean.

As for checking-in Yarn in Node (ie pr #37277), let me summarize the situation:

• The discussion in Node.js Binary Manager Summit? #35682 is about a new type of distribution for Node which would provide both Node and its extra tools via lazy fetching, letting users pin versions as they need without size worry, or having to check them inside the Node core. Something like Volta.

• However, the discussion in Node.js Binary Manager Summit? #35682 is about a new type of distribution. My understanding from the discussions we had doesn't cover the existing Node binary releases (ie node-v14.15.5-linux-x64.tar.xz).

• Given that, according to the discussions we had in Node.js Binary Manager Summit? #35682, in particular with @MylesBorins, Yarn would be candidate for being integrated to the "extra tools" offering (and thus would be available globally under this distribution strategy), this new discussion in this thread is about the pertinence of bringing parity between the binary releases and the "extra tools" offering.

In other words, removing package managers from the core is the goal of #35682. But until the tools described in #35682 come to fruition (and are widely adopted), there's still a problem in need of a solution (the same problems that initially made Yarn worth including into the "extra tools", so I'm not going to detail them again). PR #37277 thus aims to first solve these problems by following Node's existing infrastructure.

Also note that, as per the jumper mechanism I described in #37277 (comment), Yarn 2 won't be upgraded by upgrading the binary in Node. Instead, users will be expected to simply run yarn set version 2.x.y. As a result, the maintenance cost for Yarn 2+ will be almost inexistent.

Finally, contrary to npm, the deps/yarn folder contains exactly 10 files (2945 for npm). Should we need to make a change (for example if we were to release a security fix on 1.x), the impact on the Node repository would be minimal.

[arcanis]

"Yarn can be installed manually, there's no point distributing it with Node."

This might be true for you, but not for novices. Just like many people weren't aware that npm Inc. was a for-profit company, many people still aren't aware that npm is a third-party project that the Node.js org doesn't actually control. Because of this misconception, they don't look elsewhere, and are often deprived of the choice without even knowing it.

Similar situations happened multiple times in the past, the most obvious parallel being BrowserChoice.eu. Let me share this snippet from the press release the European Commission released at the time:

The Commission's preliminary view was that competition was distorted by Microsoft tying Internet Explorer to Windows. This was because it offered Microsoft an artificial distribution advantage not related to the merits of its product on more than 90 per cent of personal computers. Furthermore, the Commission's preliminary view was that this tying hindered innovation in the market and created artificial incentives for software developers and content providers to design their products or web sites primarily for Internet Explorer.

[arcanis]

"It would break tools that check Yarn's existence to know what package manager to use."

This argument doesn't really hold for many reasons:

• The main one is that this argument somehow assumes that, given the choice, such tools would choose npm. This is, by definition, exactly the opposite: if given the choice, their current behavior is to use Yarn. Yes, it would change their default, but to the default they seem to prefer.

• Once again by definition, tools that would follow such a logic (use Yarn if available, npm otherwise) are necessarily fine with being used with Yarn - otherwise they wouldn't pick it at all.

• Projects define which package manager they need to be used with, users don't. As such, checking binary existence isn't a good signal for user preference, as many users already use Yarn for at least a few projects, even if they prefer npm. A better signal is to check for $npm_user_agent, which allows to know the package manager used to run the script (when run via npx or yarn dlx).

• No project does this in a production setup. The only tools which rely on such logic are developer tools, rarely used except for one-shot commands (typically project templates).

[arcanis]

"That could pollute the global binaries."

Npm seems to have been generally fine with adding binaries - first npx, then lately by considering adding a new npr utility in one of their latest RFC. Yarn is a well-known binary, and exposing it to the PATH is unlikely to be a problem.

[arcanis]

"That would increase the Node distribution."

Npm is 31MB. The smallest Yarn distribution is 1.8MB. If size truly is a concern, we just have to ask npm to remove the markdown files to literally free 5MB and offset more than enough the inclusion of Yarn.

[arcanis]

"I don't use Yarn"

And that's fine, but you're a fairly small data point. Many people do. While we don't have a lot of data, Yarn has shown to be a consistent 30-40% of all polls on Twitter (and trust me, there are a lot of them ...). Yarn 2 itself recently polled around 14%. Thousands of developers use Yarn every day.

[bnb]

but you're a fairly small data point.

A majority is categorically not a fairly small data point?

[jasnell]

Pretty certain the intent is to direct the answer towards an individual and not a collective.

[arcanis]

"Npm is all we need now that they have workspaces"

Both projects are different in many aspects, only some technicals, and the different tradeoffs have value regardless of the exact featureset (and I personally disagree with the feature parity in general, but it's not relevant to this discussion).

[arcanis]

"What about pnpm?"

I would tend to support pnpm's inclusion as well, especially since they now support being distributed as a single file, just like Yarn itself.

[arcanis]

"Which Node.js version should start including Yarn?"

I would tend to think that Node 15 would be the best, given it's an odd release, but other contributors would have more experience with this kind of work.

[jasnell]

At this point, I'd suggest opening a PR that adds it and let's see how far we get.

[mcollina]

I agree with @jasnell.

[mcollina]

Yarn 1 vs Yarn 2

The distinction between Yarn 1 and 2 is really confusing in the documentation: https://yarnpkg.com/getting-started/install.

npm i yarn installs yarn@1, however it's considered "legacy" - I would rather not add a new dependency to Node.js that is already legacy and possibly discontinued in the future.

Note that they live in two different repos: https://github.com/yarnpkg/berry vs https://github.com/yarnpkg/yarn.

From some early checks, it seems that yarn is downloaded 1.9 millions per week, while berry is at 70 thousands.

What should we ship?

[arcanis]

Short answer: I think everyone will benefit from having Yarn 1 in global - both 1.x and 2.x users. This is what already happens, and this is in line with how our team intends to distribute Yarn 2+ in the future, whether Node includes Yarn or not.

npm i yarn installs yarn@1, however it's considered "legacy"

It's not legacy (we don't use that term anywhere on the website). We call it "Classic", to acknowledge that it's still a version that various people want to use and can use.

Generally, we still commit to fix security issues on 1.x should the need arise, but apart from that we expect it to stay mostly the same, good and bad ("feature complete", if you will). I think over time more and more projects will migrate (and our stats show they do), but we don't intend into rush this process.

Of course, if Yarn is shipped with Node, we'll also commit to keep it compatible with Node and fix deprecations.

From some early checks, it seems that yarn is downloaded 1.9 millions per week, while berry is at 70 thousands.

Those numbers are likely incorrect for both versions. Package managers typically don't get recognized as dependencies, and their download count is necessarily low since you only download them once. For Yarn 2 in particular, since we recommend to install per-project and to check-in within the repo, the download count for the binary is expected to be very low (basically a single download for each project that uses it, even counting CI).

[arcanis]

One other option could be to simply ship #35398, which is a PR ready to merge and that received a reasonable traction.

It would however be a bit more complex than just adding Yarn though (since it involves network, user configuration, etc), which is why I believe embedding Yarn 1 would be the right incremental choice until we get a clearer picture on #35682, which is intended to solve those questions. In any case, the public behavior will be the same: Yarn 1 global, Yarn 2 per-project.

[mcollina]

Thanks, this is a really good explanation!

[gitawego]

yarn 2 is the reason why we switched back to npm, because the v2 changes completely how things work, and breaks a lot of things . It seems v1 is in maintenance mode since then.

We need a package manager works and well maintained just like npm.

Why do we include a package manager that is almost at the end of the life?

[arcanis]

I'm not sure what makes you think it's the end of its life. Software lifecycles have a notion of "feature complete", and Yarn 1 just happened to have reached this state. Many people keep using Yarn 1, and don't plan to change, and this in itself is a proof that it's a perfectly fine software for its use case.

I understand that in the age of software obsolescence it may sound like a novel stance, but there is such a thing as "it doesn't get new features, but it's what I need".

[Trott]

What impact would it have on registry hosting?

The issue of what binary to ship is inseparable (to me, at least) from the issue of who runs the registries. The bundling of npm and the centrality of the npm registry bring both benefits and downsides/risks to the community. The questions there are (IMO) more significant than the general DX arguments about shipping another client.

• Who is willing and able to run a registry capable of supporting the Node.js ecosystem?
• What would motivate someone not doing it now to start doing it?
• What might motivate the only entity doing it now to stop doing it?
• What relationship does client bundling have on the above questions?
  • Would Node.js shipping a binary npm doesn't control possibly complicate the issue of maintaining the registry for npm?
  • Would it otherwise reduce the motivation of npm to host the registry? If so, what impact does that have on the registry going forward?
  • Would multiple binaries complicate the possibility of standing up competing registries?
  • Are multiple competing registries a good thing?
  • Can a community entity maintain a registry? (Other languages have registries that are community run, but even the biggest of them get vastly less usage than the npm registry. npm spends vast sums to maintain the registry.)
  • What are the benefits and downsides of having npm run the registry, and how does shipping another client change any of that?

I think one reason the client bundling issue never gets anywhere is because it attracts a lot of attention/opinions around issues that are relatively easy to talk about, like whether it's worth it to add another 1.8 Mb to the download size of a Node.js installation. But the most important question (to me, at least) is what the long-term impact on registry hosting is likely to be. This question is more existential (due to the centrality of the npm registry to the success of Node.js), probably at least somewhat unknowable, and very... I don't know... loaded? It's a sensitive issue for various reasons.

A related question, then, is: Given that much of this critical issue is probably unknowable (and hard to talk about), is Node.js willing to make a change anyway? Or does the uncertainty mean we stick with the status quo because of the risk of damage to something that's working?

[arcanis]

I don't expect it to have any impact. I don't think Yarn caused any change in the registry that they didn't intend to make anyway, and I don't see why that would change.

The bundling of npm and the centrality of the npm registry bring both benefits and downsides/risks to the community. And those things are (IMO) more significant than the general DX arguments about shipping another client.

It's an interesting discussion in itself but, as you say, it's something far more significant than the discussion in this present thread, and only tangentially related (whether Node should provide Yarn doesn't have prerequisites on changing anything on the registry), which I think makes it out of scope for the purpose of this discussion.

Said another way, clients and registries only happen to share a common protocol, and I doubt anyone really has any intention to rework it significantly (if only because of backward compatibility). The discussion you seem to want sounds more about the registry governance, which isn't something Yarn (nor pnpm I suspect) have anything to contribute about at this time.

[devsnek]

As long as we (node) want to spend time on this, perhaps it would be worth evaluating if giving money to something like entropic development would be productive.

[bnb]

The TSC has voted against including Yarn in Node.js, in favor of a corepack-like solution.

[bnb]

Should this be marked as answered?

[mcollina]

I think so, yes!