Nixpkgs: yarn2nix

[moretea]

Facebook recently published yarn, a package manager that behaves like Bundler and Cargo by having a lock file that describes the exact versions of all transitive dependencies.

I investigated the yarn.lock format, and was pleasantly surprised that it actually contains both direct download links to (and SHA1 hashes of) the .tgz files that make up packages.

I'm very much open to merge this with @svanderburg's node2nix, but I'm not sure that that's appropriate.

Progress

• [x] Get yarn into nixpkgs (https://github.com/NixOS/nixpkgs/pull/20635)
• [ ] Finish implementing yarn2nix

[offlinehacker]

Nice :)

[offlinehacker]

What happens if package.json refrences git/github package?

[phanimahesh]

Would you mind if yarn2nix was a go/rust binary instead of a ruby script? Or a nodejs script, since we are anyway working with the node ecosystem. Having to pull in ruby to work on node just doesn't feel good. I can volunteer for part of the effort. Nice excuse to try out something new too, IMO.

[zimbatm]

I think it would be more maintainable to use yarn's parser directly. Here I made a small program that turns the yarn.lock into JSON: https://github.com/numtide/yarnlock2json . The URL still has to be split to extract the sha from the fragment.

[wmertens]

@moretea do you need help with this? This finally seems like a long-term viable approach… Use Yarn to get a fixed set of deps, create the deps, and let yarn do the node_modules build.

I agree with @zimbatm that we should use yarn's own parser. I had a look at the yarn2nix and it seems very simple, so perhaps it can be built directly on top off zimbatm's script?

Also, perhaps each yarn package should be a separate nix expression, so that they get reused between projects? I think it would mean walking the dep tree an building each dep in turn, creating packages as you go, and linking them into the .cache on next invocations.

[zimbatm]

Ideally we want the source to be fetched by nix but it's not clear to me how to plug that into yarn yet. Maybe we can reproduce yarn's file layout and ignore yarn install but I suspect there are some hidden complexities there. Another solution would be to work with yarn to see if we can pass them the list of source derivations.

Regarding the node_modules folder, it's not practical to use symlinks in there unfortunately. If you search for it you'll see that others are trying to tackle the issue but in short it's due to two things: (1) some modules actually rely on the node_modules structure by doing things like require("../other_module"). (2) node resolves module paths to their absolute path at import. So now you have a module at /nix/store/...-my-module trying to load ../other_module and it breaks. What might work though is have the whole node_modules stored in a derivation so it only gets re-built when the package.json or yarn.lock file changes.

[wmertens]

>

Ideally we want the source to be fetched by nix but it's not clear to me
how to plug that into yarn yet. Maybe we can reproduce yarn's file layout
and ignore yarn install but I suspect there are some hidden complexities
there.

I was thinking that we can reproduce the cache dir layout. After all, we
have the full source available.

Another solution would be to work with yarn to see if we can pass them the
list of source derivations.

Actually that might work too, maybe with some sort of plugin system if we
can't come up with something that upstream would want to support. What
would that look like, ideally?

Regarding the node_modules folder, it's not practical to use symlinks in
there unfortunately.

I was not referring to that; AFAICT, Yarn uses its cache dir to store
tarballs and build products. These build products are then copied into
node_modules, so most likely the build product folders can be symlinks.

So the node_modules folder is a bunch of copies, but when that built
application is stored in the nix store, it can be deduped if the optimize
option is turned on. So that's not a problem.

[zimbatm]

Alright that makes sense. I have a notion that this same problem is quite
general to all package managers. The difficulty for nix is that we don't
want to deal with dependency resolution, only work over a resolved set. We
want to fetch the dependencies but not deal with installing them. But
provide library dependencies for C extensions. And finally link the result
so the package can use them.

So that means nix and $lang package manager are quite intertwined. If we
can figure out the right interfaces for each of these steps, it will be
much easier to communicate with $lang and ask them to insert the proper
hooks where we need them.

For now my best idea is that $lang provides a lockfile that contains the
dependencies, where to fetch then including the SHA. $lang developer
shouldn't have to commit the generated nix code. Nix knows how to read that
lockfile and prefetch the dependencies from that. After that it's a bit
fuzzy and probably varies. Ideally we can build each dependency in
isolation but it's not always possible.

If there is a package repository like rubygems or npn we should push for
having the SHA included to make things faster. The SHA must be calculated
the same way as nix though.

Hmm that's it for now. Not really helping this issue in particular.

On Fri, 16 Dec 2016, 20:36 Wout Mertens, [e-mail] wrote:

>

Ideally we want the source to be fetched by nix but it's not clear to me
how to plug that into yarn yet. Maybe we can reproduce yarn's file layout
and ignore yarn install but I suspect there are some hidden complexities
there.

I was thinking that we can reproduce the cache dir layout. After all, we
have the full source available.

Another solution would be to work with yarn to see if we can pass them
the
list of source derivations.

Actually that might work too, maybe with some sort of plugin system if we
can't come up with something that upstream would want to support. What
would that look like, ideally?

Regarding the node_modules folder, it's not practical to use symlinks in
there unfortunately.

I was not referring to that; AFAICT, Yarn uses its cache dir to store
tarballs and build products. These build products are then copied into
node_modules, so most likely the build product folders can be symlinks.

So the node_modules folder is a bunch of copies, but when that built
application is stored in the nix store, it can be deduped if the optimize
option is turned on. So that's not a problem.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/NixOS/nixpkgs/issues/20637#issuecomment-267690124,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAMsNQzJh7dXxWHjTU2aEtlAIQiRCYkks5rIvZXgaJpZM4K6DHj
.

[moretea]

@phanimahesh Would you mind if yarn2nix was a go/rust binary instead of a ruby script?

Not at all. It's just a prototype at the moment and I just like to use Ruby to prototype these kinds of things.

@zimbatm I think it would be more maintainable to use yarn's parser directly.

Awesome. I actually know little about javascript; I just want to package some stuff.

@zimbatm Ideally we want the source to be fetched by nix but it's not clear to me how to plug that into yarn yet. Maybe we can reproduce yarn's file layout and ignore yarn install but I suspect there are some hidden complexities there.

@wmertens I was thinking that we can reproduce the cache dir layout. After all, we
have the full source available.

I tried to do exactly that, but yarn still insists on writing random files in that cache directory. When I tried to fill it with the correct packages, it still failed because of missing packages.

@wmertens So the node_modules folder is a bunch of copies, but when that built
application is stored in the nix store, it can be deduped if the optimize
option is turned on. So that's not a problem.

It's a bit more subtile than that. The directory structure of node_modules is a bit weird; npm/yarn support having multiple versions of different packages in there. This in itself is not a weird thing to do; Cargo supports the same features.
The problem is that the directory names do not have a version number. Having multiple dependency versions is handled by having another node_modules directory in a $ROOT_DIR/node_modules/some_package/ directory. Unfortunately, I could not find any documentation on the exact logic of how these decisions are made.

@zimbatm For now my best idea is that $lang provides a lockfile that contains the
dependencies, where to fetch then including the SHA. $lang developer
shouldn't have to commit the generated nix code. Nix knows how to read that
lockfile and prefetch the dependencies from that. After that it's a bit
fuzzy and probably varies. Ideally we can build each dependency in
isolation but it's not always possible.

Fully agreed. Luckily a yarn.lock file contains all that!

---

I would very much appreciate any help with this project ;)

[zimbatm]

It's good to see that we are all pretty much on the same page even if the
details are still fuzzy. I might have some time allocated over the holiday
period, I will know next Tuesday.

On Fri, 16 Dec 2016, 22:23 Maarten Hoogendoorn, [e-mail]
wrote:

@phanimahesh https://github.com/phanimahesh Would you mind if yarn2nix
was a go/rust binary instead of a ruby script?

Not at all. It's just a prototype at the moment and I just like to use
Ruby to prototype these kinds of things.

@zimbatm https://github.com/zimbatm I think it would be more
maintainable to use yarn's parser directly.

Awesome. I actually know little about javascript; I just want to package
some stuff.

@zimbatm https://github.com/zimbatm Ideally we want the source to be
fetched by nix but it's not clear to me how to plug that into yarn yet.
Maybe we can reproduce yarn's file layout and ignore yarn install but I
suspect there are some hidden complexities there.

@wmertens https://github.com/wmertens I was thinking that we can
reproduce the cache dir layout. After all, we

have the full source available.

I tried to do exactly that, but yarn still insists on writing random files
in that cache directory. When I tried to fill it with the correct packages,
it still failed because of missing packages.

@wmertens https://github.com/wmertens So the node_modules folder is a
bunch of copies, but when that built

application is stored in the nix store, it can be deduped if the optimize
option is turned on. So that's not a problem.

It's a bit more subtile than that. The directory structure of node_modules
is a bit weird; npm/yarn support having multiple versions of different
packages in there. This in itself is not a weird thing to do; Cargo
supports the same features.
The problem is that the directory names do not have a version number.
Having multiple dependency versions is handled by having another
node_modules directory in a $ROOT_DIR/node_modules/some_package/
directory. Unfortunately, I could not find any documentation on the exact
logic of how these decisions are made.

@zimbatm https://github.com/zimbatm For now my best idea is that $lang
provides a lockfile that contains the

dependencies, where to fetch then including the SHA. $lang developer
shouldn't have to commit the generated nix code. Nix knows how to read that
lockfile and prefetch the dependencies from that. After that it's a bit
fuzzy and probably varies. Ideally we can build each dependency in
isolation but it's not always possible.

Fully agreed. Luckily a yarn.lock file contains all that!

I would very much appreciate any help with this project ;)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/NixOS/nixpkgs/issues/20637#issuecomment-267710581,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAMsEHQWABoIju_cUb-C0BOMxQmidxUks5rIw9hgaJpZM4K6DHj
.

[moretea]

I started to patch node2nix to support importing a yarn.lock file in https://github.com/svanderburg/node2nix/pull/28.

It was actually a lot less work than I anticipated.

I'm closing this issue and proposing that we move the discussion to that PR.

[wmertens]

Hmm, the approach of node2nix is entirely different: basically it's yarn
for Nixpkgs, building node_modules itself.

Looking at the huge list of issues yarn has had, it seems like that is very
hard to do correctly.

i think the approach of running yarn in offline mode is much more
maintainable.

On Sat, Dec 17, 2016, 4:30 PM Maarten Hoogendoorn [e-mail]
wrote:

I started to patch node2nix to support importing a yarn.lock file in
svanderburg/node2nix#28 https://github.com/svanderburg/node2nix/pull/28.

I'm closing this issue and proposing that we move the discussion to that
PR.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/NixOS/nixpkgs/issues/20637#issuecomment-267768726,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADWlofvaFJfPOuPS91etznaQKDk2kWkks5rJAArgaJpZM4K6DHj
.

[moretea]

Maybe so, but my painpoints were in getting yarn to use the files downloaded with fetchurl, which are all solved by node2nix already. It is something we can explore in parallel to only depending on yarn.

[phanimahesh]

Agreed. Yarn is a bunch of good ideas, implemented extremely poorly. We shouldn't trust yarn to do things right. If it can be convinced to use pre-downloaded files, that's what we should be doing. If it isn't supported, we should request/contribute that.

[moretea]

I've rewritten yarn2nix. It is able to build packages with dependencies that are in the default registry properly.

I'm abusing the offline cache feature of yarn by generating a directory with $package-$version.tgz files downloaded via Nix.

@zimbatm @offlinehacker @wmertens @phanimahesh could you take a look? I gathered that you are packaging node apps.

[zimbatm]

Nice! Looking good

[offlinehacker]

Nice, I really like the solution as it's simple, now all we need is buildYarnPackage function in nixpkgs :)

[ixmatus]

Integrating yarn2nix into nixpkgs doesn't look all that difficult; I've already integrated it into our own package overrides. Does someone already have work in a branch to do this? If not, I'd love it if we could get a PR open to contribute yarn2nix and a separate buildYarnPackage function...

[EDIT] I'm happy to help :)

[zimbatm]

@ixmatus it's not that hard, the main reason not to merge was to be able to experiment without polluting the mainline. the project is in a pretty good state but still needs a bit of polish.

[flokli]

@zimbatm Whats the current state of this? moretea/yarn2nix seems to have received some commits so far, but there also is Profpatsch/yarn2nix already coming in from hackage-packages…

[moretea]

@flokli @Profpatsch's approach looks way better than the hacks that we do in this repository. If I would need to package a yarn thing again, I would probably take a look at his solution first.

If you can use @Profpatsch's version, I'd go for that :)

[Profpatsch]

Please write issues if it takes you too long to get https://github.com/Profpatsch/yarn2nix running.
I’m using it for a few of my projects but haven’t yet written down how to use the nix library that comes with the repository.

I’m more than happy to help with packaging, to get feedback on whether the interface is any good and whether it actually works for all/most use cases. Sadly, the whole npm ecosystem is a bit of a mess and there’s lots and lots of edge cases everywhere you look.

[marsam]

With https://github.com/NixOS/nixpkgs/pull/35340 merged, I think we can close this issue

[zimbatm]

Unfortunately yarn2nix depends on Import From Derivation (IDF) to work, and it's not enabled on ofborg and hydra. yarn2nix can be used from nixpkgs but other packages in nixpkgs can't use it.

See https://github.com/NixOS/nixpkgs/pull/35334

[domenkozar]

I wonder if we could add yarn2nix stuff to nixpkgs but disallow hydra builds? As a stepping stone.

[matthewbauer]

Can't yarn2nix be modified to generate its .nix expressions like node2nix, emacs2nix & cabal2nix? That is much preferred than disabling hydra. I think that is the right way to do things in Nixpkgs & it has worked very well for cabal2nix at least.

[domenkozar]

I think we should remove yarn2nix from nixpkgs, as it's quite outdated (or merge all the upstream changes to nixpkgs).

What do you think @zimbatm @moretea

[zimbatm]

agreed

[jbmusso]

I realize that this has been removed from nixpkgs. However, would it make sense to use https://github.com/Profpatsch/yarn2nix by @Profpatsch instead?

I'm getting started with nix and happen to be a bit confused by the two yarn2nix packages. I'm quite familiar with node/npm/yarn though.

[domenkozar]

Hey @jbmusso - I think that project is less complete the last time I tested things.

I'd recommend using yarn2nix by moretea. Maybe @Profpatsch is willing to rename his project to avoid confusion?

[jbmusso]

Thanks for the recommendation, I'll experiment with moretea's package then. Since it's mostly JS, I may be able to patch things there if required - can't do that in Haskell yet.

Unsure about what Profpatsch's package could be, maybe yarn2nix-hl.

Also, thanks for building these tools.