Node: Proposal: gypkg for GYP dependency management

[indutny]

Hello @nodejs/build @nodejs/collaborators @nodejs/ctc !

I have been recently working on gypkg, which is something that I have craved for a very long time. What do you think about trying it out for Node.js core, and maybe libuv (cc @saghul) ?

This will allow us to put every GYP dependency to a separate repo, and manage them with a semver in a node.gyp file. Should be pretty neat, and should make the upgrade process easier and cleaner.

Thank you,
Fedor.

[indutny]

Oh, and also it will reduce future size of the Node.js repo tree, since we won't be adding way too much changes on each dep update anymore.

[eljefedelrodeodeljefe]

I like the work vey much, but as posted on twitter I am very strongly opposed to introducing another js-foreign dependency or something that increasing organizational dependency to one.

I think it's rather time to build our own tooling since we are big enough and can agree that most stuff out there isn't really what we are looking for out of the box and we are just piggy-backing on the Google stuff.

Since node is proliferated really much nowadays I think we would also do the software world a favor and build more cross platform tooling that anyone could use for other purposes too without having things like python installed correctly. Gyp seems orthogonal to that goal.

I am working on something wrapping shelling out to gcc, cl.exe and clang and to be honest that seems very easy to me and I don't even see the need for more advanced build system features. Everything else, like dependency management, change detection, target management, scripting can be done very easily if not much better with JS as opposed to declarative configs and declarative Makefile scripts.

[saghul]

Ohai! Here are my 2 cents, FWIW.

This project is nice in that it reduces bolireplate and makes it easier to use GYP for Node and other projects. Alas, it still requires Google's GYP, which doesn't change much of the status quo.

If we backtrack a little, it seems that the dependency on Python was one of the things people don't seem to like. Not to mention the fact that GYP doesn't work on Python 3, which adds more to the confusion. Google itself it transitioning to GN, which just generates Ninja buildfiles, and lets Ninja do the work. Personally, I think this a good approach.

My little contribution to help solve this problem was GYN a while ago. It's basically Google's GYP, with a single generator (Ninja) and ported to Python 3. I could compile Node itself. It was met with some resistance since it's still Python, which I do understand.

That said, I think a good way forward would be to port GYP to JS but make it just generate only Ninja files. We'd still be using GYP files, which Node is kinda heavily invested on it by now, and Ninja is super small and self-contained, so bundling it in shouldn't be much of a problem.

I briefly mentioned this last part to @eljefedelrodeodeljefe on Twitter. The idea being that Ninja could be replaced later, saving that stage and going directly from GYP files to binaries if need be. Personally, I'd take the Unix approach of one tool one job and let Ninja do the compilation.

One thing I wonder though, is that if this looks like a bigger problem than it is because we don't have a good story for binary packages. If we did then only the people who build the package would need the build tools, and (specially those on Windows) others wouldn't have much trouble.

[seishun]

This will allow us to put every GYP dependency to a separate repo, and manage them with a semver in a node.gyp file.

I don't think this is a good thing. Currently, Node.js repo is self-contained. This guarantees that even 10 years from now you can easily check out an ancient commit and build it. If it had dependencies in other repos, then it would depend on whether these repos still exist.

[jbergstroem]

I generally share @saghul's point of view. Adding on top: I now and then attempt to disconnect gn from the rest of the google stack which is perfectly possible but very hard in reality without hard forking; it's just moving too quick. It's relatively simple to build a standalone version of it; and I really like the idea of using gn-files/ninja as the "future" native compilation stack but I have a hard time seeing it happen if we stick with python.

[rvagg]

Some of this discussion can move to https://github.com/nodejs/CTC/issues/2 which is attempting to take a broader perspective on what to do with GYP in core and the ecosystem.

[kzc]

@saghul

I think a good way forward would be to port GYP to JS but make it just generate only Ninja files.

It's a chicken and egg problem. I think people should be able to build node without a node binary. So _not_ using JS to build is ironically preferable IMO.

[saghul]

@kzc We could use Duktape to bootstrap the process. It's a single C file, so maybe that would be acceptable? (just as a fallback) Or write the replacement in C++.

[kzc]

@saghul

We could use Duktape to bootstrap the process

Very interesting idea. The issue is that Duktape doesn't support most node standard modules like fs AFAIK, although it does have some Buffer compatibility. Could be confusing maintaining JS code that is not compatible with node.

[saghul]

Yes, it doesn't support any modules out of the box. It does have a Buffer implementation, though it has the 0.12 API. We really only need to read and write a file, so adding a couple of builtins should do. At any rate, I don't think the bootstraping is a big deal, comparing it with the rest.

[indutny]

FWIW, I'm working on JS port of GYP right now. Will keep you updated.

[eljefedelrodeodeljefe]

@kzc chicken-and-egg problem does not apply in this case. It's a common misconception. That is what cross compilation and downloading binaries is for. Go can build itself, so can gcc. We are no different.

[kzc]

@eljefedelrodeodeljefe Comparisons with Go don't apply as node can't be built with node alone. A C++ compiler is really the key prerequisite of building node. That's what builds v8, libuv, openssl and node itself. That's what I like about @saghul's proposal bootstrapping via Duktape - it eliminates the python dependency in favor of the C++ dependency that node can't avoid.

[eljefedelrodeodeljefe]

@kzc that's wrong. Any build tool needs a C/C++ compiler. In the end, no matter what tool you are using, you are calling out to your platforms compiler binary. So why using a dependency for making a call to gcc when you can do it with child_process?

[eljefedelrodeodeljefe]

@kzc fun fact: the reason why you want to use some toolchain is to avoid writing something like the below just to compile an addon.cc. But you can do it. Same goes for running make -j8 and copy pasting and output line when you compile node in order to mirror gyps (and any other build tool's behaviour)

  "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\amd64\CL.exe" /c /I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.10240.0\ucrt" /I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\atlmfc\include" /I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include" /I"C:\Users\jefe-win\.node-gyp\6.2.1\include\node" /I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include" /I"C:\Users\jefe-win\.node-gyp\6.2.1\src" /I"C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140" /I"C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0" /I"C:\Program Files (x86)\Windows Kits\8.1\Include\um" /I"C:\Program Files (x86)\Windows Kits\8.1\Include\shared" /I"C:\Program Files (x86)\Windows Kits\8.1\Include\winrt" /I"C:\Users\jefe-win\.node-gyp\6.2.1\deps\uv\include" /I"C:\Users\jefe-win\.node-gyp\6.2.1\deps\v8\include" /I..\node_modules\nan /Zi /nologo /W3 /WX- /Ox /Ob2 /Oi /Ot /Oy /GL /D NODE_GYP_MODULE_NAME=addon /D WIN32 /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _HAS_EXCEPTIONS=0 /D BUILDING_V8_SHARED=1 /D BUILDING_UV_SHARED=1 /D BUILDING_NODE_EXTENSION /D _WINDLL /GF /Gm- /MT /GS /Gy /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline /GR- /Fo"Release\obj\addon\\" /Fd"Release\obj\addon\vc140.pdb" /Gd /TP /wd4351 /wd4355 /wd4800 /wd4251 /errorReport:queue /MP ..\addon.cc

[kzc]

Any build tool needs a C/C++ compiler.

Great. Then we're in agreement on that point.

Using node to bootstrap node is a needless complication. It can be done, but it's not worth the trouble when other ways are simpler.

[eljefedelrodeodeljefe]

-.- I bail the conversation here.

[jbergstroem]

@kzc the relevant question here isn't bootstrapping node itself, rather being able to compile node native modules on a system where node is installed.

[kzc]

the relevant question here isn't bootstrapping node itself, rather being able to compile node native modules on a system where node is installed.

@jbergstroem Ah, okay. My mistake.

But perhaps this JS port of GYP initiative could have applications to bootstrapping node itself if successful.

[indutny]

Port of GYP (WIP): https://github.com/indutny/gyp.js

[Trott]

This issue has been inactive for sufficiently long that it seems like perhaps it should be closed. Feel free to re-open (or leave a comment requesting that it be re-opened) if you disagree. I'm just tidying up and not acting on a super-strong opinion or anything like that.