Gitea: Vendoring JS modules

[silverwind]

The go modules are currently downloaded and copied to vendor, which allows offline building. As https://github.com/go-gitea/gitea/pull/9114 stands, this will not be possible for the JS modules which are pulled from npm during the build.

If we want to have offline building of frontend assets, I could see the following solutions:

• vendor the full node_modules unaltered which is 20929 files at 115MB.
• use yarn PNP mode and vendor the resulting files as well as yarn itself which is 920 files at 25MB (individual modules are stored zipped in this case).
• come up with a custom solution that compresses node_modules into a single file and uncompresses it before the build.

[techknowlogick]

Im leaning towards option 2

[silverwind]

I too would prefer with option 2 because it will at least give a somewhat readable diff when the modules are being updated and we would no longer depend on npm whose development has gone a bit stale recently.

[guillep2k]

I don't know yarn but I imagine that it will update only the individual zips that need updating, so it's more granular. If that's the case, I think 2nd option is a clear winner. But we need to lock down all the libraries. Otherwise they will change "randomly" if we rebuild the project and we're back to square one.

[silverwind]

Yes, during an update only package.json, the lockfile and the zips will change.

During my testing, I noticed that 5 modules were stored "unplugged", meaning they were not in a zip but in its own directory uncompressed. From what I gather, this is the case when a module can for various reasons not fully work when "plugged" so yarn deoptimizes it. I imagine these cases will eventually be fixed by yarn and/or the modules itself.

Should I include the changes for offline JS build in https://github.com/go-gitea/gitea/pull/9114? I'm thinking it's probably better to follow up in its own PR.

[sapk]

I would be against vendoring js deps because there are proxy that store the versionned so it limit the risk for deps to disappear (which is the same for go now and maybe we could drop vendoring go mod now when go1.14 is out).
On drone we can cache the node_modules folder.
On an offline build, before when fetching the git repo (needing internet) someone would need to fetch the deps also.

This is just my point of view, if others think we should vendor I am totally ok with it.

[lunny]

@sapk So maybe we could have a command on Makefile like make fetch_deps to download all the dependencies.

[silverwind]

I would be against vendoring js deps because there are proxy that store the versionned so it limit the risk for deps to disappear

Not sure exactly what your concern is, but npm modules can not go away. Published version can only be unpublished for a very limited time, after that a module version is practically guaranteed to ever exist on npm (at least until npm shuts down).

So maybe we could have a command on Makefile like make fetch_deps to download all the dependencies.

Agree, once we un-vendor everything including go, such a make target should be there so it can be called either during the build or manually.

[sapk]

Not sure exactly what your concern is, but npm modules can not go away

Yes we are on the same page that why I would say that vendoring them in the repo is not needed.

Using the term proxy was maybe not the best term to use for js. Sorry. (I had in mind the goproxy)

[silverwind]

My point for vendoring is that I could certainly see someone having a up to date checkout but no or restricted Internet access when building. So at minimum, we should document the online requirement for the build.

With GOPROXY now being the default and the caching it brings, I don't see vendoring go anymore as a strict requirement to protect against source going away.

I would either prefer vendoring everything (go and js) or nothing, to be consistent.

[sapk]

Maybe it is time for a vote.

• :+1: Don't vendor any
• :-1: Keep vendoring Go only
• :smile: Vendor only JS
• :tada: Vendor both JS and Go

[guillep2k]

I couldn't find it, but there was a comment not long ago asking us to vendor everything so some linux/docker distributions could use a "closed package" for building, as they want the build to be 100% predictable.

That would be my only reason to vote 🎉. Otherwise, I'd go 👎 because I think there are too many repositories for our golang packages to rely on. We could use mirrors in gitea.com for some of them, but that wouldn't be very nice.

[techknowlogick]

I voted the same as @guillep2k, for the same reasons. And if we vendor JS (in addition to Go), then we can remove the pre-built CSS/JS files from source.

[sapk]

I couldn't find it, but there was a comment not long ago asking us to vendor everything so some linux/docker distributions could use a "closed package" for building, as they want the build to be 100% predictable.

@guillep2k It must be https://github.com/go-gitea/gitea/issues/9174, We could still provide a tar.gz with JS (the node_modules folder) and Go deps (the vendor forlder after a go mod vendor) for release on https://dl.gitea.io/. That what they basicly did previously by defining the GOPATH to a folder and compressing this folder for later use.

then we can remove the pre-built CSS/JS files from source.

@techknowlogick They are here not for the same reason. The goal was to not need nodejs for building gitea (when not touching js part). But I am ok with removing them since I think most people have node (or a docker version) installed on their dev machine.

[guillep2k]

@guillep2k It must be #9174,

It was that issue indeed.

[silverwind]

I went with 👍 because I do see vendored copies as redundant. Offline build is not important to me personally, otherwise I'd go 🎉.

[sapk]

I think we all agree that we need to vendor or not vendor both.
To not take more time and move forward, I think we can put a deadline for the vote in 2 days (7days in total) and wait for the vote of last owner that hasn't vote. (Gentle ping to @lafriks)

Vote comment: https://github.com/go-gitea/gitea/issues/9170#issuecomment-559480784

[lunny]

This is a choice between keeping compiling consistent and don't store too many files on git.

About code size:

Gitea: 316MB
vendor: 56MB
node_modules: 110MB

If we vendor all it will be over 430 MB, if we remove all vendors, it will be 260MB.

[lunny]

If git/gitea support a merge style to allow overwrite somefiles anytime(index.js/index.js.map), then I think keeping the current structure.(Only vendor go) maybe a better choice.

[silverwind]

merge style to allow overwrite somefiles anytime(index.js/index.js.map)

I thought removal of js/css bundles was already somewhat decided. They will be gone with https://github.com/go-gitea/gitea/pull/9114. Of course, we could introduce them later if a resolution for the constant merge conflicts is found.

[lafriks]

Problem with keeping zip files in git repository is that it will grow quite considerably as all zip file versions will be kept in history as they are binary and even if small subset of files change in node_modules new zip file copy will be in git history.

[techknowlogick]

Just changed my vote to not vendor anything, it will add slight increase to fetch required libraries for golang, but we already fetch required libs for npm/node in CI, so not much extra time.

The ticket we got about go not being vendord for the linux flavour (which is false as we do vendor go), I will propose an alternative for them in that debian actually requires each vendor lib to be a package in itself (I know that was a pain point for getting gitea into official debian repo, which the project was never able to do because we have many dependencies).

[silverwind]

One note regarding the yarn zip vendoring approach I mentioned earlier: It will require the beta version of yarn (https://github.com/yarnpkg/berry) and it seems the PNP mode is rather unstable as it stands, so I don't really consider it a valid option (yet).

[lunny]

If we don't vendor anything, we should at least add some cache for our CI, otherwise it may spend more time than currently. Or we may need ourself goproxy/npm mirror to protect external service down.
A goproxy will use goproxy.io/goproxy.cn as a upstream, npm mirror could use npmjs.org as upstream.

PNP seems not very stable currently. I think we should use stable technology as our dependencies.

[silverwind]

Voting deadline has ended, let's continue in https://github.com/go-gitea/gitea/issues/9283.