Packages: [RFC] Decision on preference (or not) for the use of codeload

[danielfdickinson]

@neheb Has submitted a number of PR's (referenced in https://github.com/openwrt/packages/issues/6584), full list of his/her PR's (which probably include the codeload thing for any github pulls): https://github.com/openwrt/packages/pulls/neheb

@diizzyy has also mentioned codeload. [EDIT: I misunderstood karlp's comment to be related, it's not] @karlp appears to feel the change is for marginal gain (based on https://github.com/openwrt/packages/issues/6584#issuecomment-410452508).

I personally think that tree-wide changes, especially for something @yousong has been trying solve another way ought t be clearly discussed and not just brought in through the back door, hence this ticket and I'll be following up on openwrt-devel as well.

[neheb]

Just to clarify, I do it for GitHub projects that lack a release tarball.

[Andy2244]

I also only use it for medium, large sized github packages, without release tarballs. see #6577

PS: I'm fine either way, the git clone + local tarball creation also worked fine for me in the past.

[karlp]

Daniel Dickinson [e-mail] wrote:

@neheb Has submitted a number of PR's (referenced in
https://github.com/openwrt/packages/issues/6584), full list of
his/her PR's (which probably include the codeload thing for any
github pulls): https://github.com/openwrt/packages/pulls/neheb

@diizzyy has also mentioned codeload. @karlp appears to feel
the change is for marginal gain (based on
https://github.com/openwrt/packages/issues/6584#issuecomment-410452508).

That's out of context. I have no objection to the use of a documented github apis.

Cheers,
Karl P

[danielfdickinson]

@karlp apologies then, I misunderstood (the possibility I had is why I tagged you so you could comment)

[danielfdickinson]

Please note it is preferred if the responses could be made as followups to http://lists.infradead.org/pipermail/openwrt-devel/2018-August/013570.html (followups to those followups) on OpenWrt-devel Mailing List

[diizzyy]

@cshoredaniel
I'm not sure why you insist on having it on the mailing list since it's not related to OpenWrt distribution itself. GitHub has always been where package related discussions occur so it's the preferred forum.

Anyhow, a few advantages

• Reduces data usage considerably in a lot of cases compared to git clone
• Faster
• Better availability as other distributions also use codeload
• Probably not an issue itself but I would imagine that codeload is less troublesome then git over https in some corp environments

As @neheb mentioned, it should only be used when there's no official tarball available.

[micmac1]

Hi all,

If codeload is prone to changing checksums (related to this description possibly: openbsd-ports discussion) then I'd stay clear of it, because that's obviously not very advantageous. When seeing a checksum test fail usually a red light goes on in your head.

On another note, if using a git checkout and adding PKG_MIRROR_HASH then the users normally fetch a tarball from the openwrt mirror. So really they don't need to clone a git repo. I like that.

Kind regards,
Seb

[danielfdickinson]

@diizzyy The infrastracture and tooling of packages, which is shared amongst core and packages is also a core matter in my view. In particular with codeload and the like the packages feed shouldn't be doing things substantially different from core, in my view. Also I feel like the mailing list gets more project eyeballs than a random issue on GitHub, and this is an issue that I think should be discussed project-wide before implementing.

[diizzyy]

I honestly think that's up for the people actually doing work in the repo but that's just me....

[danielfdickinson]

Also my single biggest issue that it's a tree-wide change without prior discussion. I don't really have deep feelings about the actual decision, I just think a discussion should actually happen.

[neheb]

@diizzyy has a point here. OpenWrt does not use codeload. Just the packages. A git clone of OpenWrt does not pull in the packages feed.

Although most of the advantages listed have mainly to do with pull requests or patches, not so much with end users.

[danielfdickinson]

TBH I'd like to know the feelings of folks with commit rights like @thess @wongsyrone @jow- @hnyman @tripolar @dibdot @yousong @EricLuehrsen think in terms of the preferred discussion forum for a question like this, as well as the question itself, more than my own opinion or @diizzyy or @neheb who are commenting from the outside.

[wongsyrone]

I would like to follow the reproducible way.

[diizzyy]

@cshoredaniel
Several committers have accepted this procedure by committing several PRs and using it in their own commits. I have no idea why you're trying to make a discussion out of something that's already accepted.

[dibdot]

@cshoredaniel
Personally I would prefer to wait with codeload migrations until github will support reproducible release downloads that are automatically generated. On the other hand I have no codeload objection if the maintainer said it's OK.

[danielfdickinson]

@dibdot I can understand the sentiment
@diizzyy The problem i had, as I said before, is that this is tree-wide change that I think should be discussed. If folks like @thess and @dangowrt and @jow-, and ... don't have an issue with things being done this way (not just codeload, but the mass filings of PR's without previous discussion) then I don't care. I just know it's been an issue for other projects and there has been heck given over things like this in OpenWrt-land in the past too, so it doesn't seem to be in keeping with existing understanding of how things are supposed to work. As @pprindeville said in another issue, folks are often reluctant to do tree-wide stuff because they don't want to step on other's toes. This seems rather the opposite of that sentiment.

[EricLuehrsen]

FYI as a release option, you can upload any binary as an "asset." A upstream release authority could instead generate the tarball and upload it. That would be static and not subject to cloud based cache churn and tools changes. It would be less expensive to download than a git clone. There is no limit to storage or bandwidth; see also Rest API release documentation.

[danielfdickinson]

@EricLuehrsen Agreed. I think the issue comes when upstream isn't doing proper releases (with uploaded assets) and GitHub's auto-release based on tags is wanted to be used as a formal release (rather than a hash and pull).

[danielfdickinson]

Also posting this on list:

Based on actual responses (most of which were on the GitHub issue not here), it seems the majority opinion is to not actually care about this / leave it individual maintainers to do decide.

If that's not the case please post on the ticket and/or here else I shall close ticket and Rosen Perev and Daniel Englberg(sp?) (@neheb and @diizzy) will be promoting this to maintainers on GitHub.

I wanted to make sure this was discussed openly, but if not enough people actually care to comment, and it seems majority opinion is to no care, then I guess the discussion wasn't needed.

[danielfdickinson]

Since it's holiday season I will wait until after first full week of September to close this issue.

I would like to say that impatience (and I'm badly guilty of this too), is a big part of the problem here and with the perceived maintainer problem with the packages feed, and/or the pushing of commits by non-maintainers (that turn out to be problematic).

[pprindeville]

FYI as a release option, you can upload any binary as an "asset."

@EricLuehrsen Maybe I'm misunderstanding something, but my hair stood up as soon as I read that...

[danielfdickinson]

As no one seems to actually care all that seems to have really happened is pissing of some folks unnecessarily. Therefore closing.

[yousong]

@cshoredaniel Thank you for the heads up. Appreciate that.

[jefferyto]

FWIW codeload hashes/checksums do change. I've been test compiling prometheus lately and this is what I see (paths changed to protect the guilty):

SHELL= flock /openwrt/testing/armvirt-32/tmp/.prometheus-2.7.1.tar.gz.flock -c '        /openwrt/testing/armvirt-32/scripts/download.pl "/openwrt/dl" "prometheus-2.7.1.tar.gz" "bfbeb434342a03b5849e2ec7a0cbe573067299cf59ccf59db0cacd8db8800bb0" "" "https://codeload.github.com/prometheus/prometheus/tar.gz/v2.7.1?"    '
+ curl -f --connect-timeout 20 --retry 5 --location --insecure https://codeload.github.com/prometheus/prometheus/tar.gz/v2.7.1?/prometheus-2.7.1.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 7106k    0 7106k    0     0  2053k      0 --:--:--  0:00:03 --:--:-- 2053k
Hash of the downloaded file does not match (file: fe239e23379ce6d50c5577f515d7f576f1c6f9396ea47fd87c08fbf8b5e163fb, requested: bfbeb434342a03b5849e2ec7a0cbe573067299cf59ccf59db0cacd8db8800bb0) - deleting download.
+ curl -f --connect-timeout 20 --retry 5 --location --insecure https://sources.lede-project.org/prometheus-2.7.1.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 7106k  100 7106k    0     0  2159k      0  0:00:03  0:00:03 --:--:-- 2159k

If I wasn't caching downloads, I would be downloading the "same" file twice every time.

[neheb]

Now we have a proper case of that :).

I ran gunzip on both tar archives, which resulted in .tar files of the same size but different hashes.

I extracted the archives and hashed every file. This is the difference:

mangix@mangix-pc:~$ colordiff 1f 2F 
2599c2599
< 67225f38b1b851c3f34d97f5b3eb80c081b3cc935c3b669e1fceaeb352b79357  ./vendor/k8s.io/client-go/pkg/version/base.go
---
> 8466cdeac38ae3017f04b607bb3404a02e8719c9a5d1511d883772724538d87b  ./vendor/k8s.io/client-go/pkg/version/base.go

So the content is actually different. My guess is upstream actually modified the tag.

edit: and the actual difference...

<   gitVersion   string = "v0.0.0-master+62e591f92"
---
>   gitVersion   string = "v0.0.0-master+62e591f928"

[diizzyy]

So codeload isn't to blame in this case :-)

[neheb]

Hrm maybe it is.

https://github.com/prometheus/prometheus/blob/master/vendor/k8s.io/client-go/pkg/version/base.go

is a template that I'm guessing codeload generates.

.gitattribute has

base.go export-subst

I would guess GitHub updated codeload's Go version? No idea.

[diizzyy]

There are reports of occational breakage when GitHub updates their backend but it's still very rare and by far the pros outweights the cons compared to fetching source code and packaging it.

[yousong]

Hrm maybe it is.

https://github.com/prometheus/prometheus/blob/master/vendor/k8s.io/client-go/pkg/version/base.go

is a template that I'm guessing codeload generates.

.gitattribute has

base.go export-subst

I would guess GitHub updated codeload's Go version? No idea.

This is worse than what I originally expected. Next time other packages may accumulate enough commits to add another byte to the %h...

    // NOTE: The $Format strings are replaced during 'git archive' thanks to the
    // companion .gitattributes file containing 'export-subst' in this same
    // directory.  See also https://git-scm.com/docs/gitattributes
    gitVersion   string = "v0.0.0-master+$Format:%h$"
    gitCommit    string = "$Format:%H$" // sha1 from git, output of $(git rev-parse HEAD)
    gitTreeState string = ""            // state of git tree, either "clean" or "dirty"

[yousong]

There are reports of occational breakage when GitHub updates their backend but it's still very rare and by far the pros outweights the cons compared to fetching source code and packaging it.

Maybe it's rare [1], but when it happens, it affects a lot [2]

[1] https://www.google.com/search?q=github+archive+file+checksum
[2] https://github.com/Homebrew/homebrew-core/issues/18044

[diizzyy]

...and yet they're sticking to it just like everyone else.
https://github.com/Homebrew/homebrew-core/blob/master/Formula/airspy.rb (as an example)
It should also be mentioned that it usually isn't as bad as it sounds as there are efforts to keep packages up to date (although it can be a struggle) which "fixes" most of the issues in the end.

[jow-]

I'd prefer to not hand-wave away these issues just because of some Github fanboyism. From a reproducibility standpoint it sucks that these codeload artifacts are not stable and actually might retroactively change due to some future changes to the repository.

It is not even possible to properly archive these tarballs in the long term when packages are forced to adapt their checksums to match the updated github artifacts as this renders the archived tarballs useless.

Can we please opt for a solution that involves stabletarballs which are reproducible locally? I don't care if everyone else uses codeload. Everyone else also uses curl random-git-hub-gists | sh constructs or compiles code straight off random git repositories. Yet its not a good idea to do so.

[diizzyy]

So other projects are plain stupid and don't care? They've obviously evaluated this solution and ended up with the conclusion that it's worth the "occational extra" effort. So, opting for a solution which severly lowers performance and availability is a "much" better solution just because OpenWrt is "different"?

[jow-]

So, opting for a solution which severly lowers performance and availability is a "much" better solution just because OpenWrt is "different"?

If this is the only way to ensure bit-identical, reproducible, long-term stable tarballs, then yes.

[diizzyy]

That's arguable as OpenWrt as well as with other projects archive/cache tarballs (again, availability).

[jow-]

Caching tarballs is of no use when the same tarball with the same name overwrites an existing one with a different checksum. It is also not acceptable that the Github tarballs are not reproducible. We're solely dependent on whatever proprietary means Github uses to produce its files.

If Github were down one day or decides to discontinue its Codeload mechanism in favor to something else, and all I have is the Git repository, I will not be able to produce tarballs exactly resembling what was previously produced by Codeload.

IMHO it also is a very dangerous precedence to instruct packagers to simply update their checksums whenever supposedly stable tarballs suddenly change. This completely undermines the idea of file verification and makes ensuring long term availability (and more important, buildability) of the entire distro completely impossible.

As it stands right now, a naive file name based long-term caching of Codeload artifacts is not possible due to the non-guaranteed stability of files. This could be solved by caching the files not only by name, but also by their checksum at that moment in time, but that requires adjusting workflows, just to accommodate for GH's deficiencies - again.

[diizzyy]

You can also look at the other way so either way has its flaws and I honestly feel it's being blown out of proportion by quite a bit.

[jow-]

I don't think that questioning a file hosting facility which randomly changes its contents is "blowing things out of proportion". I have no problem with relying on Github facilities as long as the data we acquire from there is long term stable and reproducible.

Right now it is neither, which also makes mirroring it for future use impossible. So a given tarball with a given hash cannot be acquired from anywhere anymore in case GH's mechanism suddenly decide to package it differently.

[neheb]

I am curious.

Would OpenWrt's git solution change the file to its proper format?

[diizzyy]

IMHO to even consider OpenWrt's own solution viable we need at least 3 mirrors with good peering and at least 2 shouldn't use Hetzner (blocked for people in .ru for instance) or any other blocked DC.

[neheb]

I'm just wondering if it generates a proper tarball.

[champtar]

I'm 100% for reproducible builds, so as long as Github don't provides reproducible downloads from codeload, we should stick to OpenWrt magic IMO
Just downloaded all sources for all packages (ar71xx), it takes 2.7G dl.txt
This would require some scripting, but OpenWrt could have a mirror with all sources, and the 11 mirrors that we have could also mirror that, what do you think ?
https://openwrt.org/downloads#mirrors

[diizzyy]

I'm don't like the idea of mirroring all sources as it'll both waste space and bandwidth especially for large packages that are rather frequently update such as the kernel and we already have redudancy for most packages. It's also been brought up a few times that there's much waste of bandwidth as mirrors pulls a lot more data than what they serve.

I'm fine with only mirroring GitHub packages but in order for that to happen we should have some kind of geo/round-robin facility and some kind of mirror status facility.

https://www.archlinux.org/ (Under tools)
http://spacehopper.org/mirmon/

Another option would to mirror those tarballs on github and use CDNs such as
https://gitcdn.xyz/ , https://raw.githack.com/ , https://www.staticaly.com/
That way we wouldn't need to involve mirrors at all

[poranje]

Op 22 mrt. 2019, om 11:01 heeft Daniel Engberg [e-mail] het volgende geschreven:

You can also look at the other way so either way has it's flaws and I honestly feel it's being blown out of proportion by quite a bit.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Security sometimes is difficult to balance against other interests.
In this case trust that the distributed deliverables are built from checked and committed code is severely impeded and that is not acceptable.

Regard,
Paul

[neheb]

I will close this as there doesn't seem to be a consensus. The summary is:

codeload mostly works
local tarballs mostly work.

Both break occasionally. There are still several locally generated tarballs with wrong hashes on https://downloads.openwrt.org (seen when compiling with make V=s). That should probably be fixed but is beyond the scope of this thread.

[danielfdickinson]

@neheb @hnyman @jow- I think this should be reopened. I see two people strongly arguing against and at least nine people who have expressed concerned. I think think there is a pretty strong majority against codeload. This really should be put to a vote since it seems that's the only way there is going to be acknowledgement what the majority think.