Pmbootstrap: Binary Repository #2

[ollieparanoid]

What won't work any time soon

Reproducible builds

The first binary repository approach involved reproducible builds, and it turned out not to be feasible and it wouldn't provide any real security benefit. Read https://github.com/postmarketOS/pmbootstrap/issues/64#issuecomment-334252443 for details. But we need a binary repository, it makes development much easier (compiling KDE takes forever, and we have some developers for whom compiling kernels also takes hours).

Please note, that I chose the reproducible build route to provide maximum security. We do not only run the binaries from this repository on the target devices (phones, tablets, ...), but also on the PCs where we develop (e.g. the cross-compilers gcc-armhf, ...), and sometimes even with root access (e.g. the heimdall flasher application).

What we have left from this approach and can use, are mirrors for hosting the packages (thanks to @MartijnBraam, see #365).

Alpine building postmarketOS packages

So the next best thing would be, if Alpine built our packages along with the official packages. Then we wouldn't need to trust another person for providing the packages. Two Alpine developers had the idea of providing a PPA-like system at some point in the future, and therefore I've asked on the mailing list.

The short answer is, that they would need a new build system first, for which there is no time plan and that Alpine focuses on its upcoming 3.7 release so they don't have time for this.

Looks like we can talk about this again after 3.7, but realistically even if it worked out that way, it won't happen in the next few months, which means for me that we better have a short-term solution.

Short-term solution

It seems to me, that the fastest way to get this going is if I build all the packages myself, but not reproducible. This might be the next best deal regarding trust, because you need to sort-of trust me already for the pmbootstrap code, which also runs on your computer and while you can just read it, people rarely read all code before executing it.

I would use an i3 CPU with cross-compilation (which I'm using right now to do bigger builds), which is slower than what @PureTryOut uses. But then again, we use ccache and we only need to build changed packages. As long as the packages compile regularly, the delay shouldn't be more than one day or so, which is a lot better than what we have now (no binary packages).

For the first iteration, I would trigger the builds manually and I might be able to set this up in a week or so. I would write a script, that does the following:

• Clone pmbootstrap
• Build some (later all) packages
• Upload the whole packages folder to the official mirror

Future ideas:

• Delete outdated packages from the repo (use zap -m)
• GitHub trigger to build automatically whenever a commit lands in master
• compile natively on armhf/aarch64 (maybe on old quad-core phones? even the lg-mako has a quad core - compiling natively is better to avoid bugs from qemu)
• use a bigger machine for compiling if required

Thanks for reading. What does everybody think about this proposal?

@MartijnBraam: could you configure the primary mirror, so that packages can be uploaded via rsync or similar (right now we pull them from a git repository, but that isn't so great for binary files)? And as I understand it, the other mirror automatically syncs from the primary mirror, or how does it work?

[drebrez]

I like the short-term solution 👍

[z3ntu]

I have an IPv6 accessible PC that we could use for building a lot of packages in one go.
Specs: Intel i7-3770, 16GB RAM, also SSD and HDD storage (but not really a lot right now)
I would prefer to not having it running often but eg for the initial build it can be used (or other "mass-builds").

[ollieparanoid]

Thanks for the offer! I would come back to that if it turns out, that building everything on that PC isn't feasible.

[MartijnBraam]

Both mirrors currently pull their data from github directly, I could make one mirror pull from the other one but that would introduce dependencies between the machines which isn't neat for outages. I could set-up rsync so you can push packages to both servers.

Should we have a CA for signing the packages from multiple builders when we let multiple people build the packages?

[ollieparanoid]

I think it works for most distributions, that the builders push to one server first, and then all other mirrors sync to the first mirror. Otherwise it would scale badly.

But I'm uploading an initial batch of packages to the github repo right now, which should then get pushed to your mirrors when everything goes well.

Should we have a CA for signing the packages from multiple builders when we let multiple people build the packages?

I think it is easier, if we just add multiple keys to the keys folder. Then we can easily remove keys which are no longer needed without complicated revoking mechanisms (I am not sure if apk understands them anyway).

[ollieparanoid]

remote: error: File x86_64/linux-postmarketos-mainline-4.14_rc7-r1.apk is 105.68 MB;
 this exceeds GitHub's file size limit of 100.00 MB
remote: error: File x86_64/linux-postmarketos-stable-4.13.11-r1.apk is 105.01 MB; th
is exceeds GitHub's file size limit of 100.00 MB

Okay, pushing stuff over GitHub won't even work well for one demo run. I'll run it again with the linux-postmarketos-* packages removed.

[MartijnBraam]

I'm not saying that we should push to every server individually after building, but that it should be at least two to prevent a single point of failure. Other servers in the future could mirror from those directly.

[ollieparanoid]

That sounds like a good solution!
So here is my initial push with some binary packages. Notably all cross-compilers, and the maemo folder for all architectures, as well as some device kernels and most packages for x86_64 (e.g. weston). I've force pushed over the old reproducible builds approach, which might be the reason why your mirrors did not pick it up yet.

https://github.com/postmarketOS/binary-package-repo

@MartijnBraam: Could you set up rsync access to both mirrors when you have time, so I could sync them up directly?

[MartijnBraam]

@ollieparanoid I've made [email protected] which has access through the pmreleaser key. Its set up with rrsync so the server side path of rsync is relative to the mirror webroot. If this setup works I'll can copy the configuration to the other mirror.

The automated script failed because:

$ git pull
fatal: refusing to merge unrelated histories