Pmbootstrap: Improve contribution workflow from Alpine derivatives

[ollieparanoid]

Dear upstream friends from Alpine,

we are very grateful about how easy you guys make it to work together by directly answering our questions in issues here in GitHub (@ncopa) or even joining #postmarketOS and giving valuable answers from the view of an Alpine developer (@kaniini, @Shizmob)!

So it seems like you guys embrace derivatives of Alpine Linux like Adélie Linux (CC: @awilfox) and postmarketOS.

We were wondering if we could improve the collaboration some more:

• Is there any way we can speed up the PR merging delay? (that's often blocking our development)
• The buildbots (e.g. for armhf) seem to get stuck easily, can we help with fixing that?
• Could you imagine us derivatives using the build infra-structure for our specific packages, that don't make sense to be upstreamed? (e.g. device specific kernels we need to carry around because Android likes to fork everything - our plan is to mainline them ideally)? The idea comes to mind, because this mailing list post mentions something like PPA as idea.

At least @craftyguy and I would like to invest some of our "postmarketOS development time" in improving the situation.

Ideas on how we could help:

• Help out with reviewing and testing PRs from other people
• Do any other Alpine related tasks, that would in turn give you guys more time to review our PRs
• Fix the buildbot scripts
• Do research/buildbot script changes regarding the PPA-like idea if desired
• your idea here

So what can we do?

PS: I wanted to post this in alpinelinux/aports, but the issues are disabled there. We can take this to the mailing list instead, if you guys prefer that.

[awilfox]

I made a vaguely similar thread on the ML that I hoped to spark a similar discussion.

Speaking for the Adélie team (now 5 strong!), I'd like to discuss/answer some of these points inline:

Is there any way we can speed up the PR merging delay? (that's often blocking our development)

We haven't had any issues (@Aerdan and I seem to get our PRs closed within 1 week) so I can't comment here.

The buildbots (e.g. for armhf) seem to get stuck easily, can we help with fixing that?

This is something I noticed happening too. We would love to apply our knowledge of CI, sysadmin, and weird platform quirks to helping this be resolved as well.

Could you imagine us derivatives using the build infra-structure for our specific packages, that don't make sense to be upstreamed? (e.g. device specific kernels we need to carry around because Android likes to fork everything - our plan is to mainline them ideally)? The idea comes to mind, because this mailing list post mentions something like PPA as idea.

(I am not a lawyer, but I play one occasionally) This might be a little complicated depending on who owns the infrastructure. I believe IBM has contributed the ppc64le builder and is hosting it for Alpine, for instance, so I'm not sure if that would be something they would be able to offer. (Asking IBM nicely never hurts, though.)

At Adélie, we mostly just use the rack I have in my closet (x86_64 x2, x86_32 x2, armv7, mips64, mips32el, sparc64 x2, ppc64 x2, ppc32 x1) so we haven't really needed the infrastructure either. (I used to do FreeBSD and Gentoo porting work, hence my weird collection of hardware.) Depending on your needs I might be able to set up an LXC with Alpine edge on our infra, if you need it.

Ideas on how we could help:
Help out with reviewing and testing PRs from other people

I think that would be a good/nice thing to do anyway and I helped for instance with the Python 3.6 stuff. :)

Once I finish fussing with Plasma LTS I hope to do more of that.

Do any other Alpine related tasks, that would in turn give you guys more time to review our PRs

That was the subject of my ML posting in a nutshell. I think Alpine would benefit from a tiny bit more organisation and list of things people can do, so that they can be done and Alpine core devs can get back to being core devs.

Hope this helps, look forward to collaborating more in the future.

[kaniini]

Once we have the new build infrastructure in place, I am sure that we could arrange for derivatives to make use of it. But I will need to talk with the infra team before committing us to that.

Regarding armhf build bots being stuck, as previously discussed, it is because rootbld needs to be rolled out, which allows different packages to fail separately. This is mostly down to working out the specific gotchas that are left, but the main heavy work to make it happen has already been done. Beyond that, it is just a matter of actually putting rootbld into service. I don't run any of the official builders right now, so I can't really speak in terms of a timeline for that. Implementation of rootbld is considered an important release goal for 3.7, however, so it is possible that @ncopa may block the release until it is implemented.

In regards to speeding up merge delay, we are working on a few resources for contributors to find developers willing to sponsor the merges so that they don't wait in queue.

Regarding the ppc64le builder, IBM contributed either 1 or 2 160 core POWER8 machine(s) and also contributes toward the colocation of that machine in the open source lab at Unicamp (but the day to day operations of the machine(s) themselves are coordinated by the Alpine sysadmins).

[kaniini]

Beyond that, my personal vision is to onboard derivative developers as Alpine developers so that there is more opportunity for symbiosis. For example, there is a lot of overlap between the Alpine Desktop, Adelie, and postMarket OS projects, so I am hearing you guys loud and clear on that, it's just a matter of getting things done to streamline this process.

[andychu]

@kaniini If there is some testing or debugging work necessary for rootbld, let me know... once upon a time I tried to build a containerized distro, where packages both run and build in containers, so I have some familiarity with those kinds of things.

Also I have some hardware to set up a builder but I think you probably have them all under a single set of admins for trust reasons?

(found this issue on IRC)

[craftyguy]

@kaniini pointed out in IRC that there are a few packages that keep causing the armhf builder to fail. At least one of the failures is due to binutils not being installed. There's not an obvious (to me) way to reach out to the "armhf builder admins" to notify them of this (and future) issues.

I can definitely help out with submitting any PRs to fix packages that break the armhf builder, I just need some agreement from the "armhf builder admins" and some help from them to keep their build server going so we can continue to hunt for problem packages that need fixes.

[kaniini]

As previously stated, the infra team coordinates the build servers. If you write to [email protected] you could report the problem that way.

With that said, Alpine itself is a noncommercial project and the project does not pay for sysadmins, so it is not possible to provide an SLA. You could perhaps inquire about joining the infrastructure team but you would need to be a developer to join it in most circumstances.

[kaniini]

This is just sort of an upstream gripe so please do not take it the wrong way, and I hope it explains why I feel that derivative developers must also be Alpine developers for things to work as optimally as possible.

The existence of this thread comes off badly to people who are devs in Alpine because it reads like pmOS wants special treatment. Whether or not that is the case, I am not sure, but it does read that way.

pmOS developers want:

• faster reviews
• collaboration with buildserver maintenance for the arm ports

Well, the best way to get pmOS what it wants is to get pmOS developers directly involved with the Alpine armhf and aarch64 ports as developers. Developers, in general, get to bypass the review queue, and developers can join the infra team and participate in buildserver maintenance.

The ARM ports are essentially community driven, unlike the x86, POWER and s390x ports which have established ecosystems that allow development time to be sponsored. It is unfortunate that the ARM ports do not have this same type of ecosystem, but that's largely because there's very little commercial interest in Alpine on ARM bigiron, because ARM bigiron has basically fizzled out.

I agree wholeheartedly that pmOS has the potential of greatly expanding the Alpine ARM porting community, and we are working on various ways that the community-driven ports could have sponsored development as well, but this takes time. Bringing the community ports on par with the mainline ports (x86, POWER) is something that I am working with the rest of the core team to find a solution to, but as I said, it is going to take some time.

Strengthening the ARM port starts with onboarding more developers to support the port. So to be very clear to the pmOS developers: I am personally willing to act as mentor to pmOS developers who wish to become Alpine developers, you just need to come talk to me on IRC. If there is interest in this, I am also willing to set up a build lab for ARM on my personal infrastructure which I already use to test submitted packages on x86. To be clear, however, just because you have an almost guaranteed mentor for your work in Alpine, you will still be required to meet all other requirements for becoming a developer, such as multiple references from other Alpine developers and so on.

I am also working on a project that I hope will enable prospective contributors to find developers to work with directly so that they can bypass the normal review queue delays and also get started on the onboarding process at the same time. Stay tuned on that one.

But ultimately it all comes down to sitting down and putting the time in to become an Alpine Developer. If you can do that, you'll get what you want: the ability to drive the ARM port more directly.

[craftyguy]

Sorry that us reaching out to you to offer help has come across as a 'demand', in fact it's far from it. Until your post above, there was no clear communication from the Alpine developers on how a project like pmos could help Alpine. You said we should become developers, ok! What's the process? Where do I start?
Is there a list of items "to do" that I could start working on to hopefully, after some time, gain those 'developer references' you mentioned?

[ollieparanoid]

In case this thread sounds confusing, discussion has been going in parallel in #alpine-devel (logs).

The existence of this thread comes off badly to people who are devs in Alpine because it reads like pmOS wants special treatment.

That was not intended at all, sorry.

@kaniini: Thank you for taking the time to answer this topic in detail, and for giving us a few pointers on how we can get more involved. Especially that you would mentor someone from postmarketOS who wants to become an Alpine developer is a great offer (and yes I understand, that there are additional requirements).

Thanks @craftyguy for directly taking that opportunity!

As I see it, this discussion can be closed (please reply if you think otherwise, dear reader).