Packages: Growing lack of package maintainers

[thess]

It seems we are collecting a number of packages with no MAINTAINER lately. This was not a direction chosen when the new Github repository was created and we crafted guidelines. The purpose was pretty clear that we wanted individuals responsible (OK for more than 1) for the submission of and updates to each package. The main goal at the time was to cleanup the old repository given that a number of packages were outdated, unused and/or didn't build at all. I do not wish to see this repository to degrade to that state.

That said, If there is enough consensus among the members, to drop this requirement, I will gladly go ahead and approve/merge updates without maintainers as long as folks sign-off as having tested the change and one other person has ACK'd the PR. Personally, as a package maintainer, Likewise, for my packages and dependencies, I take the time to at least build and maybe test package changes that are not always straightforward.

I am also looking for volunteers, to help review and possibly close old PR's (and issues?) given some criteria and process TBD.

[diizzyy]

I personally think your last sentence is what's causing quite a bit of extra workload and makes PRs some more or less come to a grinding halt which in turn makes contributors give up/stop caring.

https://www.freebsd.org/doc/en/books/porters-handbook/makefile-maintainer.html
I think that's reasonable?

Create a generic template on the layout of a Makefile which you should follow in general
Add generic comments on guidelines for specific sections like

https --> http --> ftp
Official tarball --> codeload --> git/svn etc
cmake --> gnu autotools
Essentially a condensed version of https://www.freebsd.org/doc/en/books/porters-handbook/

[diizzyy]

also, something like this should be taken into consideration
https://lists.freebsd.org/pipermail/freebsd-ports/2014-November/096419.html

In addition there should be something like TravisCI that actually spits out full logs or some other kind of auto testing facility in place and I think a "package manager team" should also have the right to refuse a port if it's for a very limited amount of users (specific usage).

[luizluca]

@thess, I do prefer to accept only packages with maintainers. However, in fact, it is mostly the same situation of a package that was once maintained and became orphan. If we keep orphan packages, we might accept unmaintained new packages as well (it's just an analogy, not my desire).

If a package is unmaintained, should it be removed automatically just after branching a new major release?

Or should it only be removed when a major issue appear, like a broken build, security vulnerability or it simply does not work anymore? (in fact, this applies to any package, "with a name in MAINTAINER" or not). It would be nice if a bot, after some time, could open build and security issues (mentioned in a CVE) automatically and, after some more time, open a remove PR (move to packages-abandoned).
BTW packages-abandoned do have an open PR for years)

FreeBSD ports is the same situation of openwrt-packages. Those rules mostly fit our needs. That's a good start. We do lack a little more explicit rules. For example, I do have commit access and I can commit new packages. However, I'm not sure if I must do it. I use my rights only for "maintaining my packages", not bothering others with updates/minor commits. What is the process of accepting new packages?

We do already have a team of "package maintainers" in github. We could explicitly have a new "package manager" team that rules whether to accept new packages, reset maintainership or remove packages with major issues.

[dibdot]

@thess
I'd prefer to keep the current policy and only accept packages with a (responsible) maintainer. Further on I think it's a good idea to "fade out" those stale packages (without maintainer or maintainer doesn't react over a longer time, e.g. for 3 months) with every new OpenWrt release.

[diizzyy]

@dibdot
So you're suggesting that we nuke more than half of the tree essentially because packages doesn't have a maintainer that's active?

[Andy2244]

A more pragmatic approach would be to keep the packages that still compile, until they break and no new maintainer is found?

[champtar]

I would really love to have usage statistics (send installed packages + version) and / or a list of active user for each packages so it's easy to contact other people that do care about the package and know there usage

[thess]

@Andy2244 - I agree with keeping the packages and when they break , mark them BROKEN (no longer will build). How we deal with PRs and new package submissions without a MAINTAINER, needs to be addressed too.

@champtar - If you are looking for built package downloads, we are attempting to get those stats. There is a WIP, for example, here: https://downloads.lede-project.org/stats. Also, there is no hard/fast rule preventing others from adding themselves to the PKG_MAINTAINER var. Perhaps we could encourage that.

[jow-]

I noticed that sometimes people only assume maintainership for the sole reason of getting the package back into the binary repositories because they have an urgent need for it. I can also understand that not everybody wants to enter a multi year maintainer commitment only to have a specific package available.

I think there are multiple things that can be done to partly address the problem of unmaintained packages or the lack of manpower in general:

• reconsider the one-maintainer-one-package approach as this can't scale particularly well
• formally allow other contributors to make changes in any package, especially for things like CVE fixes
• formally switch to an "opt-out" approach, means changes to packages are accepted when they look sane unless the maintainer objects to them within a specific time period
• start introducing CPE_ID tags into packages to allow for automatic security tracking at least
• keep packages without maintainer until they break building
• move packages to abandoned once they fail on the majority of targets for a given time period, e.g. 4 weeks
• improve our CI by automatically propagating build failures here

[diizzyy]

If security is of focus the keep package until it doesn't build argument will in many cases do the exact opposite.

[pprindeville]

I would really love to have usage statistics (send installed packages + version) and / or a list of active user for each packages so it's easy to contact other people that do care about the package and know there usage

Well when someone uses IB, we can track the downloads of individual packages, right?

[danielfdickinson]

Also, (I've been away a while) are there folks contributing regularly that maybe should have commit access?

[danielfdickinson]

I'm wondering if would help (I could help set it up) to have something like Ubuntu PPA's or Fedora COPR, where users can easily build packages they (and other interested folks) can use on their devices? I don't think for those repos it'd be required to build for every arch.
If that existed I think being a lot more aggressive about removing packages makes sense. I think it's desirable to reduce the amount of cruft. At the same time some packages with an obvious maintainer are clearly useful to the community and shouldn't be removed willy-nilly (perhaps there should be an option for PKG_MAINTAINER:[email protected] or somesuch (Community Package Maintainers) for such packages; what packages get that designation would be decided by whoever ends up as a voting member of that team, perhaps).

Thoughts?

[neheb]

I have ~25 pull requests with most of them having inactive maintainers.

I've thought about this for a bit. Maybe something like arch is needed where you have maintainers but you also have Trusted Users that do the grunt work regardless of the maintainers. Arch Linux has 13039 packages. ~250 are out of date. That's 2%. OpenWrt has less packages and more of them are out of date.

On throwing packages in abandoned repo: Unless hard drive space is an issue, I completely disagree. Just because there's no user today does not mean there won't be one tomorrow. I am however a fan of doing so when a better replacement is found. I have two pull requests that replace libjpeg and pkg-reconf with -turbo and pkgconf. I should probably update the latter.

On build failures: I have no idea how to read buildbot failures. An example: fdm does not compile on arm_mpcore: https://downloads.openwrt.org/snapshots/faillogs/arm_mpcore/packages/fdm/ . The logs seem fairly useless. Or I have no idea how to read them.

Many of my pull requests replace using git with using a tarball generated by github. This has the benefit of reducing hash mismatches between different hosts. There was a situation once where one of @wongsyrone 's version bumps had a different hash than what was generated locally for me from the git repo. Won't mean anything unless merged though...

Finally, since we have @sdwalker 's lovely uscan tool, might as well fix up packages that show issues: https://sdwalker.github.io/uscan/

[wongsyrone]

There was a situation once where one of @wongsyrone 's version bumps had a different hash than what was generated locally for me from the git repo.

Which one?

[neheb]

https://github.com/openwrt/openwrt/pull/815#issuecomment-376057887

edit: I believe I double checked on my end at the time as well. In any case it's irrelevant now.

[karlp]

Rosen Penev [e-mail] wrote:

I have ~25 pull requests with most of them having inactive
maintainers.

And the vast majority of them are changing packaging for marginal
gain. Inactive is also relative. For a maintainer of a package
that updates once a year, maybe less, being "inactive" just
because they're not approving changes withing a day or two, or
even a week, isn't necessarily "inactive"

I've thought about this for a bit. Maybe something like arch is
needed where you have maintainers but you also have Trusted
Users that do the grunt work regardless of the maintainers.
Arch Linux has 13039 packages. ~250 are out of date. That's 2%.
OpenWrt has less packages and more of them are out of date.

And arch users famously complain that things don't work. Updated
and compiling isn't always the same thing as functional. I'm not
saying we shouldn't update things, but updating every package in
openwrt the day upstream releases something isn't necessarily any
sort of sign of quality improvements. Particularly given the
nature of openwrt targets, it often exposes breakages.

Are there any realllll problems here? I've personally felt that
the new packages repo has been wonderful and largely welloiled
compared to the old system. There's always room for improvements,
and some packages definitely warrant multiple maintainers, but
this all feels like a storm in a teacup.

Cheers,
Karl P

[hnyman]

In addition to disappearing maintainers, we also have a challenge of low activity by committers to handle pull requests. There are only a few persons who actively merge PRs on larger scale. Quite many committers in the package feed repo seem to only care for a the few packages that they are maintaining by themselves.

[karlp]

Hannu Nyman [e-mail] wrote:

In addition to disappearing maintainers, we also have a
challenge of low activity by committers to handle pull
requests. There are only a few persons who actively merge PRs
on larger scale. Quite many committers in the package feed repo
seem to only care for a the few packages that they are
maintaining by themselves.

So give more maintainers commit rights. I was given commit rights
in the understanding that it was to streamline the maintennance
of my own packages, and I was being trusted to not step on other
peoples work.

Are other maintainers _not_ given commit rights? Why not?

Cheers,
Karl P

[neheb]

I maintain 4 currently. I have none.

[pprindeville]

Quite many committers in the package feed repo seem to only care for a the few packages that they are maintaining by themselves.

I'm not sure that's their motivation. Some of them might be hesitant to step on someone else's toes, hence they only work in their own sandbox.

I'll very occasionally commit in another package if (a) the commit looks very safe, (b) it's been discussed adequately, and (c) the actual maintainer has had adequate time to speak up but hasn't...

[danielfdickinson]

@pprindeville That's a good point; I think the 'owned' packages sentiment tends to that feeling, which (I think) is why @jow- and others have suggested a more general 'packages feed' maintainer over all packages being owned (with uneven levels of participation (for various reasons)) by an individual maintainer or (rarely) team.

[EricLuehrsen]

@jow- @dibdot @thess my bias tends to agree with your statements.

I would think part of the problem is "packages-abandoned" is not much of an option. The choice is regular package or dark abyss of doom. For new packages someone would like to dump in OpenWrt without a maintainer, then maybe "packages-experimental" or "-unstable" or "-bleedingedge" is appropriate. There could be a process to get consensus to move to regular packages after time, use, and a responsible maintainer.

Now thinking out loud, more than making any specific recommendation ... Fringe packages could be retroactively shoveled to experimental if (1) they are still apparently relevant in the wider world, but (2) they lack a maintainer or has been absent too long. This repo then could even have its own menuconfig category. This menu rule is one of few strict enforcement in "packages-experimental" and the other is Travis-CI passes. That way it can be part of a proper release, and binaries are available, but amateurs have fair warning of its quality control. But wait, there is more. Experimental can be a good place to train or test new committers before letting them have at regular packages. Will they help others? Will they "be nice to each other?" Presumably experimental is a place with a lot of churn and new widgets (which may die, but fun in the mean time).

[danielfdickinson]

@thess volunteers for triaging (once process is defined)

[danielfdickinson]

@EricLuehrsen I have some ideas about using Travis-Ci's deployment option to create a PPA / COPR type option for OpenWrt. The other piece of course is the deployment 'target' (e.g. something that publishes the built packages - that is is willing to accept them from Travis-CI). It's a significant chunk of work, but it's doable. This RUPA (Random User Package Archive) system would be partly a Travis chunk that would be versioned and used by user's in their own public GitHub repos, and published to the RUPA server via credentials one signs up to get. A final piece is an luci-app and/or cli script so users can add the appropriate key to their opkg keys, more easily (but with the usual warnings about using packages from random archives).

My though with this is to reduce how much hosting and effort the OpenWrt project has to do wrt third party packages. I do like the idea of experimental, especially as a comitter testing ground.

In any case I think a strategy change is needed.

[danielfdickinson]

As I said in https://github.com/openwrt/packages/issues/6748:

I would like to say that impatience (and I'm badly guilty of this too), is a big part of the problem here and with the perceived maintainer problem with the packages feed, and/or the pushing of commits by non-maintainers (that turn out to be problematic).

[pprindeville]

Could we move ownership of a package to at-large and have maintainers who want to step in and do reviews, lead discussion, recruit maintainers, and yes, merge PR's?

Not everyone would be qualified to be a member of the at-large team... a good overall understanding of OpenWrt and how package dependencies typically work, regular availability, etc. might be some of the desired characteristics.

[pprindeville]

Fringe packages could be retroactively shoveled to experimental if (1) they are still apparently relevant in the wider world, but (2) they lack a maintainer or has been absent too long.

@EricLuehrsen That seems like abuse/overloading of the experimental attribute.

[EricLuehrsen]

@pprindeville that snippet was from a paragraph marked as random thoughts... experimental is not an attribute but rather a good marketing name for a repo. experimental is relevant software but lacks quality from responsible custodianship, and some software today in packages meets this criteria already. abandoned is mostly done but someone may need to re-implement a package, one-off, for a network deployment in an organization with legacy infrastructure dependent on obsolete tech. Leaving packages to have high quality responsible custodianship where we all give each other business over picky attributes in our contributions... again, not well organized recommendations.

[Andy2244]

Just a side-note, but related to package maintainers.
What i have noticed as a new maintainer, is that getting a package added in the first place can be a really "grindy" process to begin with. I always wondered why so many forks, don't seem to even bother trying to get packages/patches upstream. It took me 7 months to get my package reviewed and finally merged, sure 2-3 months was about usefully changes and good feedback, yet i had to actively "campaign" via irc and github '@' pings to get anything done.

As a example look at #6357 the last comment was nearly 2 months ago and it now sits on page 4, with 75+ PR on top if it. So from my experience without the author also "campaigning" i see no chance this gets merged at all.

This process and lack of guidelines, can be really frustrating and might be a reason why potentially package maintainers may just give up or not submit PR in the first place.

I never expected to get any "quick" review/answers, yet i did not expect that after submitting a PR it would just sit there for months, without anything happening. I was actually very close to just maintaining my own fork, since it looked like no one would ever review it.

I understand that the write access maintainers have all there hands full, yet if a package passes travis and got a first "looks good" review, there has to-be a way to get this actually merged afterwards.

From my experience, there are tree big hurdles:
1) getting a reviewer in the first place to give feedback on how bad/good the package is
2) after making several changes, keeping the review "alive" or reviewer "interested"
3) after making all wanted changes and passing travis, actually getting the reviewer or someone else to merge it

[jow-]

It would partly help to adopt an "opt-out" approach, means a (new) package PR may get merged by any committer if
1) it is older than two weeks without feedbacks
2) passes the automatic quality checks
3) has no requested changes or other unaddressed feedback

If a committer requested changes that got addressed and did not respond within two weeks, the change may get merged anyway.

Eventually we may need to consider having multiple tiers, e.g. "packages" and "package universe" because right now any package accepted here will end up in the officially trusted repo. Lowering PR standards will inevitably lead to lower quality and higher trust problems - which can be an acceptable trade off when properly dealt with.

[Andy2244]

@jow- Yes this would help, but if you look closely at #6357 it does not even have a listed reviewer yet, even through some reviewers did comment on it. So the author cant even '@' ping someone to ask politely for a merge. There seems also no FIFO process, so that the author might check some status or list when/if the PR might get processed by someone.
To me it seems that if a new PR is not trivial and gets someone to initially assign, it will just sit on the pile, until the author or some other random events draws attention to it.

[jow-]

That is why I suggested the opt-out approach. In the end its just that nobody is "brave enough" to actually merge a huge chunk of work. If there is a well defined check list to go by (travis builds, signed-off is correct, no negative feedback for two weeks) it will hopefully encourage maintainers to merge packages earlier.

[Andy2244]

Yeah the "packages" and "package universe/community" approach might help, so new packages or none "essential" packages have a place to live. This lowers the need to "hunt" for unofficial external feeds, which are "invisible" for normal users anyway.
Maybe it even encourages external feeds to create more PR or get more involved with the official repos? @wongsyrone @tobiaswaldvogel

[diizzyy]

Please elaborate why we need to split things for more micro management?
The lack(?) of interest and/or "reluctantless" to actually adopt anything is pretty much the reason why there's no progress.

As far as forks goes,
In short, it's more of a chore than "offloading"/sharing work(loads) in many cases submitting upstream.

[jow-]

Please elaborate why we need to split things for more micro management?

I was bringing up the general trust problem when lowering the entry barrier to submit packages that ultimately end up in the binary repo. At no point was I proposing to micro manage anything.

[EricLuehrsen]

@jow- sounds good, instead of packages-experimental use packages-universe such as another popular distro. I just thought about LuCI. Should luci-app-x be placed in packages-universe or should there also be luci-universe companion repo.

[diizzyy]

Let me rephrase,
Why do we need to lower the standard/quality and split into different branches which will lead to more work in then end? Again, documentation and guidelines would solve that issue in most cases. It makes little sense to split since we only have a handful of packages (compared to other distributions) and I would honestly rather see that we address why it doesn't work in the first place rather than shoving a bunch of packages somewhere else for whatever reason. There are quite a few contributors/committers who reviews PRs etc however the without any documentation/guidelines it gets tedious and inconsistent. I wouldn't mind accepting niche packages as long as we have some kind of deprecation work flow that also includes packages where the submitter for whatever reason doesn't want to be a maintainer which doesn't fall into the very specific cases category.

[jow-]

Why do we need to lower the standard/quality and split into different branches which will lead to more work in then end?

I never wrote that we will need to lower the standard. The reasoning was: if we go solely by a set of guidelines and every package which looks sane on first sight and passes the automatic quality gates is accepted and ends up in the global trusted-by-default binary package repo, then we'll sooner or later have a problem with that. Custom or exotic packages where nobody knows what they're actually doing, new library packages getting introduced breaking many other ones, untested version bumps producing compilable but not working packages, packages invoking upstream Makefiles which break the build due to doing unanticipated actions, Makefiles shadowing builtin packages through the use of PROVIDES etc. There's endless possibilities why things can go wrong even if a package submission looks sane on a first sight.

Given that we're discussing a lack of maintainers here and that there are proposals to tackle that by applying consistent rules and merging everything which meets these rules, I anticipate suboptimal things to slip through which will cause problems. - At least from personal I experience I learned that often when I merged patches without own testing, they ended up breaking things I had to fix afterwards.

Again, documentation and guidelines would solve that issue in most cases.

So why not take a stab at writing some and put them into the wiki section here? You could start with https://forum.openwrt.org/t/packaging-guidelines/65 and expand upon that by describing the current mode of work in text form.

[diizzyy]

Because there are no acknowledgements on agreeing on anything...

[jow-]

Because there are no acknowledgements on agreeing on anything...

I do not understand that? It seems all agree that a set of clear guidelines is whats lacking, so lets write some.

[diizzyy]

...and exactly what would those be?
The only clear acknowledgement so far as far as I can tell is made here https://github.com/openwrt/packages/issues/6584#issuecomment-408405768

[jow-]

Well someone needs to sit down and write a RFC proposal for a set of guidelines. Without that, this discussion will not lead to any constructive outcome.

Once there is a list of ten guidelines or similar, they can be discussed and decided upon individually. I don't see that much disagreement here, the main culprit seems to be on how to deal with packages without maintainer.

So here's a stab at it:

1) Packages Makefiles should adhere to https://forum.openwrt.org/t/packaging-guidelines/65
2) New package submissions that fulfill the package criteria in 1), are properly signed off and pass the automatic build tests may be merged by any committer
3) Changes submitted to maintained packages may be merged by any committer if the maintainer does not raise objections within 14 days after getting notified
4) Security fixes may be applied by any committer without maintainer consent given that these fixes do not alter the intended behaviour of the package (e.g. do not combine CVE patches with feature flag changes or (major) version bumps)
5) When a maintainer fails to react on submissions for three consecutive months in a row, he gets removed as maintainer from the package
6) Packages lacking a maintainer are transitioned to a "stale" repository ()
7) Packages in "stale" state not building on the majority of platforms are marked "broken" (*)
8) Committers are encouraged and allowed to merge PRs for any package that qualify for 2), 3) or 4)
9) PRs with maintainer objections / requested changes that do not get reworked within 14 days may be closed by any committer
10) Tree wide Makefile updates that do not affect the behaviour of the package (e.g. adding PKG_CPE_ID, setting PKG_BUILD_PARALLEL etc.) may be added without maintainer consent

) The act of "marking" or "moving" packages must not entail physically altering the repository location, we can as well think about introducing a PKG_STATE flag or similar which is respected by OpenWrt and used to move the .ipk into a different location or to decide whether to show it in menuconfig or not
*) Personally I would even keep @BROKEN packages in the tree for a while and only finally drop them when they're not touched for a few months or years

The rules above are a suggestion and could even be automatically enforced by the use of bots to some extend once there is a consensus.

[Andy2244]

@jow- I like those points, this way most of my package PR would have been already processed, without me campaigning for them.

There is a small extra scenario i like to see addressed, namely package PR that depend on openwrt/base changes/PR. As example #6577 would be a candidate for "universe" yet, since diizzyy has no openwrt write access he could not merge it, so now this PR again is in limbo and i have no clue if it will be merged anytime soon or not. There is also no reviewer listed, so atm i would need to "campaign" for openwrt/openwrt#1233 and than ping dizzyy again, which only works because he reviewed some of my other PR and is a nice guy 😄

So removing the "campaign" aspects and just have a default time window you can expect things to happen automatically would be really nice, not that i dislike hopping into the irc.

[poranje]

So removing the "campaign" aspects and just have a default time window you can expect things to happen automatically would be really nice, not that i dislike hopping into the irc.

However much I sympatice with the idea, acceptance/merges just happening due to a passage of some period is certain to create havoc, e.g. in terms of security. If I were intend on getting a foot in the door (of others' networks), this would certainly be a vector.

[Andy2244]

@poranje Why?
We still talk about a human review process and someone with write access that does the final merge, if all conditions are met. The most notable thing that would change, is that if all is ready to-go and the time has passed, it can get merged by anyone with write access. This removes the need to babysit a PR and hopefully ensures that a PR is either merged or rejected for some reason.
Just check the PR section, there are PR from 2016, that are still "sound" and passed travis, yet for some unknown reason where never merged or rejected. This is just frustrating from a PR author point of view.

[pprindeville]

Please elaborate why we need to split things for more micro management?

@diizzyy I don't see it as micro management so much as it's calling out the packages that require at-large maintenance and don't really have package owners.

[neheb]

@poranje huh? Older versions are typically vulnerable to CVEs and have bugs.

[karlp]

@neheb don't overlook that new versions merrily introduce new vulnerabilities and bugs too. The bleeding edge has that name for a reason.

[poranje]

Op 24 aug. 2018, om 19:49 heeft Andy Walsh [e-mail] het volgende geschreven:

@poranje Why?
We still talk about a human review process and someone with write access that does the final merge, if all conditions are met. The most notable thing that would change, is that if all is ready to-go and the time has passed, it can get merged by anyone. This removes the need to babysit a PR and hopefully ensures that a PR is either merged or rejected again for some reason.
Just check the PR section, there are PR from 2016, that are still "sound" and passed travis, yet for some unknown reason where never merged or rejected. This is just frustrating from a PR author point of view.

Probably I have misread what you argued.
Still, a trade-off between requirements for merging and security exists and it is paramount that the "official" package repo can be trusted to a certain level. If possible changes to attract more maintainers improves the trustworthiness of the packages repo and certainly should not lessen that.

A multiple tiers approach such as jow proposed with an universe repo where rules for merging are less strict seems to balance this trade-off. Debian with its huge number of packages has this set-up for good reasons.
BTW, is it also feasible to the differentiate authorisations and merging rules when not having different separate repositories but tags in stead?

OT: Justus Uitermark, now professor at the University of Amsterdam, has done some nice research on communities and rules. He found that quite often (informal) communities do better with less or no explicit rules at all, but also that in certain situations these cannot be missed. It's a pity that computer security mostly belongs to the latter category.
Uitermark held a talk at a Wikipedia conference a few years ago in a track on the health of the wikipedia community.

Paul

[danielfdickinson]

I wonder if we should consider current packages as -universe (since that's largely what already is) and maybe have a new repo (packages-standard or packages-coreplus or packages-curated) or something that indicates that is closer to core strictness and nitpickiness and/or explicit active maintainer care (would still need e.g. 3 month rule on maintainership) where it's expected the maintainer has to okay commits, unless there is some security urgency and it's been over a month and it's been tested by at least two people. The current/regular packages feed could included -community versions of the same packages or somesuch that get the bleeding edge and/or regular -universe treatment.

[neheb]

I'm with @diizzyy on this. I see no value is splitting packages in separate repositories.

[pprindeville]

New package submissions that fulfill the package criteria in 1), are properly signed off and pass the automatic build tests may be merged by any committer

@jow- That seems too easy. I think we need some sort of quorum (or at least a threshold of _N_ people for _N_ >= 3) so that a single wild hare doesn't end up taking OpenWrt off in some random direction that the rest of us aren't in agreement with.

There have been increasing efforts to make OpenWrt routers into DLNA servers as well, and to the extent that this doesn't compromise security I'm not going to object. But there will come a moment where the camel's back gets broken...

PRs with maintainer objections / requested changes that do not get reworked within 14 days may be closed by any committer

14 days seems a bit brief. It took me 3.5 weeks to figure out how to get Perl building again when we went from 5.26.2 to 5.28.0 because of some upstream weirdness that didn't account for cross-compilation. If someone shows that they're actively working on a problem and explaining what efforts they've made, I'm willing to continue to extend them grace for quite a bit longer than 14 days...

[pprindeville]

Why?

@Andy2244 Because we typically review the packaging, i.e. the Makefile, patches/, and files/, etc. We typically don't look at the contents of the tarball itself.

What happens if it's a well-known package, but it's not coming from the official repository or download server, but is instead a modified set of sources?

[diizzyy]

My biggest concern with splitting/shuffling packages around is the lack of manpower, we struggle to keep up with reviewing and merging PRs in a timely manner and I'm not very confident in adding more housekeeping chores will be beneficial. IMHO it would be better if we can come up with proper packaging documentation along with guidelines (deadlines, dos and don'ts etc) and once that workflow works we can add more housekeeping...

[danielfdickinson]

@neheb That's because you and @diizzyy happy with packages as -universe (as it basically currently is) - some folks want more stable and curated packages, and you can't keep both sets happy with the same set of rules. I think @chris5560 got fed up with the impatience and lack of QA, and there are probably others like @thess who I'm guessing would like a more solid packages feed than -universe feed.

[danielfdickinson]

@diizzyy I'm not talking about splitting, but starting fresh with a whole new (much smaller) repo for those who want a more stringent feed with more emphasis on individual maintainers, who work at a slower pace than seems to be wanted by the bleeding edge -universe repo we have. I'm thinking of the current packages kind of like a a cross between -universe and debian's -testing or -unstable, and the new feed being more like centos 7. I think @poranje might be another who'd prefer a less bleeding edge packages feed. Maybe @karp too, and maybe @wongsyrone (but those are wild guesses).

[diizzyy]

Exactly what does that mean? http://howfuckedismydistro.com/debian/ ? :-)
We have the release branch which kinda serves that purpose however doesn't seem to get much love by maintainers at all as far as I can tell by looking at the commit history. I'm mostly concerned about who's going to do the actual work...

[danielfdickinson]

@diizzyy the release branch seems to me to be viewed as set in stone except for extremely important fixes (which is a little different than having a slow-moving and stable but still getting new stuff, and more bug fixes than say debian -stable feed).

[danielfdickinson]

@diizzyy I'd rather use Debian for a server than say Arch....just sayin...

[pprindeville]

I'm not talking about splitting, but starting fresh with a whole new (much smaller) repo for those who want a more stringent feed with more emphasis on individual maintainers, who work at a slower pace than seems to be wanted by the bleeding edge -universe repo we have.

@cshoredaniel We'd need a dependency graph so that nothing in the stringent repo required anything that wasn't... Just like for base.

[danielfdickinson]

@pprindeville Yes, that would be a requirement of the new repo is that it would only depend on core and itself.

[pprindeville]

All of this is starting to sound like CentOS vs. EPEL...

[diizzyy]

@cshoredaniel
I've used FreeBSD for many years without any issues which is one package repo unless you go for the quarterly branches ( https://wiki.freebsd.org/Ports/QuarterlyBranch ) which has worked just fine but anyway, how is all this work going to happen?

[danielfdickinson]

@diizzyy For the 'curated' repo it wouldn't likely be folks like you and @neheb who want bleeding edge but individual maintainers who are active in that they will fix and work on issues, just on on necessarily on a time frame that would satisfy the impatient / bleeding edge types. Part of the problem I see for maintainers who care about quality but don't have oodles of spare time, is that they feel trampled over by folks who push stuff with hardly any (if any) testing. It's not like -universe where you have to keep up with multitudes of packages you as a maintainer don't really care about. OTOH who's going to work on -universe if most folks who work on individual packages don't want to become 'at-large' and/or aren't interested in dealing with bleeding edge demands? And who would want to anyway? I certainly am not interested in dealing with -universe except to keep a bleeding edge version of the couple packages I maintain (it would however be a good way to have the development version of NUT which has something like a two to three year release cycle, availaable).

But if there is problem finding folks who want to work on -universe, maybe -universe laxity rules are a problem?

[diizzyy]

The package repo has pretty much no security/vuln awareness at all and most packages that we've touched have been very outdated. Are you suggesting that it's a better course of action to not upstream to the repo and happily ignore those issues? We're fully aware that new releases may introduce new regressions but seeing that there's no secteam (which Debian etc has) the best course is to follow upstream and report issues which seems to be agreed upon as PRs gets merged. There's no way one could test packages on all platforms and in all scenarios, that's just unreasonable but feel free to elaborate on how to accomplish that. Most maintainers/contributors have one or perhaps two different platforms they can test on, the rest is best effort.

I don't mind splitting into 500 repos/branches or whatever but I'm asking if it's realistically possible as I'd hate to see peoples effort go to waste just because someone wants to pull off something grand that isn't going to fly in the first place.

[neheb]

+1. AFAIK I'm the only one that has added PKG_CPE_ID to this repo. Every applicable package in the main repo has it but the other feeds do not.

Not to mention uscan currently lists 42 CVEs.

[danielfdickinson]

That's part of my point, really. The repo is too big and most people aren't interested in with tons of packages that don't know. That why I suggest a 'curated' (smaller) repo and this one that folks (like you) who are interested in doing treewide updates and such on packages that are released to the wild for testing, rather than through much (if any) personal QA first, can take care of. For those of use that want a few well-maintained packages that don't get tromped over by impatience, we get it, and for those that want everything and the kitchen sink that stays as longs at builds; well there is that too.

What I am opposed to is continuing course with the status quo -universe and no 'curated' repo because I don't think the proposed rules will bring things up the quality levels I at least am looking for.

[luizluca]

+1. AFAIK I'm the only one that has added PKG_CPE_ID to this repo. Every applicable package in the main repo has it but the other feeds do not.

Something to add to the "commit instructions checklist" as a suggestion if missing.

Please, don't slit packages. Don't do the oldpackages->packages once more.

openwrt/packages has, by conception, more flexible security constraints. Just check the number of committers. If a user is really concern about outdated packages and malicious commits, openwrt/packages should be used with care (or simply stick with openwrt/core).

What is the idea with the curated/universe repos? Enable both by default? It will not change anything for most users. Disable universe by default? Most users will rant. Those users that know how to enable/disable repos are technically able to evaluate packages source by themselves. Adding a Luci interface will not help a lot, as the user still needs to know that the division exists, their difference and how to enable the universe repo. It will significantly increase the complexity.

I'm against the introduction of any code to a binary release without a human intervention, as it would allow malicious code easily get into user machine. We could, although, after autochecks and objection period pass, accept commits to an already existing package if it comes from its maintainer. In fact, there isn't much more confidence in committers than in maintainers. When openwrt/package was created, I guess that any maintainer that asked for commit access got it.

A new publish CVE is potentially equivalent to "committing malicious code". The difference is that the code was already there. So, if we are against "automatically committing new code", we should also "automatically disable known insecure, potentially exploitable code" until someone tells it is secure.

We can classify a package into some states: maintained, insecure (have known CVE), outdated (have known new stable release). uscan is already doing the job. I would prefer to not touch Makefile in order to avoid a unnecessary build.

It would be nice to have a bot (uscan-bot?) that created issues whenever a new release or CVE appears. If we standardize commit messages related to version bump or patching CVE, the bot could even close the issue by itself, even not mentioning the issue number directly with github dialect. We could use these issues to track the package state (instead of modifying Makefile). If there is an open "security issue" for package foo, opened by the uscan-bot or labeled as "security issue" by a committer or the maintainer for more than X days, consider package foo as insecure and avoid shipping its binary in a new release. After an human evaluation (preferably by the maintainer), a committer could close this issue, with comments, explicitly ignoring the open security issue. The same goes for outdated packages, but they might still be shipped even with open non-security issues.

One problem is that we does not have a direct map between maintainer and github user. It would help if we could add an @< github-user > to PKG_MAINTAINER. If OpenWrt core does not want it, we could introduce a new file for that inside package directory (pkg1/.github-maintainer?).

The guidelines @jow- mentioned guidelines for a package maintainer. I was thinking about something special for a package reviewer limited to those things that still need a "natural intelligence" to check:

1. Check if package source comes from an official source
2. Check any patch for an obvious malicious code
3. Check if travis/CI passes
4. Objection period has passed and nobody mentioned it (could be a label added by a bot)
5. Commit
   Anything that does not requires a "natural intelligence" should be checked automatically (copyright, PKG_CPE_ID presence, etc..).

This could be used for new packages, even when they comes from a user with commit access, specially for the "objection period".

[danielfdickinson]

Since the notion of a smaller 'curated' repo is unpopular, how about, in keeping with what @jow- said in terms of not splitting things, a new menuconfig section and tag e.g. PKG_APPROVAL:=maintainer or something like that that indicates that changes to the packages require maintainer approval (with reasonable guidelines about what constitutes 'active' so that if a maintainer disappears that packages don't get 'stuck', even if the maintainer opted for this requirement) except for CVE's or build failures affecting more than just that package (or the PKG_PRIORITY:=critical)?

[danielfdickinson]

Oh, and can we please define a process for dropping packages: like getting whowever runs uscan to actually submit issues about CVE's not just list them in uscan (for which I can never remember URL anyway). Also uscan should have a feedback mechanism so that if the CVE is not relevant in the OpenWrt package this can be noted (for example occurs in combination with Mozilla NSS but NSS isn't even in packages repo).
Anyway if there are open issues on packages with CVE and no maintainer response, and no one caring to fix it, I think the package should, at a minimum be marked @BROKEN

[poranje]

Very interesting what @luizluca proposes. Following up on that, another idea.

As already stated by @diizzyy, OpenWrt by lack of a security team, often has no much better choice than to follow upstream, also because presumably the contents of upstream sources are mostly rarely, if ever, checked for malicious code. Would tapping onto the acceptance of updates by renowned distributions like Debian or Redhat/CentOS of package's upstream source updates be an option ?

In that case even creating and merging such updates automatically (when other automatic tests are passed) might to some extend be an option. To some extend because security commits are often released together with other changes and are for that reason back-ported onto stable versions or, other way around, up-streamed, by distributions like Debian. Do not really know what would be possible or not, and how to automate this.

Also the idea the that maintainer and reviewer could be distinguished might help to attract more interested people that could help out. It already is possible to add review remarks, anybody can, but many people do not want to be nosy or intruding. Maybe, a separate review, non-committer/maintainer role, lowers the bar.

[karlp]

@cshoredaniel can we please not automatically label things based on arbitrary existance of magical "CVEs" ? They're not bulletproof, and they're not all fatal flaws, and sometimes they're not even valid. Having the filed CVE information available is nice, but auto hiding or worse, removing packages based on them seems wildly authoritarian.

[danielfdickinson]

@karlp hence my comments about ways to address the uscan 'CVE' notation. I didn't mean no human intervention hiding, I meant that the CVE's should result in say an 'issue' to be examined, and if the CVE is valid the package is marked @BROKEN by a human. Sorry if that wasn't clear.

I did say

if there are open issues on packages with CVE and no maintainer response, and no one caring to fix it,

1. Open issue: implies a ticket no one has done anything with
2. no maintainer response ... i.e. no objection like it's not a valid CVE
3. 'no one caring' ... i.e. no action even objecting to CVE.

[danielfdickinson]

I want to apologize for my attempt to guess other people's opinions based on past statements. It's a bad habit of mine that I'm trying to rectify. It's not much of a defence but it was more by way of trying to encourage said individuals to express their actual opinion, even if it didn't come across that way.

[thess]

OK folks - thanks for all this (mostly) useful feedback. I am going to close this topic now as it has been pretty talked out. For the next step, I propose we draft a document outlining a SIMPLE review process/timeline for handling PRs. Topics covering what happens if no-one responds and the possible auto-merging or closing of neglected PRs should be included. Also, noting the difference of handling new packages vs updates to existing.

If no one volunteers to draft such a document, I will make a rough outline next week. Create a PR for a new document: REVIEW.md - Markdown preferred -- Or some other appropriate title.