Ckan: Vetting metadata through a staging process

[Dazpoet]

Problems

When a new version of KSP comes out users go crazy for updated mods, modders are often forced to implement breaking changes because Squad changed something and metadata fails to catch up fast enough. This leads to installation errors which leads to angry users which leads to sad modders who get yelled at.

Our metadata isn't perfect.

Proposal

Implement a 2-stage process for our metadata as a way to ensure better stability and less issues with installations of mods through CKAN.

Goals

• Give our users the best possible metadata
• Give the metadata maintainers breathingroom when new KSP-releases come out
• Become slightly more like Debian
• Give users the alternative of bleeding-edge metadata
  

  How could this be done

Short answer: I have no idea, I don't code.

Long answer: We split CKAN-meta into two distinct branches stable/master and unstable/dev. Everything we get from our .netkans, webhooks and even manual additions to CKAN-meta go into the unstable/dev branch for testing. After a yet to be determined vetting process the stable metadata is then merged over into the stable branch.

Users who like to live on the edge can help us betatest the unstable branch by adding it as a repository but upon doing so will somehow be prompted with a sign explaining how to report issues on github.

How could the vetting process look like

In a perfect world I'd like to have a bot which automagically creates and installs metapackages with the entire non-conflicting modset each time something is added. This would allow us to catch file collisions. A less extreme measure might be to have it atleast run some sort of smart installations trying to install all mods by the same author and/or trying to install mods which filter things that match to some percent (obviously excluding *.cs, _MACOSX and Thumbs.db).

The bot could also run some random modsets every now and then just to see whats up. A more logical method could be to point the current statuschecker to it, which would however not catch file-collisions. I also know that @godarklight and @politas is using some fancy tool to catch things going wrong.

All in all I think we have many smart things going but they should be consolidated and focused onto the unstable branch rather than the entire metadataset. The best would be if something automagically raised a github-issue whenever a tool/bot found an error and immedietly marked the offending .ckan file for manual intervention and to not be merged into stable before a human said ok.

Benefits

Assuming that modders don't go back and change the .zips for previous releases or a force majure occures the vetted files should continue to be stable.

Metadatamaintainers will get a staging area to see how mods will behave in the wild when interacting with each other and not just in the perfect vaccum that is Jenkins testing.

Metadatamaintainers will get some well needed breathing room after a new KSP release where users who frantically want the newest quickly can be shown unstable and at the same time told about the risks.

Questions

• How should the vetting process be done?
• When/How should a mod be marked as stable?
• Should .ckan files generated from metanetkans go through the vetting process?
• How do we make users not just use unstable without understanding what it is?
• Is this even a good idea or just the ramblings of a madman?

[solarsootysmudge]

I might be wrong here but is this not related to the discussion on Metadata checking #1607 ?

[politas]

:+1:

[dbent]

I support this idea. Thoughts on how the process would work (technically and organizationally).

1. Instead of the bot committing to the master branch it commits to branch staging (I think this name most clearly identifies what the branch _is_).
2. For every change the bot _also_ creates a new branch netkan/<identifier>-<version>, e.g. netkan/PlaneMode-1.3.1, it then opens a GitHub PR from this branch into master.
3. Users and CKAN contributors can then choose to opt-in to the staging branch instead of master (i.e. download https://github.com/KSP-CKAN/CKAN-meta/archive/staging.zip).
4. For each open PR from the netkan bot (from the second step) users and contributors can then "sign-off" on the PR (that the metadata is accurate and works) by simply :+1: the PR. After an acceptable number of sign-offs have been received a contributor can then merge the PR.

Extra details:

• Step 2 should really be "create new branch and open PR, or append commit to existing branch".
• The requirements for complete sign-off are a matter of policy. I suggest a point system where a minimum threshold of points need to be received with contributors and mod authors having more points, i.e. if we need 4 points for a sign-off, regular users could have 1 point and contributors and mod authors could have 2 points. Therefore for a sign-off you need 4 regular users, or 2 regular users and a contributor, or 2 contributors, or 1 contributor and the mod author, etc.
• If the metadata is incorrect for whatever reason, update the original .netkan or for one-offs, make a new commit directly on the netkan/<identifier>-<version> branch.
• The staging branch should be an "open secret", its existence shouldn't really be promoted but is there for users advanced enough to know about it and willing to perform sign-offs.
• We should have some mechanism to bypass the vetting process for self-hosted metadata. For example, the bot can inspect the .netkan and see if it has a netkan $kref, if it does commit directly to master. If the mod author screws up their own metadata, well that's on them, and having to have an intermediary step of us looking at the data eliminates two of the big draws of self-hosting (less involvement from us and faster metadata).

[techman83]

After an acceptable number of sign-offs have been received a contributor can then merge the PR.

We could have a bot do this, It'll be in the API data. Weighting votes might be overkill and does make it a little more complicated (simple mater-o-code though).

My only major concern is that our Metadata hearders, primarily @plague006, @politas and @Dazpoet do an awful lot of Checking/Merging/Testing already. The reason it works so well for debian is that they have a massive community of contributors and new features are released yearly.

Some questions:

• How big of a problem is it?
• How much faulty metadata is making it to the users?
• Is it only a handful complicated install mods that are a problem?

As an in between, we could add a flag to particularly tricky mods that go in for vetting?

As for writing a bot, whilst doing so for random mod testing against the entire modset for version X would possible, that'd be a huge amount of work to be reliable and use an incredible amount of compute to perform.

So in summary when calculating Benefit vs. Effort, is it worth it? The following XKCD is tangently appropriate.

[dbent]

Like you said since this will mostly affect @plague006, @politas, and @Dazpoet I'll leave it up to them to comment on whether it's worth it or not. But I will say even if bad metadata slipping through isn't that common, when it does happen it causes a whole bunch of headaches, pain, and diminishes our reputation likely out of scope with the problems actually caused.

[politas]

One of the biggest problems we have is that everyone wants everything straight away. And I totally understand that. This idea seems to address that, especially if automated metadata checking can be merged into it, though @godarklight 's method has some limitations, especially in regards to virtual config mods and separated out agency/common asset/flag/etc mods.

[politas]

If a points system as described above is used, perhaps a successful pass by an automated metadata checker could be worth 1 or 2 points, depending on how thorough it is.

[techman83]

The NetKAN-bot already does a schema check, we can implement other checks as well. There are plans to re-architect it, leaving room for a test suite wouldn't be super difficult.

@Dazpoet's plan is a solid one and a definite end goal, it is however quite ambitious. As in interim we should identify exactly what the issues are, how frequent they are and address them specifically.

If it mostly affects our manual indexing, lets beef up our testing. If it's zip files changing between releases and generating bad installs due to path changes, lets add some tests into pre-commit checks the bot does, we could even have the bot submit an issue back to netkan with "Hey the zip structure no longer works", then it gets fixed by a human, passed by jenkins, merged and instantly inflated by netkan again.

Just some proscati-ideas (I'm supposed to be doing my homework :laughing:)

[plague006]

"Hey the zip structure no longer works"

At the moment we catch that in http://status.ksp-ckan.org/ just fine. Having the indexer create an issue or whatnot would be nice but IMO it's a very low priority.

There are 3 common 3 cases that I would like to see caught by the indexer (or some other automated device) if possible. And I do realize some of these probably require significant coding.

1. A new release has a ksp_version or ksp_version_max lower than the current KSP release number. The most common cases for this are mods we index via github that have hardcoded ksp_versions in need of a bump.
2. A new release has a version < the version before it. The most common cases for this are versions losing their v or using odd versioning strategies which require an epic epoch bump. (I imagine this would be most efficiently done by keeping a local record of the most recent version rather than comparing it against the full set of ckan-meta metadata but I could be totally wrong about that).
3. When the indexer picks up a new release it compares the tree/file structure of the entire .zip and compares it to the previous version. This could help us catch instances where new dependencies are added/removed or new files are introduced that may conflict. This could potentially generate a lot of false-positives if it turns out mods frequently add new files and might be something to warn about while committing anyways.

I think those are probably the biggest 3 metadata errors that enter our the repository which I could imagine being picked up automatically.

I'm generally against any system that slows down metadata commits indiscriminately. If there's extra work or time required I'd like it to be well-founded.

[politas]

Of course, the other issue is when a mod adds a new or removes an old dependency, but that's really hard to check automatically. @godarklight's tool for testing that relies on bundling, and gives false positives for many common cases.

[BobPalmer]

Some carry over suggestions to improve this.

Designate a primary maintainer. Either the author, or a volunteer (not necessarily CKAN). We test changes in staging first. Not sure how we do this with self hosted - thoughts? (A good example would be a dependency restructuring where I want to make sure the whole thing works first).

Could we point staging to an alternate source? i.e. point staging to a dev branch on our self-host repo?

Then it is pushed to the official repo once blessed by the maintainer.

In the case of major KSP releases, it may be wise to have a lockdown period to give staging some time to settle. A LOT of the support issues are because of an index happening while everyone is scrambling to get their stuff updated.

[dewiniaid]

If this is implemented, as an extension would it make sense to add into CKAN a mechanism for collecting input from users regarding whether a particular mod is working or not in their particular environment? This can give some us some valuable data quickly when new versions come along:

• New mod version, 100% of users report it as broken: Likely metadata screwup
• New KSP version: Can use stats to get a quick idea of what mods were broken by the install, which mostly work, and which are unaffected.
• 15% of users reporting a mod not working: Maybe it's a conflict with another mod, and the author can use the stats (hmm, they all had SuperSpecialMod installed too?) to narrow it down.

I'm going to open a more thought out issue on this in a bit when I have more time.

[Cphoenix]

In the case of major KSP releases, it may be wise to have a lockdown period to give staging some time to settle. A LOT of the support issues are because of an index happening while everyone is scrambling to get their stuff updated.

This is a big issue with CKAN whenever a KSP release comes around. A staging area would certainly help. A little lag time is preferable to a crap-ton of support requests for everyone from everyone. People with modded installs should be used to waiting anyway after KSP releases because it takes mod authors time to recompile against the new version and do some testing. Hell, with a staging area on CKAN, you might even get a few more eyes to help do that testing (from peeps who are helping maintain the staging area). That might be nice. :smile:

If coupled with a friendly pop-up reminder in the CKAN application itself whenever KSP updates to warn users of the lag time, it might cut-out a bunch of those support requests (like "why isn't your mod updated on CKAN?"). Something like "KSP just got updated! Please be aware there will likely be some lag-time before you see mods updated in CKAN, as the CKAN team and mod authors all sort through the changes."

If users want everything right now...well tough. They can go to the forum page that's linked right in the application and install it manually or roll their own CKAN file (the process for that is straightforward for anyone who reads the wiki). Making sure the metadata is correct before it gets pushed to everyone is worth a few days (or a week) of lag time.

[solarsootysmudge]

Another advantage of having a staging area is it helps us develop web based tools. Which will automate the process later on. Right now it is early days but interesting times. https://github.com/KSP-CKAN/metadata-webtool

Vetting data is one thing. Getting in ready for verification is another job. Collecting every bit of metadata is time consuming. As is typing it all in. This should be farmed out the rest of us with more time on our hands. Under final adult supervision of those doing the gate keeping.

Later on this might be useful for pushing out PR's which are going straight to the live repo. To minimise the work load of processing new contributions. However right now output from these tools should have the option to be sent into a staging area and peer reviewed.

So far I have had to help test the tool as an end user in private. I do not feel confident in helping testing this with the live repo. Without the extra safety net that a staging area would provide. Considering that some testing would involve me deliberately making mistakes see what happens. Before letting everyone else go at it full scale. Right now the work around is for me to post "training flag mods" as test data. So if I mess up no harm is done.

Even if we want to help to develop the skills of more people to do it manualy. We need to have a safe "play area" that does not involve a process of PR's straight to the main live repo. There has to be more maintainers inside the project. Having a safe play area helps attract them. I hope. Lets be honest without more people like @politas and @plague006 we are all stuffed long term. Where are the new recruits coming from?

Then there is "development Sprint speed vs CKAN metadata lag" problem. It is possible to publish a mod and dependencies. Get good feed back on the forums. Then change the mod and publish it again. Before CKAN knows what is going on the first time around. This makes debugging hard when half the metadata is now wrong. It also confuses end users.

Sometimes there needs to be a "cooling off" period on new mod updates. Especially if they are complicated updates. It would be nice if a mod author / maintainer could park things in the staging area before a final commit into the live repo. Some authors might find this a useful feature to have. It is just another tool to help if they want it. It gives some feedback inside CKAN without the whole drama of doing it all live.

Just some thoughts. Please don't eat me.

[Cphoenix]

Lets be honest without more people like @politas and @plague006 we are all stuffed long term. Where are the new recruits coming from?

_slowly raises hand_

Seems like no one is pointing a gun at me yet so...I'd say well intentioned users. But if my short history here is any indication, as well intentioned as new users might be, they get stuff wrong. Probably very wrong.

This is a valid concern and something I also brought-up here. A more-safe area for newbies to make PRs in and not to the main repo/branch (however you want to handle it) makes the whole prospect less problematic.

Please don't eat me.

But you look so tasty! :disappointed:

[techman83]

I have had some time to ponder this and my earlier opinions stand, we simply don't have the volunteer base to to mimic a Debian style experimental/testing/staging, along with the pace at which mods move this would grind releases to a halt likely causing mod authors to be further deluged with requests about when something will be released on CKAN.

The majority of the mods are pretty straight forward and the process @Dazpoet has proposed is wonderful but ultimately overkill. However there are a number of mods with install processes that require more care or lean on functionality that changes on KSP version bumps, causing things not to work for the consumers of the mods and a support burden for the authors. Neither situation is ideal and we can do better.

I will put together a quick high level design/implementation proposal that we can discuss further.

[politas]

Thanks, techman83. We need a relatively streamlined process, but I think we do need some kind of staging process. I look forward to seeing your proposal.

[solarsootysmudge]

@techman83 just an idea. Probably a controversial choice but here goes. Probably a stupid thing to say but here goes.

Can the mods be split in distinctive groups already with existing working code. Or previous suggested update code. Something like using the version comparator to help split the work load?

GRAS without staging set with submitted metadata flag
GRAS with staging as above but without flag
STRICT always staged as depends on specialist knowledge of current game version
YOYO mode. Super user by pass but zero support option With all blame falling on the end users.

Or in a case where a generic max_ksp_version is set by mod authors to force updates. Staging is relaxed or by passed. However where this does not appear be the case. Staging is activated.

Yes I know that I am being super simplistic here. I am just to get my head around staging where there is only a tiny numbers of volunteers. It just seems to be overkill for a large groups of mods. However is still required to be in place for all the reasons presented here.

I don't know if it possible to have cake and eat it here.

[solarsootysmudge]

@politas Have looked again how much influence this is going to have. Yes we need it. Unfortunately it will not fix a lot of the problems. That would still happen after staging. Just because we agree the metadata is correct today. Does not mean that it stays that way over a period of time. The reference here is the root cause analysis report I sent you. Agreed this good idea but it's impact on quality may be less that some people are expecting. There still a case for monitoring and capture of new changes beyond staging.

[Blu3wolf]

That shouldnt be the case. A correct .ckan file should not become incorrect tomorrow - else it was incorrect today.

[solarsootysmudge]

Unfortunately @Blu3wolf this not the case. There is overwhelming evidence to support the case that the situation needs to be check again after any staging process. The problem is triggering the check. It is up to @politas to make the call here. To accept or reject that evidence. Now I am sure you going to ask to see the evidence and judge things for yourself. I think that would be a good idea. You would have really valuable input here. Unfortunately it's down to @politas. If says he no it all goes in the bin unseen. Somethings should not be shared without permission.

[politas]

I think you two are talking across each other. A correct ckan file should indeed remain correct, but netkan files can often become incorrect.

[solarsootysmudge]

We are indeed. Sorry for my part in that. The only reason to mention the decay in the value of a netkan file here. Is it may some bearing on past questions posted here.

Some questions:

How big of a problem is it?
How much faulty metadata is making it to the users?
Is it only a handful complicated install mods that are a problem?
As an in between, we could add a flag to particularly tricky mods that go in for vetting?

As for writing a bot, whilst doing so for random mod testing against the entire modset for version X would possible, that'd be a huge amount of work to be reliable and use an incredible amount of compute to perform.

So in summary when calculating Benefit vs. Effort, is it worth it?

[Blu3wolf]

Well, Im still proposing the same solution I have been for a while. Staging, retain netkans only for the experimental repo, push valid .ckans to stable after they have been checked by their maintainer.

The real question comes back down to policy. How strong are those strong guarantees on compatibility meant to be?

[Cphoenix]

Well, Im still proposing the same solution I have been for a while. Staging, retain netkans only for the experimental repo, push valid .ckans to stable after they have been checked by their maintainer.

Oh. I think I finally understand what you're getting at. (i.e. it "clicked") I'll make a flow-chart later today or tomorrow to lay this out (and make sure I'm understanding you right), so to speak.

[politas]

The real question comes back down to policy. How strong are those strong guarantees on compatibility meant to be?

That's a good question. We've been drifting away from that guarantee in favour of completeness and rapid availability (and I'll completely own up to being an active part of that), and that's one point I really want us to move back to. I don't think we can go as far as guaranteeing a functional KSP, but we ought to guarantee that CKAN installations will be as good as a correct manual installation.

Netkan inflating to staging, release from staging to main repo by maintainer sounds good to me,though we may need to define multiple maintainers for some mods, or have a fall-back solution if the original maintainer is unavailable. If we can find a way to build @godarklight's metadata checker in as a checking aid, that would be fantastic.

[Blu3wolf]

Multiple maintainers for a given mod makes sense. A lot of mods are simple, or small, and easy to vet as working or not working. More complex mods, with lots of functionality, require a more in depth vetting - depending on how committed the maintainer needs to be, to those guarantees on compatibility, and to the more recent implicit guarantees to modders, on validity of installation.

[techman83]

I'm going to refer back to here, there is a lot of back and forth about what/how without addressing the scope of the problem.

@dbent has suggest a really great process and I'm likely to propose a version of that.

I feel quite strongly that we should enable this on a case by case basis. A lot Players/Modders have grown accustomed to the speed at which we can get metadata out, but there certainly mods that have a complex set of requirements and modders that need some room to breath when releasing new versions.

The added bonus of a targeted approach is that we can get our processes nailed down without overwhelming the @KSP-CKAN/wranglers or bringing our Metadata generation to a screaming halt.

[politas]

I agree on the case-by-case basis at least partially. I think our best option is that when we find mods that change their directory structure and/or dependencies, we have a two-pronged approach. First, remove the #vref from the .netkan to stop new versions being automatically added, and add a comment explaining why (or mark it for staging if we are greenlighting mods to skip staging) Users will report new releases to us in the CKAN thread or in GitHub issues.

Second, a message should be sent to the mod author advising of the issue that CKAN may have been causing and engaging in a conversation aimed to find the best solution going forward, whether that be leaving things as they are and CKAN maintainers checking each release in detail, or author-hosted metadata that they can update as needed, with assistance from us as required to make the metadata correct.

Now, which mods need to go through the staging process and how much checking they need is a complex question. A metadata flag to turn on staging would let us use either a green light or a red light process as to what is the standard behaviour.

@dbent's points based system seems good to me, especially if we can get @godarklight's metadata checker to provide points for complete matches (or a zero-point diff report).

[BobPalmer]

Just tossing this in as well. One very good use case for the staging area will be to test out metadata/installs before pulling the trigger by mod authors. i.e. I do not use CKAN, but I would use a staging repo to make sure my stuff installed correctly, especially if I am about to do a ton of changes.

[techman83]

@BobPalmer - Excellent, I envisage that's how something like this could be used and be most beneficial.

@politas - Which mods, lets assess when we have problems, I envisage it'll be a flag in the metadata, we can submit a PR with the flag, reasons why (author request, frequent zip changes, unstable dependencies etc) and go from there. As for automation for merging can come later. It's a separate process that can be integrated into our GitHub Webhooks.

[politas]

@techman83. I agree completely. Lets get the basic functionality in place and do what we can with process initially. That'll tell us where further development will provide the best return for our efforts. Are we ready to start implementing? What tasks need to be done, and where can people help?

[Blu3wolf]

One very good use case for the staging area will be to test out metadata/installs before pulling the trigger by mod authors.

Im confused. Isnt that the only use case?

[Cphoenix]

Im confused. Isnt that the only use case?

Not at all. For the occasional mod author like @BobPalmer who wants to deal with the metadata themselves, this is a good thing. (Can push changes to something, test if it works, then clear it to be released to everyone. Fewer headaches over broken installs from CKAN.)

But for maintainers, it's also a huge boon. It means that any mod that's being submitted the first time, or had a major change, or for whatever other reason, can be put somewhere and tested first before released to everyone. For most (hell, almost all) mods this'd likely be a very quick process, but it's great to have that intermediate step as a "just in case" safety-cushion. Right now, the only real full-on test of the system is to release the mod to everyone running the CKAN app (a maintainer manually using the generated CKAN is ill-suited for this purpose because it doesn't actually test the entire distribution chain). Not exactly ideal.

And like everything in computing (or perhaps just policy in general), there's probably use cases no one has thought of yet.

[politas]

One question we haven't addressed from the initial post is:

How do we make users not just use unstable without understanding what it is?

What do people think about a button in the main CKAN settings dialog box that activates the unstable repository only for the current session? We could still use the add a repository feature to make it long-term available for maintainers. Just an idea.

[Blu3wolf]

Not at all. For the occasional mod author like @BobPalmer who wants to deal with the metadata themselves, this is a good thing. (Can push changes to something, test if it works, then clear it to be released to everyone. Fewer headaches over broken installs from CKAN.)

But for maintainers, it's also a huge boon. It means that any mod that's being submitted the first time, or had a major change, or for whatever other reason, can be put somewhere and tested first before released to everyone. For most (hell, almost all) mods this'd likely be a very quick process, but it's great to have that intermediate step as a "just in case" safety-cushion. Right now, the only real full-on test of the system is to release the mod to everyone running the CKAN app (a maintainer manually using the generated CKAN is ill-suited for this purpose because it doesn't actually test the entire distribution chain). Not exactly ideal.

Well, that doesnt really answer much. Those are still the same scenario, and same use case. I guess it doesnt matter, anyway.

How do we make users not just use unstable without understanding what it is?

Do we really need to? Whats wrong with the current system to making users not use additional repos they dont understand - disabling them by default, and hiding them in settings?

What do people think about a button in the main CKAN settings dialog box that activates the unstable repository only for the current session? We could still use the add a repository feature to make it long-term available for maintainers. Just an idea.

So long as the proposal is not the removal of the add a repository feature, that sounds good. Appropriately named labels would go a long way for this.

[BobPalmer]

The use case of staging, as noted by @Cphoenix , includes allowing the CKAN team itself to vet things out before hitting the main repo.

In any case, getting to this repo should be a pretty serious concious choice and not something the average user can stumble across by accident. I would add that it should not, if possible, even be in the default client. Because I assure you... the second the users figure out there's an easy way to get 'early access' to a mod, they are going to start doing it if there is not a large barrier to entry. Probably the best analogue would be recompilation of a mod for use in x64 vs. just downloading some 'unfixer'. And the amount of grief that could cause us as modders would be pretty spectacular.

Also (and it should go without saying, but saying it just in case) de-listing requests should extend to all CKAN repos - there was a short discussion on the KSP forums about having a repo available that included all mods (including those de-listed), and hopefully that particular loophole has been sealed.

[Blu3wolf]

I would add that it should not, if possible, even be in the default client.

Please dont enforce a fork of CKAN.

there was a short discussion on the KSP forums about having a repo available that included all mods

Plural case would have been more appropriate, but not all of them were public.

[BobPalmer]

No need for a separate client, just make it a separate download that one has to actively go and pick up with appropriate technical hurdles beyond 'click these three buttons'. i.e. something that is meant for, and used by, the CKAN team and modders interested in testing out there work without it getting into the wild. Otherwise, it's going to be a train wreck, for both the modders and the CKAN team.

Regarding the 'all mods whether you opt out or not' discussion, let me be very clear. If released either by or with the wink and nod approval of the CKAN team, that repo would be a pretty blatant loophole around the de-list option and would pretty much kill the point of opt-out and set us back to where we were before the great kerfuffle.

I agree with an earlier comment by @politas - to paraphrase, providing accurate data and a reasonable guarantee that stuff works is more important (and a more appropriate focus) than speed of delivery.

[BobPalmer]

On a related note. I wish I had a staging area right now. because having to bump versions each time I have an 'oops' with changing a core dependency is going to really be a drag.

[Blu3wolf]

No need for a separate client, just make it a separate download that one has to actively go and pick up with appropriate technical hurdles beyond 'click these three buttons'. i.e. something that is meant for, and used by, the CKAN team and modders interested in testing out there work without it getting into the wild.

So, a separate client. For it to work the way you want it to, it would need to be either under a restrictive license, or not be open, or both.

In any case, getting to this repo should be a pretty serious concious choice and not something the average user can stumble across by accident.

Completely agree, incidentally.

I would add that it should not, if possible, even be in the default client.
make it a separate download that one has to actively go and pick up with appropriate technical hurdles

Right up until the point where its FOSS. Someone can easily flatten those hurdles, is the point Im trying to make.

Regarding the 'all mods whether you opt out or not' discussion, let me be very clear.

Likewise. Assuming such a thing existed, it would constitute a loophole, whether it had the approval of the CKAN team or not. Seeing as the metadata is public domain, their approval would not be required exactly. Of course, so long as things remain hypothetical, its a moot point.

I agree with an earlier comment by @politas - to paraphrase, providing accurate data and a reasonable guarantee that stuff works is more important (and a more appropriate focus) than speed of delivery.

Indeed, its nice to see a return to pjf's stated intent for the project, as a focus.

On a related note. I wish I had a staging area right now. because having to bump versions each time I have an 'oops' with changing a core dependency is going to really be a drag.

It would not be overly difficult to set up a simple such area, as discussed above. We can certainly agree it would be a good idea to have such an area, over and above the existing testing you can do offline, with the tools already available.

[BobPalmer]

No, it would not need to be under a restrictive license. FOSS has as much to do with this discussion as ham does with hamsters. There is a world of difference between what someone can do with a fork of this project (which is not the subject of discussion), and what the CKAN team should do with this project (which is the topic at hand).

Access to the official CKAN staging repo (and I maintain there should only be two - staging (for modders and staff to test, and the public production repo) should be set up in such a way that there is a technical hurdle to setting up the client to read from it (i.e. not something a user will just stumble into) for the reasons stated above. That's pretty much it.

[techman83]

@dbent already addressed this. It doesn't need to be any more complicated than adding a repository.

The staging branch should be an "open secret", its existence shouldn't really be promoted but is there for users advanced enough to know about it and willing to perform sign-offs.

It also doesn't even need to be a separate repo, in fact I'd suggest that'd be over-complicating things for no added benefit. You can instead use a link to a staging branch like this: https://github.com/KSP-CKAN/CKAN-meta/archive/testkan.tar.gz

I'm a fan of starting simple and building bridges when we need to. Lets not build the golden gate when all we really need is a ford and a four wheel drive.

[techman83]

I've done a bunch of work on the Indexer to support an initial staging process. KSP-CKAN/NetKAN-bot#55

There likely isn't many Perl developers among us, the indexer has pretty good test coverage and I'm comfortable merging/deploying it if it seems to address the initial problem set.

[Cphoenix]

This is how I understand the current ideas about vetting metadata being discussed so far. Most of what is done in this chart can be done right now, at least on the GitHub side. The CKAN app may just need some tweaking so one can pull from CKAN/staging.

It'd (hopefully) ensure that only good metadata gets through to CKAN/master, but it'll also incur a lot of overhead for maintainers. Some way to "fast-track" certain mods so they are inflated from their netkan file directly to CKAN/master might help, as long as they are simple. Obviously ideally we'd have a way to test mods automagically instead (though that's a bit hard to do when the rules for manual installs can almost be literally everything). I also see a lot of cherry-picking in future but that's a very minor concern.

Moving a mod's ckan file from CKAN/staging to CKAN/master can probably, most of the time, just be done with a cherry-pick. For large/complicated/high-dependency mods like ModuleManager or the like probably best done with a PR, so everyone takes a look. If one of those mods breaks a whole lot of other things go with it.

Sorry that took a while. Spare time as common as a flying pig and all that.

[techman83]

@Cphoenix that's pretty close to the workflow. However my proposal would be targeted staging, rather than staging everything.

Installing a staged mod requires no tweaking to CKAN Client, just add the staging branch as a repository (with the understanding that _do not use this branch to manage your mods_).

Also moving from Staging isn't necessary, the bot creates a separate branch + Pull Request. So it's a case of verify the install, merge the PR.

[Cphoenix]

@techman83 Here's the xml file if you want to tweak it. I used draw.io.

[techman83]

@Cphoenix neat! At this stage looking for feedback on the proposed changes in KSP-CKAN/NetKAN-bot#55 - @nobodyhasthis @Blu3wolf @BobPalmer seemed to be the most active in discussing the concept, thoughts?

[Blu3wolf]

So in the proposal, metadata in the staging branch would be tested how? Im not sure what the testing workflow is meant to be.

[techman83]

Add the staging branch as an Extra repository, ie: https://github.com/KSP-CKAN/CKAN-meta/archive/staging.tar.gz

[Blu3wolf]

In that case, that seems to be a most excellent proposal.

[solarsootysmudge]

@techman83 looks good to me. Can I please ask a few more questions?

This does not work if "x-netkan-staging" : false which is the generic setting for most mods. This is targeted staging. Do we agree that most well established mods will not need to be staged?

If a mod author / maintainer sets "x-netkan-staging" : true it stays that way unless they change their mind. This is choice made for any of the reasons we have discussed earlier in the issue?

As it stands all good but an idea for future..

If a mod trips up during bot checking http://status.ksp-ckan.org/ Can the bot set x-netkan-staging" : true. After a resonable delay to account for normal update behaviours. If it looks like a neglected error can this summon help?

In effect it is not a new set of metadata needs checked. It is old set that was working before and has now failed because the mod has be updated / changed without informing the CKAN team. Just becuase something can pass CKAN/staging to CKAN/master. Does not mean it always stays there. If something does go wrong. Sometimes the metadata maintainers are the last to know. Is there an edge case for moving some mods in the other direction. CKAN/master to CKAN/staging?

[techman83]

This does not work if "x-netkan-staging" : false which is the generic setting for most mods. This is targeted staging. Do we agree that most well established mods will not need to be staged

I think so, no need to unnecessarily create work for ourselves when most things tick along without any dramas.

If a mod author / maintainer sets "x-netkan-staging" : true it stays that way unless they change their mind. This is choice made for any of the reasons we have discussed earlier in the issue?

Probably worth documenting the policy down, but I have no personal objection if an author prefers to stage their mods. Maybe we should comment why something was staged, just for future piece of mind.

If a mod trips up during bot checking http://status.ksp-ckan.org/ Can the bot set x-netkan-staging" : true. After a resonable delay to account for normal update behaviours. If it looks like a neglected error can this summon help?

Potentially, though the bot operates on the KISS Principle and currently a mod that doesn't pass validation doesn't get staged. What might be better is to have a separate service consume the state data and notify if something is broken for an extended period of time. The state data is a JSON object.

Is there an edge case for moving some mods in the other direction. CKAN/master to CKAN/staging?

There is no clean way to do that. If we're really in a bind we could remove the offending CKAN and then set the NetKAN to Staged.

[politas]

How much of a hassle would it be to use "x_netkan_staging" values of "always" "once" and "pass"? Then we could have changing from "once" back to "pass" as part of the approval process.

[techman83]

With the current indexing process, rather difficult. Possibly not something the indexer should be doing, that could tie in with the webhooks or something.

[techman83]

As far as I'm concerned KSP-CKAN/NetKAN-bot#55 is good to merge. However before deployment it'd be great if a couple of mods that are destined for staging be flagged as such. So that we get no surprises for the first run that contains them.

[politas]

Well, that's one, for sure. @BobPalmer, would you like to set your USI mods to staging? And @NathanKell, how about RealismOverhaul mods?

[politas]

@KSP-CKAN/wranglers , any other mods that have tricky Dependencies that will probably need to be checked on upgrades?

[techman83]

@Cphoenix @nobodyhasthis @Blu3wolf @BobPalmer - The initial concept has been deployed to the indexer. If a mod is flagged as staged it will go through the new workflow.

[solarsootysmudge]

Looking for stuff that is finished. To tidy out of backlog again. Can this close now?
Ref https://github.com/KSP-CKAN/NetKAN-bot/pull/55