CKAN 🚀 - Opt In / Opt Out | Required changes to CKAN.

The representations made by the first part of that post are inappropriate, and false.

(edit) Ahh good. I see he edited it.

BobPalmer on 24 Jun 2016

You mean the part I changed after having noticed I made a mistake?

VasVadum on 24 Jun 2016

I like the greyed out idea as an additional feature
provided that a modder that do not want to be listed in any way shape or form has the option of being completely removed from any CKAN related list

Sigma88 on 24 Jun 2016

I also suggest that you include a method for authors to write their own meta-data. No more allowing non-authors to submit data to CKAN too. Authors must be the ones to submit their information to CKAN, and a file that comes with mods can correct any meta-data CKAN has on the mod.

The only problem with this approach is the number of cases where authors don't care if their mods are on CKAN, but don't feel like managing it themselves.

As far as opt-in/out/list goes, I think it might be better implemented as:

Supported: The author(s) of this mod officially support installing through CKAN-based installs.

Available: This mod is available for CKAN-based installs. There is no statement of support (in either direction) by the mod author. This is the default for all mods that don't explicitly state a policy.

Unsupported: This mod is available for CKAN-based installs, but the authors will not provide support for them.

Prohibited: This mod is not allowed to be installed via CKAN.

External: This mod is only available for CKAN-based installs when using a separate repository. (A pointer to the new repository should be in the metadata).

With the above in mind, here's how I think this can be fixed:

Add a new "policy" field to the metadata for updates, reflecting the above:
Add a highly visible "maintainer" field to metadata that reflects the person who is responsible for the metadata. The special value "author" or "authors" indicates that it is strictly under control of the mod author(s); PRs from anyone else to change the metadata should be rejected automatically (exception being abandoned mods under a new maintainer). This gives both players and mod authors a point of contact if there's issues (technical or otherwise) with CKAN metadata.
To reduce burden on mod authors, see if KSP-AVC's spec can handle embedding partial metadata within its own JSON format -- namely the policy field. (My theory is it will ignore fields it doesn't recognize, but this should be formalized). This allows mod authors to clearly specify their own policy while not increasing the number of files they have to update on a release.
Refuse to install a mod version whose policy field states that it should not be installed by default. _(Prohibited)_ Find a way to make it where someone who's savvy enough to do their own troubleshooting (including uninstalling from CKAN and doing an author-supported install prior to posting) can workaround this -- either by compiling CKAN themselves with the relevant options disabled, or by manually editing a configuration file. (CKAN being open source means there's no way we can truly prevent this anyways). This should still very loudly warn that the author does not support this method of installation and does not provide support for CKAN installs; ideally the warning requires more effort to dismiss than merely clicking "OK".

dewiniaid on 24 Jun 2016

👍3

höhöhö.

It's about time that this policy got itself revised, IMHO. I, too, would support the additions that @dewinaid, @sigma88, and @VasVadum propose.

DuoDex on 24 Jun 2016

CKAN has officially become a threat to the community because mod author are beginning to change their mods over to All Rights Reserved in order to stop CKAN from listing.

@VasVadum @pjf explaiend all this stuff very well in #1078

IMHO This whole discussion about opt-out is pointless (and not motivated from legal standpoint). Mods authors should learn how to use licenses properly.

This will hurt modding in the long run

No, it will not. Few moders will sentece their mods to disappear in future, oh well. ARR mods will disappear sooner or later, that's why I don't use it. There is plenty FOSS mods.

TeddyDD on 25 Jun 2016

👍6 👎3

any mod i make has been changed to ARR until ckan respects modders

legally speaking ckan crosses into the same territory has torrents, i have no issue with how ckan software itself works, i have no issue with ckan auto fetching mods / having an opt-out if a modder decides ckan is an issue to him. what i do have issue with is those who run ckan having policy to abuse "what they can legally do" without respect to mod authors. i disagree with ARR whole heartedly, thinking it's rude to the community, as ARR blocks other mod authors from learning / using / expanding upon what i (or other authors) learned / expanded upon from the community in the first place, and yet its the only option i see when disrespect is purposely given by a few in the community.

add at least as far as "de-list mod no questions asked", and - just given the past disrespect in policies - a policy to not re list if author had it removed. i like what ckan does, i accept bugs in software, i dont have to accept disrespect towards myself or others in the community.

anxcon on 25 Jun 2016

👎4

I like how zealously you keep doing all your best to keep all the FOSS mods listed on here tho ;)

Sigma88 on 25 Jun 2016

legally speaking ckan crosses into the same territory [as] torrents [...] [I] disagree with ARR whole heartedly, [...] and yet its the only option i see when disrespect is purposely given by a few in the community.

Torrents involve distributing (possibly-)ARR content that you do not have a license to distribute. CKAN does not handle the actual distribution of mods, it only points at a source that is (presumably) authorized to do so. Even in the case of an ARR license, CKAN is still only deep-linking to your content -- a practice that -- while controversial, has generally been ruled legal in various jurisdictions[1][2][3]. So ARR does not solve this issue at all (though it was a valid option in cases of e.g. the 64-bit debacle)

i have no issue with how ckan software itself works, i have no issue with ckan auto fetching mods / having an opt-out if a modder decides ckan is an issue to him. what i do have issue with is those who run ckan having policy to abuse "what they can legally do" without respect to mod authors.

To clarify, you have no _technical_ issue with CKAN; the problem is more one of policy and the human factor? (This is not an attempt to derail the discussion or diminish its importance, just an important distinction that I think needs to be made.)

Technology is rarely the solution to what is -- at its core -- a human problem. We can come up with a technical solution to aid in implementing and abiding by some set policy, but technology cannot replace policy as a whole. My earlier post, for instance, only holds weight if CKAN contributors actually abide by the intent of the policy definitions specified.

I'm personally of the mind that a complete and total opt-out of CKAN is harmful to everyone:

It's harmful to authors, because it means people are either less likely to install their mods or less likely to update them before reporting problems. CKAN updates do have a lag time (though perhaps ways to improve that can be developed), but they do at least generally show up for the mods it can detect updates to.
It's harmful to players, who don't have many other mechanisms for comprehensively browsing a list of available mods, nor quickly checking for updates to all of their mods at once. (KSP-AVC is often touted as a solution to the latter, but requires explicit author effort and support -- and mods have shipped with incorrect .version files before)
It's harmful for CKAN as a whole, since less mods listed on it translates to less usage overall.

If authors have the ability to restrict CKAN solely to update checking (as opposed to delisting entirely), it mitigates most of these issues.

[1] https://apps.americanbar.org/litigation/committees/intellectual/news_0209.html
[2] http://fairuse.stanford.edu/overview/website-permissions/linking/
[3] https://en.wikipedia.org/wiki/Copyright_aspects_of_hyperlinking_and_framing#History_of_copyright_litigation_in_field

dewiniaid on 25 Jun 2016

👍2

After reading forum I think I underestimated scale of problem. It's sad that good tool becoming a threat because of users that can't read and spam modders with CKAN issues :>

TeddyDD on 25 Jun 2016

👍2

@dewiniaid first off, torrents are do nothing more than ckan, a torrent is the equiv of ckan meta, instructions on where to find the file, information on the file, etc. what ckan does is go a step further, accually handling distro of the file itself. it does not host it, but it does aquire it and deliver it, which falls under distribution. if you goto the corner drug dealer, distributes drugs, taking them from a higher dealer and delivering them to you. yes ckan itself uses code to deliver to you. that is why earlier (before bit torrent protocol) programs got hauled into court, the user was hosting the file, but the program (and sole reader of "meta data" was downloading and delivering the files. torrenting nowadays has the niche they dont accually download / distribute the file, as they don't touch the file.

2nd yes, my issue is the human factor and disrespect that standing policies have towards the community. disrespect drives authors away, which hurts the community far more then having to manually click the download link on forum post, and unzipping

that said, ckan has made no effort to lessen that burden aside from full auto install, which granted is a noble goal, but software will always have bugs, and lessening the burden of manual install (buttons to open appropriate folders without navigating, etc is the simple minimum step) can be nearly as good

anxcon on 25 Jun 2016

👎2

Refuse to install a mod version whose policy field states that it should not be installed by default. (Prohibited) Find a way to make it where someone who's savvy enough to do their own troubleshooting (including uninstalling from CKAN and doing an author-supported install prior to posting) can workaround this -- either by compiling CKAN themselves with the relevant options disabled, or by manually editing a configuration file. (CKAN being open source means there's no way we can truly prevent this anyways). This should still very loudly warn that the author does not support this method of installation and does not provide support for CKAN installs; ideally the warning requires more effort to dismiss than merely clicking "OK".

This is the only suggestion I take issue with. Mod authors shouldn't have the authority to dictate what tool a user can use to install their mods. Don't support it, fine. Don't be willing to troubleshoot installs made with it, fine. Don't want it automatically listed? Absolutely fine. But this is over-reach in the extreme. If CKAN supports it and a user wants to install a mod directly so they can use CKAN to manage it, why shouldn't they be able to?

Seriously how would anyone expect people to react if in other communities like the ones around Bethesda's games had mod authors suddenly saying "Oh no, you're not allowed to use Mod Organizer to install this one! You have to use NMM!" (Or any other particular mod tool.) I don't see that particular change in anyone's best interest. At best it's going to lead to annoyed users who will just pester the mod authors more asking why they would do such a thing.

I fully support every single other suggestion in that post. I myself brought up similar suggestions on the KSP forums not long ago, but I'm pretty sure they got lost in the animosity mod authors (or at least a vocal few of them) have expressed at CKAN as a whole. Ferram et al. made it abundantly clear on the forums that suggestions like this to mitigate the issues they're experiencing aren't acceptable to them.

But again, I'll reiterate my support of this, sans prohibiting installs. It would provide more information to users and _hopefully_ direct them to the _right_ support areas instead of pestering mod authors with issues that are actually with CKAN itself. (Basically I would hope to see users directed to CKAN's support pages first for Available, Unsupported, or External installs. Sending them to a link for common troubleshooting steps here on the GitHub wiki would probably help redirect users to the right spot.)

After reading forum I think I underestimated scale of problem. It's sad that good tool becoming a threat because of users that can't read and spam modders with CKAN issues :>

That's why I personally think having a way to use CKAN to direct users to the right place for support would be very beneficial to all concerned.

Cphoenix on 25 Jun 2016

👍2

@dewiniaid first off, torrents are do nothing more than ckan, a torrent is the equiv of ckan meta, instructions on where to find the file, information on the file, etc. what ckan does is go a step further, accually handling distro of the file itself.

A torrent has a source (or many fragments of a source) _somewhere_, and the people responsible for said sources (e.g. someone else running a client) are not authorized to distribute the content.

In the case of CKAN metadata, the source of the data is the author's own website/Spacedock/Github Releases/Curseforge/etc; linking to that source is perfectly legal. I'm not infringing on copyright by linking to, say, one of the early releases of KSP on Squad's own website, but if I link to the exact same file on freepiratedsoftware.example.com I'm potentially participating in "contributory infringement" (the same way various torrent sites have been shut down), and the owner of said site (where the actual data lives) is definitely infringing. Never mind that it's a free download, the ARR license means "Hey, only I have the rights to distribute it."

dewiniaid on 25 Jun 2016

👍3

@Cphoenix "Prohibited" is intended to serve as a way for an author to explicitly say "No, I don't support CKAN at all, please install my mods using my preferred distribution method(s)." There is no technical way to actually enforce it (aside from DRM of some kind, and nobody wants that), but the _intent_ is that the author's preference is made very clear and cannot be disregarded by means of "user is conditioned to just click OK without reading the prompt". I don't like it either, but it seems it a necessary option.

Remember that one of the issues here is that various things -- from CKAN metadata breaking an install to "Your mod doesn't work" "Did you read the install instructions and edit the .cfg files?" have led to an increased support load for several mod authors. There _has_ to be some way for them to go "Nope, can't/won't handle that" -- the lack of that option is the crux of the current issue. That said, I still feel the best possible result is to expand the options that CKAN metadata has in such a way where delisting/"Prohibited" becomes the option of last resort in these circumstances, rather than the _only_ option.

dewiniaid on 25 Jun 2016

@cphoenix It's not a matter of legal standing, it's a matter of common sense. If a modder asks you to remove your listing as a matter of courtesy, you should do it, otherwise they're likely to come back with an ARR licensed mod, and ARR mods are a pain in the ass for everyone involved.

DuoDex on 25 Jun 2016

From my perspective it looks as simple as adding a mandatory "maintainer"-field (obsoleting "x_maintained_by") analog to "author" and adding a field "manual_install" set to true for mods that must not be auto-installed.

janbrohl on 25 Jun 2016

"torrenting nowadays has the niche they dont accually download / distribute the file, as they don't touch the file."

torrent sites only host metadata. torrent clients actually do file distribution of the bits that they download. that is where it gets legally a bit grey.

ckan's github site only hosts metadata. the ckan clients themselves do absolutely no distribution of the bits that they download (other than delivery to the end user running the client -- pulling from the publicly available URL that the author of the mod has made publicly available).

legally i don't see how it could be more clear that this is a "deep linking" issue vs. a "bittorrent" issue. ckan clients do not deliver bits to other ckan clients. ckan is not bittorrent.

lamont-granquist on 25 Jun 2016

if mod authors want to control their own distribution, they could introduce a similar scheme to the way that oracle requires a time-sensitive cookie to download java, requiring users to click through the EULA.

lamont-granquist on 25 Jun 2016

Nothing could stop authors from moving their files (invalidating the CKAN-download-URLs) and relicensing them with the same conditions plus "CKAN-Exception" stating installation via CKAN client is forbidden. (Not making them ARR this way.)

janbrohl on 25 Jun 2016

fun fact, majority of torrenting is 100% legal, most MMOs use it to lower the server load, lets get that bit out of the way before someone claims its all bad - the bad bit is the places that decide to share stuff they arent allowed to, and ofc the media will fuel any fire that gets em ratings, nobody cares "this guy did nothing bad today, neither did this guy"

@janbrohl many licenses can't be changed from what they are now. and ckan policy as stated in another issue, is if permission is given it stays given - edit: clearly noting if alternate downloads of the mod exist, switch to them

anxcon on 25 Jun 2016

@Cphoenix improving the backend to cover more usage cases and minimize the number of errors is all well and good but these issues have been known for over a year and instead of fixing them this project seems to have instead prioritized keeping up with the metadata.

As a result what the authors want is a way out NOW with the option to come back when the service improves LATER. simply put there is no confidence that the needed code changes will be made in a timely manner.

passinglurker on 25 Jun 2016

I think we've gotten off-topic here. No mod author has ever stated that indexing mods is against the law (regardless of license), and discussions about that are pointless anyway. Although the fact that arguments seem to be heading towards an implicit "we should index ARR mods regardless of modder opt-in/opt-out too, it's legal just like for torrents!" has me concerned for the resulting workload.

My main concern is that CKAN's policies are set up so that create more support requests for me, and then provide little option to protect myself or to convince CKAN to change those policies. Because of these policies, I have no confidence that CKAN and its contributors can be persuaded to implement proactive error checking methods or whatever else may be needed to reduce modder workload in a timely manner, considering they have the option of simply doing nothing and the mod remaining listed.

For context, about a year ago CKAN created a gigantic support nightmare for a FAR release due to a metadata error (actually, several of them back-to-back), burying crucial bug reports that I needed to fix actual critical bugs in the mod. The "no courtesy de-list" policy was officially adopted a month later. An issue to address the cause of the FAR debacle only got created in April. No progress has been done on it since.

As of right now, there are a lot of modders that are very ticked off because they have no simple means to deal with the consequences of CKAN's errors. The extent of those we have are to license more restrictively (at least for now), to put in the effort to manage the metadata ourselves (though given the recent happenings with @BobPalmer even that isn't a guaranteed fix) or to simply quit modding. This is not a healthy situation that could easily be addressed through changes to CKAN policies.

ferram4 on 25 Jun 2016

👍1

The problem that I see is that a "Opt-in only" policy means a lot of mods would never be listed on CKAN through author apathy.

I would like to find out how satisfactory it would be to change the policy to allow open licensed mod authors to opt-out from CKAN installations, so that CKAN is not pushing authors to switch to restricted licenses?

This seems to me to be a simple policy change that would allow us to quickly lower the level of antagonism in the debate. Any technical solution for listing mods for user tracking without CKAN installation is going to take time. Time during which the current situation is maintained, and mod authors are getting angrier and angrier.

So I propose a change in CKAN de-indexing policy to say that open licensed mod authors can opt-out of CKAN installation. Initially, this would effectively mean opting out of CKAN listing entirely, until CKAN has a system in place for listing mods with no installation.

@ferram4 , @BobPalmer @anxcon, @Sigma88 , could you please respond to this specific suggestion?

politas on 25 Jun 2016

I'd like to cover that suggestion, also toss out a total of three (IMO reasonable) things that can sort at least the concerns on my end.

Yes, the option to opt out regardless of licensing and be absent from any listing (even just the mod name, though I doubt many would do that, I do see some cases where a mod goes on hiatus, or has been deprecated (how long was Regolith in CKAN?), etc.).

A staging area (as suggested before) so that pull requests are vetted. So anyone messing with FAR, or adding one of my mods. or even the case where I fat-finger my own metadata. Make sure it at least passes some cursory checks. This ties into the next point (which again - helps everyone).

I agree with @politas on modder apathy. I also expect that 95% of the modders are fine with their stuff magically appearing. But we also want to make sure we handle the few edge cases.

A new mod comes up. Ask said modder 'We are planning on listing this... let us know if you are not ok with this'. Wait 48 hours (or whatever). If no response, put it up. Done. If they say 'no thanks', respect that and move on. If modder comes back later and says 'please take it down', do so. In all cases, there's a conversation that takes place. The wishes of content creators are respected, but CKAN is not stuck having to chase down every modder getting explicit permission,

An answer may also be 'please don't ever list any of my things - I will sort them myself'. Plunk that author on a list. This is an easy gate to check in pre-staging. Ferram may prefer to vet his changes himself. I may prefer to do my own metadata, or not list all of my mods. 95% of everyone else may be fine with the default system. I think the changes below sort it for most everyone (though I cannot speak for @ferram4, @anxcon , and @Sigma88

To recap.

Ask first. If no reply, go ahead and list it.
If a request comes to de-list, do so,
Add a staging step to catch some of these common issues.

I respect that the last may take time, so steps 1 and 2 are a good start. Tho I think step 3 is pretty important to help alleviate the frustration.

BobPalmer on 25 Jun 2016

@BobPalmer , can you expand on what you believe is the minimum acceptable level of vetting? What checks should be carried out?

politas on 25 Jun 2016

@politas - At a minimum - make sure the file is valid, and that the dependencies are there / respected. Check against authors known to publish their own metadata, or who want a chance to review the netkan before it changes. Pie in the sky would be to have maintainers who would actually check the CKAN download first and make sure KSP loads, etc. before giving the thumbs up.

That last one is of course an edge case - relevant to things with complex dependency paths or very compex installs (like RSS, etc.). Me, I'd be happy knowing that my Netkan gets a spot check, and that if a netkan slips in where someone accidentally submits one of my mods, it can get caught and someone can ask me before it pops into the index.

BobPalmer on 25 Jun 2016

Another good example will be when we have the bruhaha post release where everything gets messed up due to incompatible versions. A quick 'is this compatible with the new version? Mind if we set that to push other mods through?' goes a long way.

BobPalmer on 25 Jun 2016

Another issue is the lack of clear APIs for dependencies. CKAN by default assumes that dependencies are not version-specific, and that assumption has messed up CKAN installations of FAR releases. We probably need to disable automatic metadata updating in such cases and switch to vetting each new version. It's a pain in the arse, but that would make things better for CKAN users and affected modders both.

politas on 25 Jun 2016

Agreed, but good news is that will be an edge case. If a modder sees something blow up, they can always de-list and have the conversation on what steps need to happen to sort that - including a manual or joint vetting process.

i.e. if mod X has some weird issues, agree that it stays in staging until either the modder or a designated maintainer can vet it out.

BobPalmer on 25 Jun 2016

I'll back up @BobPalmer 's suggestions, they seem good, although I would personally prefer a stricter Opt-In only policy instead. There is a risk under the proposed method that an apathetic modder's first interaction with CKAN will be discovering that someone indexed one of their mods because the PM got ignored, or the post got buried, or that the author was away for a bit and that they're getting issues they didn't ask for. Not gonna get a lot of CKAN popularity there, tbh. At least they will have options though.

I think there are three other things needed besides metadata vetting:

Mandatory public postmortems for large-scale CKAN failures (read: don't sort themselves within a day or two, say, during KSP updates). That would allow finding out what parts of metadata vetting are weak, if someone simply dropped the ball, if new policies are needed to address things. As time goes on, it'll allow for a large number of case studies to point towards what can go wrong and how to avoid it and by making it a procedure it'll build confidence in CKAN again because failures will be publicly acknowledged and addressed.
An abridged history of this entire mess. If @TeddyDD 's reaction is any indication, there seems to be a disconnect between what CKAN devs/contributors and what mod authors think is going on. Things got and have been very bad but no one with any influence to change things seems to have realized it. If new CKAN contributors join and only ever talk to the CKAN contributors and not modders, things stay this way; seeing exactly how bad things can get should be part of the "things you need to read and know before contributing" to try and prevent the disconnect in the future.
An always-installed plugin that dumps the installed CKAN data into the output log when the game starts up. It will help with troubleshooting CKAN installs and also identify issues that might be caused by people trying to get around any of these policy changes through unofficial repos.

Sound good?

ferram4 on 25 Jun 2016

👍2

I'm appearing late to the conversation, so thanks in advance for everyone's patience while I'm still getting up to speed. There are lots of concurrent discussions going on; I'm going to try and contribute here and link in from others. I _very_ much want us to have a consistent policy regarding how mods are indexed. There's been a lot of discussions, and while I can't address all of them, here are my arguments for many of them:

We do whatever the mod author wants, or we don't list the mod. ARR is a clear indication that the authors have absolute control over the code, and we should respect that.

Temporarily de-listing mods

We can and should continue to do this when a mod is causing issues that can't be solved by fixing the metadata. In the cases where something can be fixed via metadata, a temporary de-list may still make sense while that's sorted out. Installing mods in a way that is broken benefits nobody.

Permanently de-listing open source mods:

Some mods have _dozens_ of contributors, and open source licenses make it clear that _anyone_ has permission to use, modify, and redistribute mods distributed under an open source license¹. RealismOverhaul is a great example here, it's got sixty-three independent authors. Even FAR has a dozen listed, albeit most other than Ferram are reasonably minor.

I don't think I should be able to permanently de-list RealismOverhaul, for example, and what's more there should never be a reason for me to ask this. If a mod is working correctly, and has a free and open source license¹, then we're not respecting the wishes of that license or its contributors by permanently de-listing.

The Attic

I _do_ think it's entirely appropriate to modify existing metadata to make sure extant mods don't get installed on systems they're not designed for. I'd also say it would make sense to have an "attic" repository where old mods go to gather dust. Users can opt-in to get mods from the attic, but there's no risk of typical users getting mods that are no longer maintained or otherwise outdated.

Re-licensing of FOSS mods to ARR

I view this as a matter between the mod authors themselves. While one can't retrospectively remove a license from already released code¹, but it's certainly possible to release new versions under a different license, although I won't debate the mechanics of that here. If mod authors mark new releases as ARR, we should respect that. I feel the correct objection here is to fork and maintain a copy of the last FOSS version of the mod.

Addressing a few comments in particular:

@ferram4 : I would personally prefer a stricter Opt-In only policy instead. There is a risk under the proposed method that an apathetic modder's first interaction with CKAN will be discovering that someone indexed one of their mods because the PM got ignored, or the post got buried, or that the author was away for a bit and that they're getting issues they didn't ask for.

For restricted license mods, I'm in 100% agreement. I'd be totally fine with an opt-in only policy for restricted license mods.

For FOSS mods, I still feel we should be trying to CC in authors/maintainers as much as possible. It's polite, and we'll likely end up with better experiences for all.

Mandatory public postmortems for large-scale CKAN failures

An abridged history of this entire mess

We're a team that consists entirely of volunteers, but to the best of our knowledge our processes and communications are public. I'm very supportive of being able to analyse failings and friction, and find ways in which we can improve. However, that also requires finding someone with the time and motivation to do these things.

An always-installed plugin that dumps the installed CKAN data into the output log when the game starts up.

This sounds like an excellent idea, and the CKAN core has been built specifically so it can be loaded by KSP should we ever need that, Again, we need someone to code it, but I can see very few downsides of having this at all.

@politas : CKAN by default assumes that dependencies are not version-specific, and that assumption has messed up CKAN installations of FAR releases. We probably need to disable automatic metadata updating in such cases and switch to vetting each new version. It's a pain in the arse, but that would make things better for CKAN users and affected modders both.

I agree here. The metadata _does_ allow for version ranges of dependencies to be specified, but often this hasn't been specified. A simple "vet-new-releases" flag for NetKAN would be a fairly straightforward change that means human attention can be better directed where it's needed. In the case of staging (see below), "vetting required" releases need a human to authorise them, and won't be assumed good by the bots.

Staging

This gets its own heading, because I think it's _so_ important.

@RoverDude: A staging area (as suggested before) so that pull requests are vetted. So anyone messing with FAR, or adding one of my mods. or even the case where I fat-finger my own metadata.

This sounds like a great idea to me. We already support multiple upstream repositories, and it makes sense to have a staging repo which users would need to opt-in to receiving. I'd even weakly argue that _all_ new mods appear here first, and are then bot-moved to the main area after a couple of days. Users who "need the latest mod now" can always opt-in to the unstable area, and are valuable for helping catch bugs early, but your typical user will only see mods from stable.

Under the hood, we can have an anointed branch in our existing CKAN-meta repo, which would mean we could more easily track how metadata moves should that ever be needed in the future.

One could also argue for there being "supported" and "unsupported" repos. There'd need to be some discussions as to how mods end up in each, but they're extremely possible with the existing clients and infrastructure.

I am still catching up with conversations, so thank you again for all your patience.

TL;DR: I _really_ like @RoverDude's suggestion of a staging area, and think there's potential to solve a lot of problems by having separate stable, staging, and unstable repositories, with users needing to opt-in to anything but stable.

¹ We use the Open Software Initiative's definition of open source. Any mod that doesn't have a perpetual, transferable license to use, modify, and redistribute is already listed as "restricted' in our licensing metadata.

pjf on 25 Jun 2016

👍2 ❤1

In the interest of community goodwill, allowing a modder, regardless of their license choice, to have the option of de-indexing their mod is core here. Let's not lose sight of that. Because without that, we'll just move to ARR licenses (those that are not on them already).

If you are willing to respect a modder's implicit wish in the case of ARR, why can CKAN not respect a modder's explicit wish to not participate, for whatever reason, in CKAN indexing?

Currently, there is a tremendous backlash building up against CKAN because of this - something that can be solved with one sentence in the de-listing policy by extending the courtesy afforded to those of us who now use ARR licenses to all users. It's also the right thing to do.

Without this, you will continue to see more modders switch to restricted licensing and de-index. And while it is simple to say 'we would just fork the last good version', experience has proven that that's rarely feasible, except in the simplest of part mods. And we're not talking about part mods here.

So. Let us start by looking at what it is going to take to get that particular policy changed. Specifically, That if someone, for whatever reason, does not wish their mod indexed, it can be removed regardless of their license.

@pjf , @politas - thoughts?

BobPalmer on 25 Jun 2016

👎3

@BobPalmer opt-out is straightforward in the case of mods that have only one author, but what about when that isn't the case? Does only one have to want it de-listed? Do all?

blowfishpro on 25 Jun 2016

It should be the main maintainer. In almost all cases, this is pretty obvious (whether it's the original creator, or the current curator). Looking at the account tied to the Github repo or KSP thread will point you right there. ie. there's no doubt who speaks for my mods, even though I have had dozens of contributors accross them. Nor is there any doubt who speaks for FAR.

If there's a doubt, a conversation can be had (and I expect those will be very few and far between).

BobPalmer on 25 Jun 2016

ckan has already made the claim for legality that it can list anything, even ARR, but wont list ARR for courtesy reasons. ie listing cannot be stopped. now the other direction, users cannot demand a mod be listed on ckan, atleast legally.

so now that the legal bit of "do we list it" is out of the way leaving the policy "do we list it" up to policy, lets look at that. single author is accepted as straight forward, so we are left with multi author mods and policy (not legality) of what to do, since policy can state anything that is wanted (p.s. policy against my little pony mod plz) you have 4
1) all agree to list else dont (i dont see ckan happy)
2) any 1 (or majority) agree list else dont
3) ignore all, list anyways (seems ckan fav idea)
4) and rover just posted above me primary maintainer
and having policy state they need repo control etc, not just submitted a PR to contribute, limits those who accually control it

anxcon on 25 Jun 2016

I would agree with 4, unless someone gets control transferred per the maintainer from 4. i.e. for Firespitter, that is Snjo even tho I do most of the updates these days as a courtesy. If I ever wanted Firespitter pulled for whatever reason, I would have to get Snjo to do so, or to officially transfer the mod over to me. Repo ownership and thread ownership are two very easy ways to sort this.

BobPalmer on 25 Jun 2016

This sounds like a great idea to me. We already support multiple upstream repositories, and it makes sense to have a staging repo which users would need to opt-in to receiving. I'd even weakly argue that all new mods appear here first, and are then bot-moved to the main area after a couple of days. Users who "need the latest mod now" can always opt-in to the unstable area, and are valuable for helping catch bugs early, but your typical user will only see mods from stable.
Under the hood, we can have an anointed branch in our existing CKAN-meta repo, which would mean we could more easily track how metadata moves should that ever be needed in the future.
One could also argue for there being "supported" and "unsupported" repos. There'd need to be some discussions as to how mods end up in each, but they're extremely possible with the existing clients and infrastructure.

Id like to argue for supported vice unsupported repos. One simple solution would be to move all metadata in the present repository to a new repo, 'Unsupported'. This unsupported repo would function much as the present one does, but would not be the default repository.

Mod Authors or a nominated Maintainer could add metadata to a new repository, Experimental. This repo would hold new versions of the metadata (and new metadata for newer versions of the mod) for a period of time, afterwhich it would be moved to another new repo, Stable.

The only repository enabled by CKAN by default, would be the Stable repository. Users could enable Experimental, or Unsupported, with a corresponding decrease in the level of support offered by the modder.

Potentially, CKAN Contributers could add mods to Experimental (nominating themselves as the Maintainer), with Experimental and Stable repos always permitting mod authors to Opt Out.

Ideally, mods displayed in CKANs listing would then be color coded to indicate which repository they come from, perhaps a subtle difference in the background color of the list elements.

This set of Official repositories would likely also assuage concerns over users creating their own sets of repositories, by obviating the need for it.

Blu3wolf on 25 Jun 2016

Except getting someone to admit to something they know precludes them from support pretty much never works. We've been over this with Win64, etc.

BobPalmer on 25 Jun 2016

👍2

Thats a fair and reasonable complaint. I seem to recall it was suggested somewhere that it would be possible to have CKAN log data in KSP, as a plugin? So that users submitting valid bug reports would be easily caught out by that?

An always-installed plugin that dumps the installed CKAN data into the output log when the game starts up. It will help with troubleshooting CKAN installs and also identify issues that might be caused by people trying to get around any of these policy changes through unofficial repos.

That.

Id hate to think that the only solution that worked for Win64 would be applied here, although it has been threatened.

Blu3wolf on 25 Jun 2016

👍1

If you are willing to respect a modder's implicit wish in the case of ARR, why can CKAN not respect a modder's explicit wish to not participate, for whatever reason, in CKAN indexing?

Because a Free and Open Source Software (FOSS) license is a hard promise that others _can_ use, modify, and redistribute code, even in ways that the original author objects to. It's one of the reasons why FOSS projects are often so vibrant, because users and contributors know that the code will always be there for them to use and change.

Nobody's ever forced to release their mods under a FOSS license, it's a choice that mod-authors have to opt-in to, voluntarily and willingly. Contributions to those mods are done under the same license, in good faith that those contributions will remain open and available in the future.

A FOSS license in which the author says "You can't use this in a ways I don't like" _isn't_ a free license, and the software should not be licensed as such.

It's not my goal to make the lives of authors difficult. I _really_ like the idea of having staging and testing areas, of making it so that users have to opt-in to things which may not be working right, or may not be properly checked for correctness, or which need human vetting. I _absolutely_ support mod re-licensing where that can be done, and I absolutely support us temporarily de-listing mods where we know they're causing problems. It's only the permanent de-listing of mods that are under a FOSS license that I struggle with.

My head's become fuzzy from staring at a screen for so long, so I'm going to break for some exercise and to see the sky.

pjf on 25 Jun 2016

👍2

@pjf this isn't one of your big professional FOSS projects this is a modding community a labor of love just because we give you a license with such freedom that you can kick us in the face doesn't mean you should. You need to play nice if you want to spread FOSS here.

passinglurker on 25 Jun 2016

👎3 👍2

I feel we're starting to get off-topic here. @Blu3wolf, @passinglurker, I think you've both made excellent points. In the interests of keeping the discussion easy to follow would you both be okay with letting this issue be used for discussing the other (but related) issues which have been raised?

pjf on 25 Jun 2016

👍1

As @anxcon noticed here opt-in unsupported repo could not works so well as we think. Many users would activate it mindlessly and spam author thread anyway. It's too easy to add repo url in ckan gui. That's why you don't find AUR helpers like yaourt in offical Arch repos - to force noobs to actually read a wiki and build package manually.

I'd suggest making CKAN completely ignore repo marked as unsupported. So noob user would not be able to install unsupported mods using CKAN. When it comes to advanced users... I'm not sure. AUR like approach (grab *.netkan, build package for yourself, install it) could not work here because some mods don't have netkans. Maybe CKAN could allow for adding repositories from local directory so advance user could just git pull unsupported repo manually?

And I think that even FOSS mod authors should be able to opt-out (move his mod to unsupported repo effectively delinting mod for normal / noob users) Its against logic, its against FOSS license _they chosed_, its against users interest but moders overreaction with changing license to ARR is nothing good in long term. This way we keep modders and advance users happy :disappointed:

An always-installed plugin that dumps the installed CKAN data into the output log when the game starts up.

This is really god idea but this could be just first step. CKAN could help with preparing bug reports (for example automatically find all KSP logs and upload them to pastebin on user request?). CKAN should also keep log similar to pacman.log (it looks like this ) and also allow to revert last transaction (so player would be able to get back to working game without hassle)

Mandatory maintainer for supported ckans/netkans is also pretty good idea :+1:

I'm also thinking about some bots improvement that would allow modder to nuke mod release in seconds. For example when your mod is causing trouble with CKAN you just open issue nuke XYZv1.2.ckan and boot would delist mod, close this issue and open new one devoted to repairing mod. It would require to check if nuke issue was opened by mod author but that could be done (github username check?)

TeddyDD on 25 Jun 2016

👍1

Sorry @pjf but that pov remind me a bit too much of the sourceforge mess.

sarbian on 25 Jun 2016

While I understand and agree with @pjf on the fundamental concepts of FOSS licenses, I also agree that CKAN should be responsive to community standards in the specific realm of KSP game mods, which as mentioned, are not the same as large FOSS projects. Courtesy delisting for FOSS mods by primary maintainer request should be our policy.

EDIT: When I say de-listing, I mean no installation, which currently equates to de-listing

politas on 25 Jun 2016

For mods we mark for no installation, we could still have the "download contents" button available and an "open zip file" button to make manual installation easier. Then we could have a "mark as installed" button so that CKAN is aware of which version is installed of a listed, but not automatically installed mod.

politas on 25 Jun 2016

👍1

@TeddyDD is right that end users will still spam mod authors by using the opt-in unsupported by default. Ref issue #1502

Also can we look at @Dazpoet ideas again under #1679. As it might help here.

solarsootysmudge on 25 Jun 2016

I wonder if forcing installing mods from unsupported repo via cli would do the job

TeddyDD on 25 Jun 2016

👎1

what it might do is convince folks to learn how to write a GUI.

Blu3wolf on 25 Jun 2016

A good way to deactivate installation could be to set the "install" field to a string (with instructions for manual installation).
This would stop the ckan client from installing it (not sure if it would still be listed) and future versions could show these install instructions while the entries would still be useful for finding the mod like with my Web-View.
If you do not want your mod listed in the Web-View just contact me on the Forum as i can blacklist mods idependently of CKAN.

janbrohl on 25 Jun 2016

@janbrohl Even simpler way is to change .ckan extension. CKAN client detects only *.ckan (like all .kerbalstuff packages)

@Blu3wolf You are right I suppose.

TeddyDD on 25 Jun 2016

I very greatly require CKAN to manage my mods because there are just too many mods to manage manually. However, CKAN should not be doing things against an author's will, regardless of a mod's license.

I'm fine with default supporting adding mods that have an open license. But if an author comes by and asks to be de-listed, then his request should be honored regardless of the reasons for wanting to be de-listed.

I still hope for the method of "no installation" listing that I had mentioned "Opt List" so that I can still find mods via CKAN even if I have to go to the author's page to install it. I just don't want to be forced to always check a browser folder of bookmarks to find out if a mod updated or not and Mini AVC is a dumb way to check for updates (because you have to run the game first instead of checking prior to launching the game).

With the current system, authors may end up closing the mods behind an ARR and prevent anyone from sharing and using code to make better mods in the future or should a mod author suddenly disappear, his great work will die too because he had locked it behind ARR to prevent CKAN from touching it.

I'd also still like meta files within the zip file to be able to correct CKAN information about a mod too, instead of trusting whoever added the mod to CKAN in the first place. This includes a "de-list my mod" line which would auto-de-list it if CKAN scans the zip and finds that line in the specified file, preventing people from listing it again after its been de-listed. I'm not sure how people list a mod to begin with but this would make de-listing automatic and in complete control of the author where he doesn't need to contact anyone to get the mod de-listed.

I skipped a lot of conversation here because I got a headache and pain in one eye so bleh. I skimmed through though. I had to turn off email notifications too cause I got over 10 emails in less than 10 minutes.. -.-

VasVadum on 25 Jun 2016

I should probably weigh in as a mod developer that generally sits on the CKAN side, all of this has turned into a much bigger deal than it otherwise needed to be. @ferram4 and Mr "Community Friendly Alternative" did the correct thing making their mods ARR as that more accurately reflects their position on licencing.

That should have been the end of this whole issue, but it now seems like enough damage has been done that changing the policy to allow the removal of mods would probably be a good idea at this point.

Modders: I'd still advise against doing so though, all you are doing is annoying players who use your mod. Remember, although CKAN has problems (metadata checking, hanging), it doesn't mess up anywhere near as spectacular as a player installing manually can :P

godarklight on 25 Jun 2016

👍3

Or use a timebombed license: ARR until X condition is fulfilled, at which point it goes to [FOSS license].

DuoDex on 25 Jun 2016

👍3

@DuoDex - Which is what I put my assets under.

BobPalmer on 25 Jun 2016

👍1

@Cphoenix "Prohibited" is intended to serve as a way for an author to explicitly say "No, I don't support CKAN at all, please install my mods using my preferred distribution method(s)." There is no technical way to actually enforce it (aside from DRM of some kind, and nobody wants that), but the intent is that the author's preference is made very clear and cannot be disregarded by means of "user is conditioned to just click OK without reading the prompt". I don't like it either, but it seems it a necessary option.

@cphoenix It's not a matter of legal standing, it's a matter of common sense. If a modder asks you to remove your listing as a matter of courtesy, you should do it, otherwise they're likely to come back with an ARR licensed mod, and ARR mods are a pain in the ass for everyone involved.

I wasn't suggesting that mod authors not be allowed to request their mods to be delisted, and whenever possible have that respected. In fact I didn't mention that at all in my post. I was saying that requiring a user to compile CKAN themselves, after forking it and editing-out the prohibit code is an insane imposition to put on anyone. It doesn't actually fix the underlying issues, it just means users are going to make guides to do just that and then do it anyway. And then you have people working around the entire system that you just put in place to supposedly relieve people's headaches over this, except because the "Prohibited" status would be ignored, they would be extremely likely to add listed mods with that status but not get any warning whatsoever. Headache for troubleshooting it, headache for mod authors if the user goes to them first, headache for CKAN if they come here first. More pain and annoyance all around.

There's already a perfectly reasonable "harder than just clicking ok" option: make your own CKAN file. It's a bit out-of-the-way, but it's there. And it should remain an option. Introducing a complete block on managing a given mod with CKAN would do little more than hand mod authors a "Spite users?" button.

Again, what software a user uses to manage mods is up to them, and should be up to them. Whatever solution is reached shouldn't try to enforce those kinds of restrictions. It's a waste of everyone's time (including mod authors). Whatever policy is put in place needs to have a good chance of users actually following it, instead of it being likely they'll try to break it, for it to be effective.

Right off the top of my head, you brought-up DRM. Just think about which DRM schemes have been most successful. Steam comes immediately to mind. Why? Because their DRM doesn't suck (or at least it doesn't anymore, but that's beside the point). Compare that to Ubisoft's checkered past, where legitimate users would download cracks so they could play the game they bought.

I'm only saying this because you brought-up recompiling CKAN oneself as being the only solution. If the solution to the problem were simply to create their own CKAN file, that'd me more reasonable. Though I still worry that will lead to a lot of unofficial .netkans floating around. I'm already putting together my own repo of them for my own use.

Remember that one of the issues here is that various things -- from CKAN metadata breaking an install to "Your mod doesn't work" "Did you read the install instructions and edit the .cfg files?" have led to an increased support load for several mod authors. There has to be some way for them to go "Nope, can't/won't handle that" -- the lack of that option is the crux of the current issue. That said, I still feel the best possible result is to expand the options that CKAN metadata has in such a way where delisting/"Prohibited" becomes the option of last resort in these circumstances, rather than the only option.

Install instructions in the metadata would be nice, but actually I think in that case it would be a great idea to create a standard similar to .fomod . In fact I think that's a rather excellent idea, especially since it could also fold-in KSP-AVC's .version files (for example, if you just had one .kspmod file in the mod archive, KSP-AVC could search that instead; that way authors would still only need to maintain one to two files at most). It's not like that has to be limited to CKAN either; other mod manager apps could be made to support such a thing.

@Cphoenix improving the backend to cover more usage cases and minimize the number of errors is all well and good but these issues have been known for over a year and instead of fixing them this project seems to have instead prioritized keeping up with the metadata.

If you look at the history of users working on CKAN, you'll realize that a lot of contributors (politas mentioned this very thing on the forums) don't actually write code, they help maintain metadata. JSON is relatively easy to understand. C# + Perl and all the moving pieces in maintaining an application and bots to support it is less so.

Cphoenix on 25 Jun 2016

@BobPalmer

I'm just going to say that all those suggestions (16 hours ago ish, so way up on this thread; sorry, I'm insanely busy this weekend and barely have time to read just this thread) are great. I'm glad we're at least making some progress here. I'd be happy to help the CKAN guys with whatever I can and have time for with this, mostly because through my own faux-pas I saw what can happen when we don't have a list of mod authors who prefer to manage or vet their own metadata, and other checks to make sure that stuff doesn't get merged into the NetKAN master branch, causing frustration for all. Even if that just means having another set of eyeballs on PRs for metadata.

Cphoenix on 25 Jun 2016

👍1

@TeddyDD

This is really god idea but this could be just first step. CKAN could help with preparing bug reports (for example automatically find all KSP logs and upload them to pastebin on user request?). CKAN should also keep log similar to pacman.log (it looks like this ) and also allow to revert last transaction (so player would be able to get back to working game without hassle)

That's a fantastic idea that has the potential to make everyone's a bit easier. Easier for users to create and submit _accurate_ and _useful_ bug reports (instead of just "your mod doesn't work"), easier for authors to sort them out in a timely fashion, easier for the CKAN team to deal with the same bug reports when it's CKAN's mess to deal with. Slap a button for that feature right on the GUI and make it obvious.

Sorry for the spamming posts guys. Never know how much time I'll get to read these things. >.<

Cphoenix on 25 Jun 2016

@BobPalmer It was more intended as a general suggestion ;)

DuoDex on 25 Jun 2016

Ok, I've had some sleep so now I can explain my actions. I created PR Allow foss delisting #1795 and have started behaving as though that change were accepted. As a result, I have delisted some mods on request. If there are any delisting requests that I haven't got to, I apologise for that.

Given that ModuleManager is one of the mods I delisted, CKAN is effectively useless for most users now. Sarbian handed me the launch codes, and I fired the nuke.

So why did I do it? Mostly as a wake-up call to my fellow CKAN contributors. Insisting that CKAN is following FOSS ethos by keeping all FOSS mods available was not ultimately in the best interests of either the FOSS community or the KSP community. I have been thinking a lot about this over the last couple of days, and here's what I think is the fundamental problem:

CKAN's policy was all about accessibility, and the rights of users. It was based on the assumption that all CKAN was doing was facilitating the installation of mods. Where it went wrong is in not taking into account the fact that in the minds of many CKAN users, CKAN also implicitly defines a support relationship, by linking to the "homepage" forum thread. We have attempted to disabuse users of that idea, but those efforts have been unsuccessful, and are likely to remain so, because people are people.

It is this expectation of support in the minds of users that has antagonised the modders, as far as I can see (combined with a perception of arrogance from CKAN team members and supporters). As such, I am going to act as though my PR #1795 has been accepted. Either we change the policy, or the CKAN team kick me out of the project.

politas on 26 Jun 2016

👍4

@politas and if they kick you can fork ckan and we can get the forum mods to kick the original arrogant ckan from the forum or at least lose its stickied status so the plan is fool proof either way! ;)

passinglurker on 26 Jun 2016

I wish I had the talents and connections to be able to fork CKAN, but I'm afraid I don't. Not that I want to fork CKAN, I want to _fix_ CKAN's problems.

politas on 26 Jun 2016

👍1

For what it's worth @politas, I think you did the right thing. I think it's sad that it has come to this, but still.

and if they kick you can fork ckan and we can get the forum mods to kick the original arrogant ckan from the forum or at least lose its stickied status so the plan is fool proof either way! ;)

Could we maybe keep this sort of posturing out of this? Please? The KSP forums already escalated to an astonishing level of absolute...I don't even know what to call it. Other than absolutely maddening.

Like @politas, I would much rather fix CKAN's problems than see it resigned to the gutter. Let's focus on _that_.

Cphoenix on 26 Jun 2016

I agree. Assuming the policy remains in place - and I'd want to see @pjf confirm that first based on the prior discussions in this thread - then I will be re-listing my mods (likely after a version bump, and I have some dependencies to sort).

(For specificity, I agree we need to move on and fix CKAN's problems).

BobPalmer on 26 Jun 2016

👍2

Is this entire mess due to the fact that the metadata is all distributed via a github repo where the authors of the modules don't have commit access to maintain their own metadata?

All the other package managers I've touched (rubygems, npm, cargo, chef's supermarket, etc) all have model where there's a website that distributes metadata that the author of the package owns and is responsible for keeping up to date. Since CKAN grew organically it was probably necessary to allow community curation of metadata, along with indexing bots, and the CKAN-meta repo was a really easy solution to bootstrap the product. Its pretty clear that its moved beyond that, however, and you really need a way that mod authors can maintain their own metadata and a button that they can push to submit it where the results are nearly instantaneous (barring some cloudfront/cloudflare caches being updated or whatnot, but it should be a matter of seconds or minutes for fixes to go live).

Since CKAN is only distributing simple metadata blobs and the problem of shipping the mod bits is 'outsourced' to spacedock/curse/github/whatever the bandwidth cost of moving to a real website hosted on something like heroku shouldn't be that onerous or costly. The website would need to get built but you really just need a CRUD interface over the .ckan json files, with some basic account management that nearly every web framework has some kind of plugin for, along with a download point for "the whole tarball of all the metadata". I'd do it, but I'd write it in ruby on rails, which nearly nobody else in the community would be able to support.

lamont-granquist on 26 Jun 2016

Is this entire mess due to the fact that the metadata is all distributed via a github repo where the authors of the modules don't have commit access to maintain their own metadata?

It is possible to do in current system. I don't remember exactly how but you can add reference to netkan so it pull metadata from external source (directly from author).

I don't think writing new website is good option. There is plenty of work with CKAN client itself.

TeddyDD on 26 Jun 2016

Is there something broken with that then? Too long of a delay in the scraping bot? Scraping bot broken too often? Just nobody knows enough about that option to use it?

lamont-granquist on 26 Jun 2016

I suppose this works well. Here, I found this option: https://github.com/KSP-CKAN/CKAN/blob/master/Spec.md#ckannetkanurl

TeddyDD on 26 Jun 2016

I suppose this works well. Here, I found this option: https://github.com/KSP-CKAN/CKAN/blob/master/Spec.md#ckannetkanurl

That's how @BobPalmer was managing his metadata before this mess. It works alright as long as no-one goes around making metadata for his mods before he does ( :flushed: ) . @politas' proposed opt-out list ( #1795 ) will go far toward alleviating that problem. The benefit of that system is that it very clearly points to who is the real maintainer of the metadata, the mod author in this case, and they've pretty much got complete control (unless someone overwrites the .netkan that's pointing to the metanetkan, which would be rather rude).

For the record, while I know that mess earlier was caused by me, a naive user at the time who didn't know just how much a headache that can cause, I don't think it's a good idea for any policy trying to fix this problem to completely eliminate the possibility of new metadata wranglers. Naive users should still be allowed to pop-in an help-out if they can, just with enough checks to catch these sorts of screw-ups (like not merging their changes until they are very thoroughly checked by someone more experienced; a staging area would help). Many hands make light work, after all. I'd hate to see anyone tied-down feeling like they need to be on here constantly fixing metadata because there's no one else to chip-in, since everyone here is a volunteer. :disappointed:

Cphoenix on 26 Jun 2016

It kind of seems like that method has failed to me, and the opt-out list will probably work extremely well for as long as this issue is in the forefront of everyone's minds, but that at some point some relatively new CKAN maintainer will merge a PR without checking the opt-out first and you'll wind up right back here again.

However, if the opt-out list keeps the mod authors in this thread happy for the time being and averts the crisis and the FusterCluck of removing all the ARR mods, including ModuleManager, then it seems like a decent first step. I'd really encourage everyone involved in the conversation to keep considering better technical solutions to the problem, though, since it smells like the ckan has just been kicked down the road...

lamont-granquist on 26 Jun 2016

👍1

Best way to solve this. If the submitter is not the original author, just check it against the opt-out list. Or if in doubt, just ping the original author. Do all of this while it is in the staging area. Stuff should not be merged from stage to the live repo unless we're sure (a) the submitter is the project maintainer, and (b) the project is not on the opt-out list.

BobPalmer on 26 Jun 2016

👍2

(The opt out list could even be automated a number of ways - like restricted repos or author names).

BobPalmer on 26 Jun 2016

As far as your first comment goes.... I've found that anything repetitive mechanical process involving humans invariably breaks orders of magnitude more often than one that is just backed by code -- "its simple just check the opt-out list" is a recipe for failure.

As far as your second comment goes... Yes, I'd argue that is nearly a requirement.

lamont-granquist on 26 Jun 2016

👍1

As far as your first comment goes.... I've found that anything repetitive mechanical process involving humans invariably breaks orders of magnitude more often than one that is just backed by code -- "its simple just check the opt-out list" is a recipe for failure.

The policy change is a band-aid to placate everyone temporarily, no question. (I don't mean that in a "mod authors are chumps" kind of way, to be clear.) A solid solution backed by code as you say and not error-prone meatbags is the end-goal, I think. It's just that that takes time, and comparatively policy changes are far easier to do in the short-term.

Cphoenix on 27 Jun 2016

👍1

In the interests of completeness, and in the renewed atmosphere of cooperation, are there any people who are still demanding opt in, before this issue gets closed? I am happy to explain in detail why I am against such a policy (and plan to do so soon, anyway).

politas on 28 Jun 2016

I would prefer @BobPalmer 's suggested approach above:

Ask first. If no reply, go ahead and list it.
If a request comes to de-list, do so,
Add a staging step to catch some of these common issues.

Which is technically opt-in if you can get in touch with the author, opt-out if you can't. From the author's perspective it means that if you're active you have the chance to say, "hold up, don't index it _yet_" if something is going to change soon, which avoids a lot of metadata issues that opt-out may still cause. It also means that an active modder's first interaction with CKAN doesn't risk being a metadata issue that they had no idea was coming.

I think I understand the reasons for wanting opt-out rather than opt-in though, which is why I'm alright with it for "no reply." I'd guess that those are as follows:

Apathetic modders aren't going to get bothered by CKAN if their stuff is listed
Oh God, oh God, if we go opt-in only what do we do about all the mods we've already listed?!

Just make the time to wait for a reply a week before it switches from opt-in to opt-out, make the procedure and time well-known (or as close as possible), and I'd be happy with that.

ferram4 on 28 Jun 2016

👍1

@ferram4
One scenario that the above doesn't cover is the CKAN PRs from SpaceDock. It's my understanding that those only happen if the whomever is uploading the mod to SD checks a "CKAN" box. I think those should be taken as implicitly meaning that the mod author _does_ want the mod listed and is passing the work off to the metadata wranglers here.

Still, it would be prudent to send them a message, either directly on the forums or whatever other method is listed, to just say "Hey, as requested your mod was listed CKAN [insert link to relevant PR/commit]. If this is in error please let us know." Or something to that effect. That way at least there's a chance to catch the occasional mis-click. And hopefully if that does happen being pro-active about it will help alleviate any frustrations.

Cphoenix on 28 Jun 2016

What I'd like to see in the long run is some integration of the web gui netkan creator being worked on into Spacedock, which would make it more than a single click and we could take those as a definite opt in.

@ferram4: I was planning to write a much clearer procedure for adding mods which would include @bobpalmer's "PM the author" suggestion, along with a message template for gathering information. A week seems a long time, though. Could we say that a week would need to pass between sending the message and the mod exiting staging/being merged?

politas on 28 Jun 2016

👍1

@politas Yeah, that sounds good; ideally, the metadata should be sitting in the staging repo marked that it's waiting on author permission / time before being merged. That would allow the .netkan and .ckan to be linked to the author in the request so they can look it over, possibly with instructions for how to get CKAN to install their mod using it while it's in staging so they can test it themselves if they like.

The main reasoning behind a week is that depending on people's schedules, they might not have the time to deal with the PM in a shorter timespan, which then means annoyance when they get to it and find out what happened. A week should give enough time for an entire cycle of the workweek to handle all of that. And if they're gone longer, they'll be less annoyed because you did give them awhile to get back to you.

ferram4 on 28 Jun 2016

👍2

Yep, I know all about life schedules delaying responses. Agreed that delay reasons would need to be made explicit in the staging / release process. Introducing an entire week's delay at the start of the process makes getting the mod into CKAN slower, so some hybrid of 48 hours before going ahead with creating metadata without a response and a week for opt out before release sounds like the way to go.

politas on 28 Jun 2016

You are gonna get a bunch of folks peeved at getting opt outs three days after initial contact XD

Blu3wolf on 29 Jun 2016

Yeah, I think as long as folks have a (reasonable) amount of time to put on the breaks then that is a good thing. tbh an initial delay is probably going to be an edge case. Anything where they click the CKAN box on SpaceDock can of course be fast tracked, and there you just want to do a quick 'Hey, we saw you wanted this on CKAN. Do you want to check out the metadata and make sure we got this right?'

I think the timeframes involved are less important than the basic process. And I also agree with @ferram4 that the the goal sjhould be making sure that the first interaction with CKAN is a positive one.

BobPalmer on 29 Jun 2016

👍1

In fairness, most mod authors are not going to have any idea if the metadata is right or not.

Blu3wolf on 29 Jun 2016

Most, but not all. And it's those cases that put us where we are today.

Point being that there's a dialog, the modder has an idea of what's going on, and experience has shown us that a bit of proactive prevention goes a very long way in these cases.

BobPalmer on 29 Jun 2016

Also, it can be as simple as 'It looks like you depend on Firespitter and Module Manager - are there any other dependencies or does that sound right? Let us know if there are any issues, and we can work together to get them resolved for you'. Make the experience positive for those with content, and you're going to start getting a lot more enthusiastic supporters among them.

BobPalmer on 29 Jun 2016

Re: Spacedock and other submissions of its nature:

If we have information that a mod author most likely intended to submit something to CKAN (I.e. checking the box on Spacedock), is there any real reason not to immediately go ahead with that submission? If they checked it by mistake, they'd most likely be understanding of the issue (and it can still be fixed after the fact).

This is particularly true if we add the staging repository and send the author a message more along the lines of "Per your request on Spacedock, we've added your mod to CKAN. If this request was made in error or you wish to maintain your own CKAN metadata, please [instructions]".

dewiniaid on 29 Jun 2016

No real debate there. If they explicitly opt-in (i.e. via spacedock) then go right into making metadata. Letting them know it was done and that it's in staging would be a nice courtesy (gives them a chance for an 'oops!').

BobPalmer on 29 Jun 2016

👍2

Letting them know it was done and that it's in staging would be a nice courtesy (gives them a chance for an 'oops!').

For explicit opt-in from SpaceDock and the like, I'd say give it 24hrs or so just to be safe before it gets pushed to the release repo/branch (that's more of a wishy-washy "eh, seems okay" bit regarding the 24hrs; I pulled that out of my...you know). Longer if there is some confusion about what the metadata should say (see#https://github.com/KSP-CKAN/NetKAN/pull/4149 for an example), obviously.

@BobPalmer is right I think that it's more the process that matters, not the timeframe. We can probably just settle for whatever is "close enough ish sorta" for time.

Cphoenix on 30 Jun 2016

TL:DR. Staging area yes but not as we know it. Something else is brewing.
Opt out is really an panic button option / debugging tool. Sometimes it really just needs to be there to give us time to think.

After a bit of digging around through my notes....

The stage area idea is correct way to do it but it is also overkill. Initial suggested reason only applies to a few mods. Plus in some edge cases that will not fix some of problems we are seeing. It is possible to submit metadata and have it pass all checks. Even the best of us will not see an error. Then things still go wrong in a tiny number of cases. None the less. We need a staging area anyway. See #1679. Some of this could be bashed out of existing code. Although that is a huge guess on my part. We kind of already have a way to do this but I need more time research the idea.

In short we mark as incompatible and require yoyo style version control. That allows full experimental installs but only for the technical savvy. If it all works out we switch back to GRAS. Then everybody gets the mod. I admit that I am grasping at straws here.

Also ....

The op out policy sometimes needs to be though of as an emergency stop button / debugging tool. Recently I have seen mistakes where metadata is correct and we get temporary problems. Nothing too bad really. This may have been acceptable in the past but numbers of forum users has grown. To the point where a minor problem causes too much spam. Those issues have be captured where we need to fix something. A quick example

Mod A depends on Mod B. Both sets of data good. Thumbs up.

However fails because we forgot Mod B depends on Mod C. Turns out C is out of date.

That's the simple version of the story. If you want to dive deeper into the actual bug report see #1646. Now lets be clear this does not current effect USI mods. As far as I am aware. However it may affect others in the future. So the issue is still alive. At the time this one happened @BobPalmer got error reports. We did not know what was happening. It was pain to tell people just to stop and give us time to investigate the problem. Now an opt out might be overkill here but it nice to know there is a "panic button" procedure when the forum starts to go nuts.

There is also another reason for opt out. Here is the controversial part. Sometimes it has nothing to do with CKAN at all. However we can't sit back do nothing. Sometimes we going to have to delist the mod anyway. Just give us time to think the problem through. Or talk to the mod author through solutions. No I am not going to write an example. It would look like a witch hunt. To error is human. To forgive is just good PR policy. If a mod author calls for it lets accept by default and then try to help resolve the problem.

solarsootysmudge on 30 Jun 2016

Sorry, had trouble following all of that. But I'll reinforce my preference for a basic staging process - it solves a lot of woes. There have absolutely been cases where I wish this was handy.

RE opt out - IMO the policy is in a good place. An option that is on the table, no questions asked, for any modder that desires it. Conversations should of course take place, but there are going to be people who do not wish to have a mod be a part of the indexing at this time. And yes, it's also a nice 'kill switch' that lets people withdraw until said conversations take place.

For Explicit opt-in (think SpaceDock) I can go either way - immediate listing into staging or not, and trust the team will make the right call on that specific process (and we can always adjust if people get grouchy).

BobPalmer on 30 Jun 2016

👍1

In terms of specifics, I assume the best option here is just to add two branches to KSP-CKAN/NetKAN called say "staging" and "experimental"? (Having entirely different repos would be needlessly complicated and overkill. Also for anyone interested, this is the workflow that would be required from the people with write access to the NetKAN repo.)

Cphoenix on 1 Jul 2016

Well no, it wouldnt be overkill. Just have one repo (experimental) use NetKAN as now, with NetKAN submissions from everyone, screened by the Maintainer for the relevant mod, while the other repo uses only .ckan files that have been declared stable by the Maintainer.

Blu3wolf on 1 Jul 2016

The reason I say it's overkill and complicated is because it would involve splitting the relevant history of each edit among different repos. That means no one would be able to get a complete picture of how new .netkans were added and eventually committed to the master branch without looking at two distinct repos. It'd add a lot of overhead for maintainers.

Edit: An (sorta) alternative workflow is just to cherry-pick commits from staging into a new branch, and then merge that into master. Would probably be a little clearer what's going on that way.

Cphoenix on 1 Jul 2016

.netkans would not BE added to the master branch of the stable repository. The stable respository would consist only of .ckan files already confirmed, by usage and testing either by the modder or maintainer, to work properly, having been generated and tested on the experimental repo, which would have the .netkan files and its generated .ckan files.

Blu3wolf on 1 Jul 2016

I hate to interrupt again with a question. Why can't we do this on the local client?

The reason I am asking is the following may help. I admit it has nothing to do with staging. I know that. However it sounds like staging to me. It can provide the same function. Plus we already have done a lot of the work. Although it needs tons more work. Here goes.

Right now. I can simply switch off the metadata version controls. To do so requires at the minimum. Knowledge of the system and use of the CMI. The normal GUI does not allow it. So I can do stuff that most people can't. In effect sometimes. I get access to mods before anyone else gets them. Anybody can do that. If fact we had a massive debate on how to control this. Right now it is not fully enabled as far as I am aware.

The key point is some get to install mods in CKAN that others can't. I am guessing there is only a few in relation to the general population of users. I am guessing that perhaps because they had to jump through technical hoops. They hopefully don't send mod authors crap bug reports.

Once that population users is happy. When there is no install bug reports coming back. We then do a minor automated update. Say in the .version file. Suddenly everyone gets the mod.

Here is another way of looking at it. Mods did not actually get deleted. They just got hidden in a place that a tiny number of people know how to access.

Is any of this relevant. If not that is ok. I can bugger off and try again. If it has got some use then I could do more research in this direction.

solarsootysmudge on 1 Jul 2016

I have another question. On the staging issue. We should be pushing the conversation to #1679. Just so @techman83 sees it in one place. There is also a nice definition of overkill there and how traditional staging is difficult to do.

Back on topic here. I thing we a now all agree that the policy on on opt out is correct. If mod authors have objections to CKAN we need to respect that. Details are irrelevant. It just the right thing do. it does not have to a permanent decision. We can switch it on and off. I think there been a convincing demonstration of this recently.

If there is any way to improve on the methods used to delist at will. Or sort out any other grievances. I sure the CKAN team want to listen.

solarsootysmudge on 1 Jul 2016

You think incorrectly, there. For certain conventional values of 'we', anyway.

Blu3wolf on 1 Jul 2016

I am only too painful aware of this and was trying to be diplomatic. I apologize if any offense was caused. Lets be honest here. I am not in any special privileged position here. I am just once guy that reads everyone's side of the story. I am trying to understand the majority whilst not ignoring the minority.

Oh course the words majority and minority are personally subjective.

Here is a bit of information that might change perceptions. I am disliexic disaletic dissalictic....

Oh sod it you all get the point. If I use the wrong words I am sorry.

solarsootysmudge on 1 Jul 2016

👍1

I have another question. On the staging issue. We should be pushing the conversation to #1679. Just so @techman83 sees it in one place. There is also a nice definition of overkill there and how traditional staging is difficult to do.

Thanks for that; I really have to stop posting things after I've done a bunch of overtime that day. My brain gets all mushy and forgets stuff. >.<

Cphoenix on 1 Jul 2016

For whatever communications are to happen between a mod's maintainer and the CKAN indexing team (disallow CKAN install, provide maintainer's metadata to override CKAN repo's metadata), may I suggest that a comparable additional mechanism is provided to communicate the same info inside the mod package itself (e.g. in KSP-AVC format), because:

this channel of communication is already always available; if the package can't be read then obviously there will be no CKAN install;
and this channel is already under the maintainer's control, without needing manual intervention from the CKAN team to make changes.

KSP-CSE on 3 Jul 2016

This method is already in use, and deemed not useful enough by modders.

Blu3wolf on 3 Jul 2016

@Blu3wolf - as in, there's already a way to specify, within a mod's own files, that CKAN should not install it, and CKAN already respects this setting where found?

[Edit - did you mean via a metanetkan? If I've understood them right, a metanetkan is still an item at the CKAN end, under the CKAN team's control; a mod maintainer wishing to opt out needs to alter something at the CKAN end - yes, as a one-off (barring errors) - and I can see why this might be not useful enough. I'm thinking more along the lines of a netkan in the mod package, if present, being the primary and compulsory netkan even if the CKAN repo netkan is not a metanetkan. A mod maintainer suddenly wishing to opt out could do so by adding a netkan to their own mod package as rapidly (or as slowly) as they like, not dependent on the availability and response time of any CKAN maintainers.]

KSP-CSE on 3 Jul 2016

Put two conflicting .version files in the mod release. If the .netkan (like most of them) uses the avc file for versioning, it wont know what to make of it.

Blu3wolf on 4 Jul 2016

That would be treated as a bug to work around, not a signal.

politas on 4 Jul 2016

👍3

That would be treated as a bug to work around, not a signal.

Very much agreed. The whole point of these discussions is to come-up with a system that works for modders as well as users (and the middlemen maintainers). Having to intentionally break the .ckan-generating bot to communicate anything to the CKAN team will just end in misery. 😢

Besides eventually someone on the CKAN team is going to think "Why the heck isn't the versioning working? Sod it, I'll do it myself." (Forgetting to check if there are two version files, or not seeing that there are two, etc.) Not ideal.

Cphoenix on 6 Jul 2016

👍1

@politas it might be worth updating the https://github.com/KSP-CKAN/CKAN/blob/master/policy/de-indexing.md file now that policy has stabilised. At present, the file contradicts itself - perhaps it would be clearer if the policy simply said that mods will be removed at the users request?

Blu3wolf on 17 Aug 2017

👍1

We've been running with the policy changes for over a year now with no further complaints from mod authors. I'm going to close this issue off now.

politas on 16 Oct 2017

🎉1

Then I will forward a PR to clarify the policy file - it currently presents two contradictory policies in the same document and asserts that both are the position taken by the CKAN leadership.

Blu3wolf on 16 Oct 2017

Ckan: Opt In / Opt Out | Required changes to CKAN.

Most helpful comment

All 108 comments

Related issues