Cirrus-ci-docs: BETA mode for new features

[cevich]

Description

In the light of https://github.com/cirruslabs/cirrus-ci-docs/issues/275 it would be cool if the user-community had a way to test upcoming new features. Would it be possible to have .cirrus.yml setting to enable this (e.g. 'beta: true')?

Context

The user-community could utilize 'beta: true' on specific/dedicated tasks which help validate their specific workloads, against future CI changes. This, along with communicating a list of upcoming features, could help reduce surprises as they roll into production.

[RDIL]

+1. Regarding that specific situation I know the integration tests are there to catch things like that, so that was a mistake that most likely won't happen again.

[fkorotkov]

I wonder how many of people will actually have beta enabled and then still it can break things for people who will never use it once a feature is promoted from beta.

Right now each feature gets it's own integration tests, in this particular case there was a lack of one for FreeBSD (my personal mistake that I though there is not much of a difference with Linux environment). IMO it's better to work on better coverage and faster reporting before allowing user customization that only few will use.

There is also an idea of having sampling integration tests for potentially breaking changes. Run new and old configuration side by side for some percentage of OSS builds to compare results. Costly but should help with deployment of bigger changes. But sampling integration tests are way down the roadmap. Doesn't seem like a good time investment at the moment.

It's unfortunate that #275 happened and I'm very sorry. As a follow up I've already added integration tests for FreeBSD to make sure any future changes to script execution logic will be tested on FreeBSD as well.

[RDIL]

I'm tied between the two, while beta boolean would be nice, real world tests would also be great, but that may confuse a lot of users.

[cevich]

Nope, not, never, nobody ever going'ta ever complain about more integration coverage :smile: Rather I was thinking, a beta feature toggle would help the user-community do some testing for you on big features / changes. Not necessarily small-stuff.

I though there is not much of a difference

Yes, I've heard: "assumption is the root of all evil" :blush: I too have been bitten by 'bash != bash' in many environments. Even between different versions of the same environment :confused:

Anyway, I agree, 'beta: true' covering a week-or-two worth of features is probably not worth it. But for big things, it might be a helpful way to get feedback beyond what your own testing is able to provide. Anyway, just an idea I thought I'd share, no worries if it doesn't make sense here.

[fkorotkov]

There is now experimental field on a task to enable and try new features if need to slowly roll them out.