October: Speedup bleeding edge updates releases

[ayumi-cloud]

The reason to open this issue up is because I would like to request a change in the way the October team handles Milestones in this repo.

I read a statement from daftspunk in Slack saying releases should be every 1-2 weeks.

If we look at the current Milestone v1.0.465 see here: https://github.com/octobercms/october/milestone/12

It currently says the following info: 36 open and 28 closed.

The problem

• The problem is pretty simple, the current method doesn't work and it's _very slow_!

• There is a _long delay_ between each release to complete most of the Milestone.

• Another issue is testing.

• My finished pull requests are now sitting there for many weeks! They should have been added to the bleeding edge updates section! Instead they will continue to sit there for more weeks until the _Milestone is full_.

• Many people are working on projects for clients and discover a issue or bug in October and raise an issue and then submit a pr. It affects their relationship with their clients, having created a pr and it sits there! When they need the pr added to the cms via bleeding edge updates to continue work on their projects.

Solution

I want to request the following solution it's pretty simple!

Let's break up the Milestone's into groups of max 10 pull requests.

They can get released into the bleeding edge updates quickly every 1-2 weeks.

Testing

In real life you can't expect every release to contain no bugs or errors - that's not how the real world works!

The pull requests sitting in the Milestone with the labels Testing needed have been checked by the admins and there is a really small chance of an error! There is no need to have them sitting around waiting for even more testing before adding them to the bleeding edge updates and then having another round of testing!

All testing needed labels should not just sit there and do nothing! Remove this pointless step and
push them out to the bleeding edge updates and let's get the October community to test them in the wild.

Grey area

Having Milestone pull requests just sit there with testing needed labels is not working, no one can be bothered with this step.

Forcing these pull requests into the Bleeding updates will get people more pro-active into the testing process and help ease the work-load for the admins.

Bleeding updates vs stable updates

Bleeding updates should allow a small set of errors! This way the community can give feedback and increase the speed of the stable updates!

Currently, Bleeding updates are being treated like Stable updates and trying to remove every single error! This is an overkill and a bit ocd.

Discussion and action

1. Limit Milestone pull requests to 10.

2. Remove Testing needed label (by this stage it's ready to be added to the Bleeding updates for testing instead).

3. When 10 pull requests have been completed they should be merged into the Bleeding Edge updates and the Milestone reset to the next ~Semantic version number.

4. Security and vulnerabilities should be fast tracked through into the bleeding edge updates and not need to wait for 10 completed pr's.

5. Discussion and input from community.

[bennothommo]

@ayumi-cloud I'll let @LukeTowers and @daftspunk come to a decision on this one way or the other, however, I just wanted to add a couple of points:

• Irrespective of whether we continue the current release pattern or implement yours, the issue of testing (and the lack of testers) is still going to remain. That, I feel, is more critical than trying to speed up releases.
• The purpose of "bleeding edge" updates is not to put untested code to the masses - it's to find bugs that may result from other factors, ie. plugins, unique setups, configuration, etc. It's synonymous with beta releases.
• People who want the most absolute bleeding edge code are more than welcome to use the instructions here to set up Composer to retrieve the latest code from the develop branch every time they update their Composer dependencies.
• Security vulnerabilities that are properly disclosed are resolved and released as soon as possible in a new release.

[ayumi-cloud]

@bennothommo interesting thanks for your extra info and input - we're going to try that instruction setup via composer so we can test and use the latest develop branch. That would definitely help us!

[prhost]

I would like to comment if you allow me, I believe that testing at the same time is necessary, there is a delay problem and people to test, but releasing something in beta for people to test, where there may be places that contain problems and no one will test, what would be the parameter of this? the time? I mean, for any adjustment to go from beta to stable without anyone testing, who would control it? Sending broken things to stable is a problem. I would not like to update my CMS already in production and it will explode. Your idea is good but it would be necessary to mature. Perhaps in my opinion, put unit testing for example as mandatory for pull requests by some criteria.

I believe that composer is the temporary solution for you @ayumi-cloud I use it whenever I want to upload something more urgently while the pull request is not approved.

[ayumi-cloud]

@prhost thanks for your input.

I'm just going to write a comment, to be a bit clearer of what I was trying to raise.

I was thinking the following:

1. Websites in production mode should use stable releases and not bleeding edge updates.

2. Websites in development mode can use bleeding edge updates and be willing to beta test bleeding edge updates.

No production website should ever explode or break due to stable releases be fine 99.99% and bleeding edge updates being 98-99% fine and allowed to be rolled back or updated.

1. Most PR's have been tested by at least one or more admins and then a label of testing was getting added. Asking for extra testing, but I feel this is a not needed stage and the next level of testing can be in the bleeding edge. Having created many PR's for this repo, I can say the high quality of the releases and the amount of testing gets done on the PR's before they get a testing needed label is the reason I can say this extra step is not needed.

2. The worry I had before I wrote this issue was the following: There was around 40-50 PR's sitting there waiting to be merged in a milestone, if most of them had issues, then it would be a real pain to sort out in the bleeding edge updates, hence breaking them up into chunks of 10 or less. Makes the max number of errors 10 or less and prevents any issues of a Worst-case scenario.

3. ~Can't say anymore as I leave it to daftspunk and luketowers to decide. (Have complete faith in what they decide).

[prhost]

Yes production only stable versions, but if there is no one to review it can happen to changes from bleeding edge to stable without having the necessary amount of tests.

Not everyone has good quality in their PRs and they test before sending. Anyway, we need moderation, or the idea of requiring at least one unit test as a complement to that PR. It would be good for CI/CD, but it would generate many unnecessary unit tests in the future, but I believe that a manual scan, merging and discarding tests from time to time would solve this.

[ayumi-cloud]

Not everyone has good quality in their PRs and they test before sending. Anyway, we need moderation, or the idea of requiring at least one unit test as a complement to that PR. It would be good for CI/CD, but it would generate many unnecessary unit tests in the future, but I believe that a manual scan, merging and discarding tests from time to time would solve this.

Sounds good.

[bennothommo]

@ayumi-cloud

Most PR's have been tested by at least one or more admins and then a label of testing was getting added. Asking for extra testing, but I feel this is a not needed stage and the next level of testing can be in the bleeding edge.

Just on this point, we only put Testing Needed on PRs that we have not yet tested ourselves and are asking for community effort in testing. If we ourselves test something and confirm it working, it generally gets merged immediately.

[ayumi-cloud]

@bennothommo thanks for clarification, I didn't realise that.

---

Was just looking at:

    /*
    |--------------------------------------------------------------------------
    | Bleeding edge updates
    |--------------------------------------------------------------------------
    |
    | If you are developing with October, it is important to have the latest
    | code base. Set this value to 'true' to tell the platform to download
    | and use the development copies of core files and plugins.
    |
    */

    'edgeUpdates' => false,

Maybe could update the description to warn people not to use bleeding edge updates in production mode (sometimes it best to state the obvious).

[LukeTowers]

@ayumi-cloud the problem really comes down to what we actually use edgeUpdates for. We are of the opinion that develop is the bleeding edge, and any build that gets released (pending or otherwise) should be as stable as possible. The purpose of edgeUpdates is actually to have our better equipped users be the last line of defence against any issues that could possibly be found in production.

Essentially, I would like to see every single user that has their own development resources such that they can respond to issues with updates in a way that doesn't affect them that much always use edgeUpdates in production. If you want to use the absolute latest of October (which we strive to keep as stable as possible too, but at least it's not subject to the delays of finalizing a build), then always use the develop branch (or even the one-off upgrade branches like for L6)

[ayumi-cloud]

@LukeTowers I'm on a agree to disagree with your comment. Yah I get what you saying and agree with it. But you only need to open the pull request section and see there is a real issue with testing and a blockage going on!

See screenshot, loads of pr's with the Testing Needed labels just sitting there gathering dust and no one's really bothered in testing them. Probably because we are all really busy with work.

Take this example, we need one of the pull requests asap and it has a Testing Needed label on it right now. We need this feature in one of our production websites. We could grab the code from the develop branch into the production website and start using that pr. But what's to stop an admin suddenly changing the code in that pr or changing their mind that it's not a good idea now etc. and then the production website now has this issue.

My point to my previous paragraph is that there needs to be a middle ground to get more people to test these pr's and speed up the process. Adding the develop branch direct to a website carries more risk than it needs - I think we could find a better solution.

[edit] I would rather add a beta feature from a bleedingEdge update than a beta feature from a develop update into a production website - I guess that's what I'm kind of saying.

---

Can't October just create three different levels then:

1. Stable version (for production and develop modes - error rate less than 1%)

2. Bleeding Edge version (for production and develop modes - error rate less than 2%)

3. Experimental version (for develop mode only - error rate maybe higher than 2% at times)
   Used to help speed up releases and get more October Guru's involved in the final pr stages, like a staging ground.

[prhost]

There is level 3, it is exactly you to fork from october and work on the branch dev. I believe the Testing Needed flag is to say that a review is necessary, that is, a test. This is also common, no one approves a PR without first looking. Now what can be done besides the unit test? That also requires greater knowledge of who is going to do PR, and that can be a problem. You need to reduce the severity of the test on PR.

Perhaps then create and convene groups of testers and mark them as reviewers in the PRs, after that define a minimum amount of approvals for that PRs to be dipped. I'm just "throwing ideas in the air" hehe.

P.S. You can also checkout and merge all open PRs.

[ayumi-cloud]

@prhost thanks for throwing out some ideas - much appreciated.

[LukeTowers]

@ayumi-cloud

Take this example, we need one of the pull requests asap and it has a Testing Needed label on it right now. We need this feature in one of our production websites. We could grab the code from the develop branch into the production website and start using that pr. But what's to stop an admin suddenly changing the code in that pr or changing their mind that it's not a good idea now etc. and then the production website now has this issue.

This is exactly what the Paid Support program is designed to solve. https://octobercms.com/premium-support

If there are PRs that your company needs to be running in production and needs them to be reviewed by the core team, then they should pay for the core team's attention to those issues. No amount of changing how we issue releases is going to change the facts of how much time any one person has nor will it change how we decide to prioritize our time.

Ultimately the bottleneck remains the same, that the core team does not have an infinite amount of time to dedicate to solving everyone's issues. As such, for your specific situation (and anyone whose business depends on October) I highly recommend you pay for the attention of the core team through the premium support program. That way, your issues take priority and you're guaranteed the attention of the core team so you're no longer limited by the time that the core maintainers decide to volunteer to resolving issues.

[ayumi-cloud]

Yah very good point, going to pass your comment to my office managers to read! Though our managers announced any business changes will be done after covid-19 has finished.

[github-actions[bot]]

This issue will be closed and archived in 3 days, as there has been no activity in the last 30 days.
If this issue is still relevant or you would like to see it actioned, please respond and we will re-open this issue.
If this issue is critical to your business, consider joining the Premium Support Program where a Service Level Agreement is offered.