Openj9: Proposal: "interim" monthly releases between quarterlies

[mstoodle]

I'd like to float an idea a few of us have been bandying about, to see what the community thinks of it. I'll mention this issue at today's Community call, and I'll bring the feedback collected here to next week's meeting. It would be nice to reach a consensus after the week, but let's see where we are at that point.

More frequent releases

OpenJ9 currently produces six releases a year - two corresponding to the new OpenJDK releases and four corresponding to the OpenJDK quarterly releases.  All of these releases are supported on a single code stream.  For example our April update produced JDK8, JDK11, and JDK14 binaries from the same OpenJ9 0.20 source code release.

Some stakeholders have expressed interest in more frequent releases.  A more frequent release cadence - monthly in fact - has historically worked well for IBM allowing users faster access to fixes and improvements.  It also provides a faster way to get critical fixes into production releases.  

For those reasons, I would like to propose that we produce "interim" release builds (better name welcome) that would include important bug fixes on top of the most recent quarterly release.  We've re-spun releases in the past to include fixes for issues found very late in the release cycle or even immediately after release; this proposal is to make these releases a regular feature of the OpenJ9 project to provide quicker access to critical fixes. This isn’t a radical suggestion, though I suppose it’s more typically done in projects with longer release cycles. 

As a straw person proposal, we could consider producing two point releases one month and two months after the last quarterly release. For example, the 0.20.0 release in April could be followed by 0.20.1 in May and 0.20.2 in June leading up to 0.21 being released in July. Then 0.21.1 in August, 0.21.2 in September, etc. Bug fixes would always be delivered to our master branch first with some fixes being selectively ported to an interim release branch. The September (and March) releases would be more troublesome, since we're already doing new feature releases (i.e. new JDK level) releases in those months.

This change would have some costs for the community associated with it, of course:
    - process to propose and accept bug fixes
    - person to cherry-pick fixes into the new release branch
    - person to assemble release notes (propose we keep it as simple as possible), documentation, and web site update
    - CI work to build the new release branch
    - increased pressure on our machine farm
    - people to triage and cope with any test failures on the release branch
    - more versions for team to be aware of when investigating problems
...but I think the value of monthly releases might be worth these costs, especially if we think about how we could minimize their impact.

There would also clearly need to be a discussion with the AdoptOpenJDK community, who would have to agree to produce binaries for these interim releases, which will introduce a similar set of costs for them too. I think we can only decide to do this in collaboration with AdoptOpenJDK, but I think the OpenJ9 project should agree on this as a desirable direction before we approach AdoptOpenJDK to begin that discussion.

Do we think the benefits of interim releases are worth the costs to manage those releases? I would especially welcome input from our users on whether they would see value in such releases.

Some alternatives:
    - status quo
    - move to continuous monthly releases (maybe reduces problem, but probably increases release management effort?)
    - only release “on demand” (harder to coordinate and plan with AdoptOpenJDK)
    - something else?

[andrewcraik]

So one question I have around the model for this where we are going to try and put the cost of porting patches to release branches if we ship the point releases off of the release branches and who is going to make the call on what is going in to a release branch.

The key theme under pinning this is that we, as a project, have a pretty fast moving codebase. This means that even over a month or two the APIs and conventions can change quite a bit. A contributor may not know/care how to back port a particular fix - who is going to do that work? Further, a committer may not be the best person to judge the merit of whether a bug fix should go into a service stream or not and there is the additional cost of the extra testing etc for delivery into the service stream.

Without considering the above it is hard to judge if the project could manage a branch-based monthly bug fix release cycle. The other alternative is to ship from head - the question there is how can we get enough testing to make sure the release is stable and how do we accommodate some of the bigger changes that may destabilize things for a longer period of time as they deliver.

I certainly see why people would want bug fixes more frequently than we currently deliver them, but the practicalities of how to deliver it is what I'm trying to work through in my head. I don't believe it is insurmountable, but the operational details will be key to how we can make such a change a success IMO

[pshipton]

The above reminds me we don't have Pull Request testing capability (jenkins ...) for branches. Backports either need to be tested manually before merging, or merged with subsequent release branch testing. The latter is what we do now for the quarterlies, but 99% of the time the change to the release branch is identical to the head stream change.

[DanHeidinga]

I've created https://github.com/eclipse/openj9/issues/9349 to discuss adding PR testing to release branches

[mstoodle]

I think the process to select and perform backports should be a collaboration among the project leads and committers with input from users and potentially helped by contributors where ever possible and where they are amenable. I would like to believe the selection process would take into account the extra amount of work required for a back port as well as the risk of doing that work when considering back port candidates.

The other alternative is to simply to continuous monthly releases. But whereas a "point" release I don't think requires the same Eclipse release process, continuous monthly releases would require that work (and reviews, etc.). I suspect, though I could be wrong, that the cost for backporting most bug fixes would be less than adding 8 more Eclipse releases cycles.

I think we should prioritize our user community's interests here. I don't think we need to unthinkingly cater to every whim, but I think we could and should do more than the current 3 month gap between continuous releases (especially given the velocity @andrewcraik mentions in his message).

[pshipton]

Eclipse moved to a yearly approval model, doing continuous monthly releases doesn't require any extra approvals afaik.

[andrewcraik]

I think the process to select and perform backports should be a collaboration among the project leads and committers with input from users and potentially helped by contributors where ever possible and where they are amenable.

This sounds great idealogically do you have any ideas on how we would implement this practically to keep the overhead low? Perhaps some kind of label and a business item in the weekly meeting to review the list and change the label to confirm a backport vs remove the label? Maybe an automated job to @ the submitter and other commenters when the backport review is added? I just think we need to be clear on who is going to do what when to get the decision made quickly since we won't have a long turnaround on backports.

continuous monthly releases would require that work (and reviews, etc.)

@mstoodle - in light of the comment from @pshipton have you changed your opinion on this? Does it alter the balance of one vs the other for you?

I think we should prioritize our user community's interests here.

Completely agree this really comes down to a speed of delivery vs stability issue both for features and fixes. I agree that with 3 month gaps we probably haven't quite achieved the right balance.

[DanHeidinga]

This sounds great idealogically do you have any ideas on how we would implement this practically to keep the overhead low? Perhaps some kind of label and a business item in the weekly meeting to review the list and change the label to confirm a backport vs remove the label?

Labels would work pretty good for this. There's already a backport:candidate label that could be added when a PR is merged in master. Once a backport is agreed on, we could add a backport:accepted, and maybe a backport:complete label after the backport is merged? And maybe a backport:rejected label for cases where we decided not to backport.

There should be a fairly low cost workflow that can be implemented with labels and maybe if we want to get really fancy, a new backports project kanban board.

[DanHeidinga]

Created https://github.com/eclipse/openj9/issues/9370 to track nailing down exactly what that work flow would be as we can do that after this proposal is accepted (or not)

[mstoodle]

@mstoodle - in light of the comment from @pshipton have you changed your opinion on this? Does it alter the balance of one vs the other for you?

I'm less concerned with the details of what goes in each release, though I suppose I did initiate the discussion by taking a standpoint. I am confident in our committers and leads being able to figure out how to arrive at a sensible definition of monthly content. I suspect "interim" builds would be more appreciated by users (but could be wrong...I know it flies in the face of "continuous"). I think it minimizes the risk that a user could hit regressions in multiple releases, but I think that risk is already pretty low for most of our users.

Given most of the feedback has been on "how" rather than "whether", I would like to bring this up again at the community call this week and decide whether we have enough consensus to initiate more "formal" discussions with the AdoptOpenJDK community.

I would characterize the feedback so far as mildly supportive with no objections, though I wish we could get more overt feedback rather than just lack of objection.

[SueChaplain]

One missing element from the list under "costs for the community" is the user documentation and the website. Moving from 6 releases a year to 12 releases would double the effort to close, test, and publish the docs and website. The difficulties I guess are along the lines that Andrew has discussed for code ... what needs double fixing and the timeline pressures that this presents for monthly releases. Docs work is always at the backend of a cycle, which carries a workload spike. I'm not saying this isn't possible, but it is another consideration to add.

We'll consider how docs might be handled. This shouldn't hold up more "formal" talks with AdoptOpenJDK - that is a much larger hurdle.

One thought:

Moving from 6 to 12 is definitely challenging for the project. Given we have 4 quarterly security updates, and 2 extra releases for the mid-Sept / mid-March feature releases, adding a further 2 to bridge those other long quarters, i.e. mid-Dec / mid-June, might be a sensible interim step?

[DanHeidinga]

Thanks for raising the documentation costs, @SueChaplain.

Moving from 6 to 12 is definitely challenging for the project. Given we have 4 quarterly security updates, and 2 extra releases for the mid-Sept / mid-March feature releases, adding a further 2 to bridge those other long quarters, i.e. mid-Dec / mid-June, might be a sensible interim step?

That's an interesting idea and makes a lot of sense from a spread out the project releases perspective but I don't think it meets the end-user needs as well. Users waiting on a critical fix in Feb on an existing release (8 or 11), won't be able to benefit from the new JDK release in March (JDK 14). Even though the project is already doing a release, it's not a version they immediately put into production if they are still on 8 or 11.

[SueChaplain]

Users waiting on a critical fix in Feb on an existing release (8 or 11), won't be able to benefit from the new JDK release in March (JDK 14). Even though the project is already doing a release, it's not a version they immediately put into production if they are still on 8 or 11.

I guess I didn't expand my suggestion enough. I was thinking 6 week cycles with the LTS releases shipped in March and Sept alongside the new feature release aswell.

[DanHeidinga]

Thanks for the clarification @SueChaplain

[0xdaryl]

Setting aside the operational mechanics, I do support adding at least one interim release between releases for the purpose of delivering a critical fix to an OpenJ9 consumer who cannot wait for the next release. I think it is an important safety valve for the project to have to deliver an out-of-band fix should it be required.

I imagine the number of fixes to backport in the interim releases will be relatively small, if not zero, for any given release. But the standard for backporting should be high to justify the cost of labour.

I am ultimately in favour of doing monthly continuous releases from HEAD, but I realize the project isn't ready yet from an infrastructural or resources point of view to support that. Performing interim releases first might help us refine our approach to delivering releases with increased frequency.

[dmitripivkine]

I agree with Daryl.
There is a fundamental problem with current release model (procedure). It is too heavy, not so much automated and required a lot of a manual labor. And we are practicing very early code split. And there is a set of requirements (mostly external) for release (including reliability). I think would be hard to achieve doubling or even tripling number of releases using this model without significant overload of people responsible for (even we move part of work for developers). Some automation might help obviously. I believe existing model can be used for preparing iFix builds (based on previous release one) with selected PRs on exceptional basis. And it might be not so easy too because of possible dependencies between PRs.
Looks like the alternative (and right) way (in theory) to have release based from HEAD (continuous in term that any HEAD build can be selected for release). An automated testing and generated update for documents would help (ideally with a few clicks). But this approach required changing of model: no early code split, loosening requirements - internal and external (is it possible? are we ready for this?).

[mstoodle]

Thanks for the additional feedback, everyone!

I want to be clear that the proposal was not to add 8 new releases that are like our current set of 4 quarterly releases (although that's listed as an alternative). I was trying to keep the new releases lightweight, with no feature content (so less documentation work, only release notes). As @0xdaryl suggested, I was imagining only a few high priority defect fixes making it into these releases; certainly not a "port every bug fix" or even "most bug fixes".

I suppose the frequency could be every 6 weeks rather than every 4 weeks, with the advantage that these interim releases wouldn't overlap with the feature releases. Of course, users will need to wait two weeks longer to get a fix for an important problem.

I haven't heard any strong objection to the proposal to add interim releases at this point, so I'm tempted to say we're ready to reach out to AdoptOpenJDK. Given the flurry of recent comments, though, I'll give it to the end of this week (Friday May 1) for anyone to express an objection to our moving forward along these lines.

[mstoodle]

One user endorsement via twitter: https://twitter.com/kjjaeger/status/1253172900837961733

[DanHeidinga]

Some issues that had been requested for backport:

• https://github.com/eclipse/openj9/pull/8598#issuecomment-587513950
• https://github.com/eclipse/openj9/pull/8367#issuecomment-591649374

I feel like there have been more leading to other discussions about potential .1 or .2 releases but can't find them now

[DanHeidinga]

I haven't heard any strong objection to the proposal to add interim releases at this point, so I'm tempted to say we're ready to reach out to AdoptOpenJDK. Given the flurry of recent comments, though, I'll give it to the end of this week (Friday May 1) for anyone to express an objection to our moving forward along these lines.

I think we have support to move forward with this proposal and start the discussion with the Adopt community. @mstoodle have you started that discussion?

Discussion on the mechanics of how the workflow of this will work can be continued in sub-issues:

• Create label based workflow for interim releases https://github.com/eclipse/openj9/issues/9370
• Enable Pull Request testing capability (jenkins ...) for release branches https://github.com/eclipse/openj9/issues/9349

[mstoodle]

The topic was raised at one of the AdoptOpenJDK TSC discussions (even before we had settled it here, actually), and I understand the general feeling was that they were open to the notion (@smlambert @sxa let me know if I'm mis-characterizing the conclusion of the discussion). We may need to do some work on our side to help smooth out our impact to the release process at AdoptOpenJDK before they would be willing to commit to a monthly release cadence. We'll need to join the discussion on the latest release retrospective at AdoptOpenJDK if we aren't already involved there.

As you've pointed out already, in parallel we should continue to work through the details on how frequently we want to introduce interim releases (monthly, every 6 weeks?), what we would want to put in interim releases (ship from master or back port select fixes), how we would decide what goes in, and how we can minimize the cost to everyone to produce interim releases. When we have a concrete proposal, at least on cadence, we should open an issue at AdoptOpenJDK to communicate our request and make sure the details are worked out.

I don't think there's a burning need to make this change immediately, but given all the things needed to line up perhaps we could target either August or October time frame for our first interim release? (October only because I'm not sure we want to start interim releases just before JDK15 is released in September).

[sxa]

I'm personally in favour of it from an openj9 perspective. Your summary of the TSC discussion is accurate, although the main discussion was triggered by another potential requirement for interim releases, rather than this requiremnet, and I mentioned that this was being considered for OpenJ9 more widely as well ...

[mstoodle]

Does anyone want to register their preference for 6 week (technically 1.5 monthly -- is there a word for that?) interim releases as opposed to monthly releases? Monthly helps keep the backports more likely to be straightforward, though I understand it's more work around the documenting of releases. But if we only backport bug fixes, then maybe the release notes can be as simple as the list of PRs that were applied (and maybe the GitHub issues they addressed, if specified in the PR)...

My vote is for monthly. I don't really want to make this into a formal vote, so maybe let's just gather strong opinions for or against monthly / 1.5 monthly releases to see if there's significant contention on this point?

[DanHeidinga]

I'm also in favour of monthly. It makes it straightforward to know when a build is likely to be available (mid-month, every month) rather than trying to figure out how to map every ~6 weeks from the previous release.

[lumpfish]

I would agree that monthly should be the goal. My concern would be can monthly be contained, resource wise? Perhaps if we started with one inter-quarterly release and if / when that becomes 'easy', increase the frequency to monthly?

[mstoodle]

Without any further feedback, let's aim for a monthly build, perhaps in August as our first one, and if necessary, we can choose to not do a September monthly release (since we'll be focused on the JDK15 release then) if the first one is found to be extra painful. Then we can get back to it in November.

[lumpfish]

Do you propose to fix a date in the month - e.g. the 'middle' Tuesday of the month which is the (quarterly / half-yearly) date for openjdk releases?

[DanHeidinga]

Do you propose to fix a date in the month - e.g. the 'middle' Tuesday of the month which is the (quarterly / half-yearly) date for openjdk releases?

That makes the most sense. We should pick the a date, like the 15th and aim to ship monthly releases on that day. This would line up nicely with the existing release dates of the quarterly releases

[pshipton]

I like Simon's proposal, the 'middle' Tuesday of the month, rather than the 15th. I propose we start the final builds at Adopt on this date, similarly to the quarterly releases.

• seems this is what OpenJDK does, it keeps things consistent
• it avoids rescheduling due to Monday or Friday holidays, or when the 15th is on the weekend
• it gives Tues - Thursday/Friday to complete and triage the builds, without spanning a weekend

[lumpfish]

As far as I can tell, openjdk quarterlies are released on a Tuesday, no earlier than 14th (so a range of 14th to 20th). That would give us the dates (through to the end of 2021) of:
Date | openjdk release
-- | --
14 July 2020 | <-- openjdk quarterly
18 August 2020 |   |  
15 September 2020 | <-- jdk15 (tbc)
20 October 2020 | <-- openjdk quarterly
17 November 2020 |   |  
15 December 2020 |   |  
19 January 2021 | <-- openjdk quarterly
16 February 2021 |   |  
16 March 2021 | <-- jdk16 (tbc)
20 April 2021 | <-- openjdk quarterly
18 May 2021 |   |  
15 June 2021 |   |  
20 July 2021 | <-- openjdk quarterly
17 August 2021 |   |  
14 September 2021 | <-- jdk17 (tbc)
19 October 2021 | <-- openjdk quarterly
16 November 2021 |   |  
14 December 2021 |   |  

For the new releases (jdk13, jdk14 etc.), the 14th to 20th rule does not always apply (jdk9 was released on 21st Sep, jdk11 on 25th Sep) - so there may be a decision to be made whether to align the new release and the monthly release in Mar and Sep.

[mstoodle]

Looking at the potential for an August monthly release, I suspect the anticipated flurry of activity as the AdoptOpenJDK community plans their move to the Eclipse Foundation [1] probably means an August release is unlikely. Until we have fixes identified to put into such a release, we don't have a strong need to do a release either.

That probably means the reasonable next time to revisit a monthly release would be in the November time frame, at which point hopefully the Eclipse Adoptium dust has settled and we can plan whether a November interim release makes sense for both OpenJ9 and Adoptium.

[1] https://blog.adoptopenjdk.net/2020/06/adoptopenjdk-to-join-the-eclipse-foundation/

[knn-k]

Hi,
What is the recent outlook for monthly releases after moving to Adoptium later this year?