Tribler: Improvement Request: Monthly Releases

[drew2a]

We want to deliver one release per month.

Given

Now we have 7 developers that actively contributing to the codebase.

As a source of tasks we have 324 open issues that partially sorted in several milestones:

• Backlog https://github.com/Tribler/tribler/milestone/19
• V7.6: Stability, usability, performance https://github.com/Tribler/tribler/milestone/35
• V7.5: core refactoring https://github.com/Tribler/tribler/milestone/31
• Next-next release https://github.com/Tribler/tribler/milestone/36

Problems

• It is unclear, what tasks we plan to deliver and when
• Release has no schedule
• Task priority is implicit
• Changelog is missed
• Rules are not transparent

One step to improve the situation

As a transitional form from the current delivery process to stable and planned delivery, I would propose the following:

1. We have to schedule the release date
2. We have to reorganize milestones
3. We accept one new rule: if you made a change to Tribler codebase, you should add the corresponding issue or PR to a milestone

We have to schedule the release date

The monthly release will change the rhythm of development.
The new mindset should be like: "I can spend a month on something that will improve the Tribler and it can be delivered at the end of next month".

From the beginning, it would be difficult to adopt, because now we have no release schedule.

To help with this issue I propose to add two notification to slack:

• Start release date is 27'th day of each month
• Publish release date is one week later (3-5'th day of next month)

Start release

On this date, we have to start a new release branch from devel.
We will have about a week to test it and stabilize it.

Publish release

On this date, we have to close the release branch by merging it to devel.
The new version of Tribler should be uploaded and be available to users.

Branching model

I suggest using OneFlow as described in https://github.com/Tribler/tribler/issues/5569 and migrate from devel to master branch. But it is less important and optional.

Versioning

We have to follow SemVer notation and, in general, we'll be releasing an increased MINOR version every month.

Eg:

• 7.6.0 November
• 7.7.0 December
• 7.8.0 January

In the case of hotfixes, we have to release an increased PATCH version:

• 7.6.1 November
• 7.6.2 November
• 7.6.3 November

We have to reorganize milestones

We have to use:

1. Backlog milestone. It will contain all issues related to development.
2. Top Priority milestone. It will contain top priority issues, that will be done likely next month.
3. MAJOR.MINOR.PATCH milestone. It will contain all completed tasks that will be included in the corresponding release.

Top Priority milestone can be used as a tool for prioritizing issues.
MAJOR.MINOR.PATCH milestone can be used as a Changelog.

The flow is the following:

• If we create an issue, that related to development (not a topic to discuss), it has to be added to the backlog
• If we want to increase the priority of an issue, it has to be added to Top Priority
• If a developer completes the issue, it has to add the issue to the MAJOR.MINOR.PATCH milestone

It is possible to add an uncompleted issue directly to the MAJOR.MINOR.PATCH milestone if you promise to solve it in this particular version.

Conclusion

If we implement this step, then we can resolve four of the five described issues:

• Release has no schedule
• Task priority is implicit
• Changelog is missed
• Rules are not transparent

Related topics

• Milestones and Estimation https://github.com/Tribler/tribler/issues/5558
• Branching model https://github.com/Tribler/tribler/issues/5569

[synctext]

:cool: The idea of having November, December and January releases makes things very concrete. Easy mental model for deciding what goes into a release. :white_check_mark:

Lets do four releases with this process and after the February release we do a detailed "lessons learned" reflection and try to nail things down (further tuning). The first days of each month we need a process where everybody "commits to a commit". 30 open issues in Next-Next and 131 in https://github.com/Tribler/tribler/milestone/35 need to be slimmed down. How would we do that monthly road-mapping?

[devos50]

It would definitely be nice to have a better idea when things will be released. This would also help to schedule the developments around the bandwidth accounting mechanism/reputation mechanism (the whole token economy stuff).

if you made a change to Tribler codebase, you should add the corresponding issue or PR to a milestone

I strongly agree that we should add every issue, and even PR, to a milestone. I think we are already trying to do so, but it is often unclear to which milestone an issue should go.

How would we do that monthly road-mapping?

Well, the "old way" to reduce the list of issues was to sit together and for every single issue decide whether we should fix it in this release (and immediately assign someone), postpone it or close it. This has been the methodology since Tribler 7 has been released, but it results in some issues being moved to a next milestone more six times already. Essentially, such issues will never be fixed because they are either very minor (e.g., GUI stuff) or close to impossible to fix.

[ichorid]

I see two potential pitfalls with "monthly releases" approach:

1. What if there are very little changes made to devel during a month? Should we skip the release in that case? 7 developers is not enough to justify a release _every_ month.
2. What if a feature cannot be built completely in a month? To prevent rebase hell, we will have to use Feature Flags

Personally, I don't mind faster release cycles if we're not going to rush releases just to have a release.

[ichorid]

Having more conventions that can make the team less reliant on communication is even more important now, when we are forced to work remotely.

[drew2a]

@ichorid

What if there are very little changes made to devel during a month? Should we skip the release in that case?

My position here: we have to release it.
Even if it will be only one developer that will fix only one bug, it might help a dozen users.

In Privatix, we had about 4 developers, and we released every two weeks.

What if a feature cannot be built completely in a month? To prevent rebase hell, we will have to use Feature Flags

Partially agree. If a feature cannot be built completely in a month, then just don't merge it to devel. If you merging something to devel it shows that you want this change to be released in the next release.

If you are afraid of "rebase hell", don't do rebase. Just merge devel once a week to your branch. It should help your branch keeps up to date with devel.

[drew2a]

@synctext

How would we do that monthly road-mapping?

Planning is out of the scope of this IR because planning is always a quite complicated thing. Especially in teams that haven't this process yet. Especially during the new work environment (remote).

We should go step by step. Like one small management improvement each month :)

In addition, we can add two meetings:

1. The same date that release starts: Month Retrospective and Discussion about Month Results.
2. Start of the next month: Planning meeting. "What should we deliver at the end of this month?"

[ichorid]

Well, at least for 7.6 there is a slight technical problem: our directory versioning system forks config dirs on minor x.Y.z versions. Config dirs can get pretty big because of the channels. Like, 4-8 GBs. There is still no procedure to garbage collect the older config dirs. We need to develop a GUI popup for that. Having this GUI subsystem is mandatory for monthly releases.

[tuqueque]

Hey, sorry for chiming in here when I'm not really involved with the project, I'm "just" a casual user of this program, but part of being a casual user is that I'm a Linux Manjaro user and the .deb version you offer in the releases section doesn't work with any non-Debian based distro (a pretty big portion of the Linux user base, BTW)... So... Would you consider releasing an .appimage version for Linux users instead of a .deb?... That would make Tribler work pretty much with any major Linux distro out there, plus it also takes care of the usual "dependency nightmare" of Linux.

I'm not a programmer, but as far as I know, creating an appimage executable is a relatively straightforward process. There are several tools made for that purpose.

Thank you in advance.

[hbiyik]

May be a remark as a as a user, please dont exhaust the expectation of the userbase with frequent releases. I know devops is super cool, and everyone wants to hop in this train but it is a sharp knife. If it is executed without the necessary infrastructure to roll the releases with few user interaction, this ends up badly and generally organizations are pushed into a situation where they are trying to manage bunch of versions with lower quality attributes and bigger quantities. So not only user base is exhausted it brings more pain than gain. Generally this causes loss of sales, and companies are faced big financial consequences when motivated only with hype.

There are also good examples as well, may be i am painting a dark picture, but most of those examples are the ones where they roll auto updates with an opt in/out config, generally for broad target with a single managed source code.

[devos50]

Would you consider releasing an .appimage version for Linux users instead of a .deb?

Thank you for your suggestion! We definitely have plans to distribute Tribler beyond Debian-based platforms (e.g., Fedora/Arch Linux). However, we found out that this requires more work than we thought. While it might look easy to build an appimage, Tribler has many dependencies which, unfortunately, complicate this process.

For the Tribler 7.6 release, we do have some plans/preliminary builds for a Snap image though.

May be a remark as a as a user, please dont exhaust the expectation of the userbase with frequent releases.

Valid point! Currently, our release cycles are long, yet every release has a substantial amount of new features/bug fixes. For the past two releases, I would say that this was necessary since we had many breaking changes that could not be shipped incrementally (i.e., moving to Python 3 and asyncio). "release exhausting" is definitely something I can relate to and that we should take into consideration. Some of the software that I'm using has new releases every few days. Making sure one always has the latest version installed becomes a chore that way.

[drew2a]

Accepted