Syncthing-android: Better Release Management

[vext01]

Hi,

Thanks for your ongoing work on Syncthing for android. It's awesome.

The non-android version of Syncthing will upgrade minor versions, but not major versions automatically (unless a build time switch is flipped). To upgrade a major version, user intervention is required. This means your nodes will always be able to talk to each other.

I package Syncthing for OpenBSD. Under this scenario, we disable auto-updates. Also note that people may be using packages up to 6 months old if OpenBSD-release is used.

The problem I have is that I can't keep up with Syncthing for android. The android client updates major versions automatically, meaning that when this happens, my android devices no longer speak to my other nodes, as they are running older (incompatible) Syncthing versions. This happens silently. You would have to log in to the web GUI to notice.

I think that the android version should be distributed in a staggered fashion. One package for the current protocol version, and another for the legacy protocol version. When a new major version is released, you would retire the legacy version, make the current version the legacy version, then the new release is the new current version (wording could be better, but I hope you get the idea). The client should not upgrade itself to a new protocol version automatically. Instead, when a protocol breaking update becomes available, you could display a notification asking if the user wants to update.

Cheers.

[licaon-kter]

Why not read the release notes and NOT update the Android client?
Your Android package manager of choice has such options.

[vext01]

Not recieving updates at all is not the right way to go about this. E.g. Vulnerabilities certainly _do_ need to be patched.

[wweich]

Google doesn't allow Apps to update it's components outside the play store, so the App cannot update the core syncthing on it's own, so no way to display a warning.

Just disable auto update for the syncthing app. Every time there is an update and you click on "update all apps which need update" the play store will ask if you want to update syncthing or skip it. Then you can check if the new version is still compatible and update it.

It's not that the Syncthing App is updated all the time, so this is not really that big of a deal.

[capi]

So basically your idea is to have one app per incompatible protocol version in the store? It is an approach, but has the problem that users won't get the newer version without explicitly installing a new app from f-droid or Play Store.

The protocol changes in 0.14 were introduces to that in the future there should be fewer needs for breaking protocol changes. They want to stay in 0.14 for "long", until they need a breaking 0.15 change. For instance, 0.14.1 introduced a huge change that's fully backward compatible to 0.14.0 and would have been a breaking change in the future: delta indexes (especially interesting for us on mobile).

So given the huge amount of other issues, I think we should not invest too much time in this at the moment (while the suggestion in itself is very valid).

[licaon-kter]

@vext01
It's a matter of people and time, if you have both do fork this and backport fixes for 2 versions, since the last 6 months cover releases for 0.12, 0.13.

[C0R10L4NU2]

IMHO, the issue is much bigger than @vext01 describes. The debian repo went to 0.14 last week, yet the fdroid repo has still no 0.14 version.

The same happened at the last switch from 0.12 to 0.13. That means every few month no sync between debian and android devices for about a week. I would consider that a huge bug.

The only workaround here seems to be disabling the debian repo. Which is quite a horrible security practice.

IMHO, maintaining 2 forks would not be necessary. It would be sufficient to simply keep legacy builds available.

@vext01
you could switch to Fdroid's syncthing. It not only lags in updating (sorry for sarcasm), but it also provides legacy builds and gives you a fine-grained control over when to update.

[capi]

In Play Store we have two options:

1. Bundle _both_ binaries and let the user chose. Since the bundling of 0.14.0 was pretty straight-forward, this would be available within a day or two after release. Disadvantage: both binaries included, increases the APK size considerably. Also it requires, that the application can handle both REST API versions (would have worked in this version). Also requires, that we add the UI for selecting the version, for changing the binary. All doable, but requires effort.
2. Have two different APKs that need to manually installed to upgrade from one version to another. This requires that we need to point people to the new APK somehow. We also need to create new packages at Google Play Store for every branch. It also makes it hard to migrate configuration from one application to the other.

f-droid better handles the issue, since you can downgrade versions, which you cannot in Play Store. But they lack maintainers as well, causing the current delays.

All in all, I think that at least for Play Store we cannot handle the effort with the team we have and people need to manage their upgrade strategy of Syncthing. You can disable auto-update per application in Play Store. Syncthing itself does not auto-upgrade between incompatible versions.

Packagers for e.g. Debian need to thinkt about this strategy as well. Because what if I have a Debian and a Fedora machine? The updates based on packages will not be available at the same time as well.

Packaging for instance for PostgreSQL, MySQL and other applications, that break between versions, consider this. They have different packages for the individual branch, with a virutal-package that combines it. But so it's a user's choice to update from psql-9.2 to psql-9.3.

All in all, I'd say it's not only a Android problem, but a general problem for all supported platforms.

[gdt]

For the most part, I think this is not a syncthing-android problem. The root cause is that syncthing has made incompatible protocol changes on a regular basis, but hopefully that will be far less frequent now.

In pkgsrc (a multi-OS multi-CPU packaging system), I've adopted the rule not to update syncthing itself until the new syncthing-android version appears on f-droid and it's been tested with the new syncthing to be committed to pkgsrc. With pkgsrc (and f-droid), one can of course go back to old versions.

I would be tempted to suggest publishing a beta with the new version, so that f-droid build issues can get sorted out, but that won't help because other packaging systems adopt versions with breaking protocol changes immediately. This really needs either versioned packages in (perhaps all but one) packaging systems, or some sort of 14-30 day notice period for testing prior to release. This all gets to be too much work pretty fast, and the only reasonable answer is to avoid breaking protocol changes, which happily seems to be the plan now.

[Nutomic]

@gdt We've done beta releases on F-Droid a few times. Unfortunately, they have to do that manually, so they don't really want to handle that.

[vext01]

Having thought about this some more, I have a couple of new insights:

The real problem is that the syncthing protocol has (historically) lacked backwards compatibility. The fast release cycle, exacerbates the problem. Stock syncthing tries to work around this by not upgrading major versions automatically.

Another issue with not having backwards compatibility is that for a single device to talk to servers with different protocol versions, you would need to use >1 instance of syncthing. It's not hard to imagine scenarios where you need to talk to one cluster which uses 0.13 and another than uses 0.14. Once you start thinking in this way, your realise that the "holding back package updates" strategy is moot, as a syncthing binary can only ever target one protocol version (and I still think holding back updates is a bad idea for security regardless).

These problems are not unique to Syncthing for android. It merely inherits these problems from the main syncthing implementation. The difference is, the google play store moves much more quickly than (some) distribution package managers (e.g. Debian, OpenBSD) and this emphasises the problem.

I think the proper solution is to start enforcing backward compatibility on the syncthing protocol itself. And I know... that is hard...

Failing this. Packagers (Linux, BSD, Android) would need to package Syncthing in such a way that multiple instances can be run so that different protocol versions can be used. This would mean a separate syncthing-0.13 and -0.14 package (how far back do you go?). All in all this isn't super practical either.

(To clarify a prior comment in this thread. I am not suggesting that the native part of syncthing should update itself -- that's not a good idea.)

[mulander]

Agreed with @vext01 the lack of backwards compatibility is what causes the biggest issue. Currently a syncthing major version bump ignores older clients forcing everyone to update or not being able to sync. Maintaining a N-1 backwards compatibility would still allow the protocol to move quite fast and deprecating older versions (N-2). This would smooth out the adoption curve and would not require changes in the tempo of deployment. Obviously this is associated with more effort required to maintain that compatibility.

[capi]

As pointed out, bundling two versions of native binary would be a workaround, at the cost of vastly improved APK sizes. And the additional effort to support two REST API versions plus additional UI. Given the current development speed, I'd say it's not going to happen so soon, but we can keep it on backlog. Maybe someone would contribute code for this?

Regarding the arugments of @mulander, I'd suggest opening an according thread in the forum or opening an issue in the main project, it's outside the scope of _this_ project.

And (again) pointing out, that 0.14's goal was to build a basis for improved backward compatibility. So let's see how fast we'll hit a 0.14 -> 0.15 transition.

Also to quote from the README of the Syncthing project:

The two are evolving together; the protocol is not to be considered stable until Syncthing 1.0 is released, at which point it is locked down for incompatible changes.

[Nutomic]

@calmh: Can you say something about incompatible versions in the future? Will we see them as often as now? Is it something we should handle in the Android app?

[vext01]

@capi thanks. In light of the quote you pasted, I guess we should wait until 1.0.

[calmh]

@Nutomic there should be fewer and fewer incompatible releases

[Nutomic]

@calmh great, I think the current approach should be fine then.