Mumble: Update OpenSSL binaries for 1.3.1

[SC1040-TS2]

Context
The current OpenSSL library implemented into Mumble (1.0.2k as of 1.2.19) has had an expired long-term support for a few months now. With this in mind, and given the latest OpenSSL version supports TLS 1.3 among other features, it is highly recommended to update to the newer binaries for security purposes for the 1.4.0 Mumble client release.

Describe the feature you have in mind
Update OpenSSL binary packages to 1.1.1e or a later current version of 1.1.1.

Describe alternatives you've considered
If not retention of the current unsupported library or moving to a different TLS library, there may not be any alternatives.

Additional context
https://www.openssl.org/blog/blog/2018/09/11/release111/ was the initial release announcement. I have yet to find the specific release notes for 1.1.1e at this time, but will edit it in later if possible.

[Krzmbrzl]

I assume you are referring to the OpenSSL that is built into the windows application? 'cause to my knowledge we use dynamic libraries on Linux and thus we don't have influence on what version is actually being used.

I assume that we'll update the libraries for the static build as soon as the new build system (cmake) has been integrated.
/cc @davidebeatrici @ZeroAbility

[SC1040-TS2]

Yes, I am, @Krzmbrzl, for my use case is the Windows x64 build. However, a basic Linux end user whom does not use dynamic libraries may also be affected.

Based on prior version release notes found on the old wiki.mumble.info site, 1.2.16, 1.2.17, and 1.2.19 were the last times major releases of OpenSSL binaries were implemented, as evidenced here.

1.2.16(Relevant because some servers and clients are still on it.): https://wiki.mumble.info/wiki/1.2.16
1.2.17(The HopelessDemocracy.com server is still on it.): https://wiki.mumble.info/wiki/1.2.17
1.2.19(Most non-1.3.0 servers and clients still use this version.): https://wiki.mumble.info/wiki/1.2.19

[SC1040-TS2]

Also keep in mind that updating the static binary packages may or will help open the door to resolving https://github.com/mumble-voip/mumble/issues/3918 in the future.

[Krzmbrzl]

What has to be kept in mind though is that we can only use features of OpenSSL that are included in the version that ships with the oldest Ubuntu LTS. Right now this is Ubuntu 16.04 which ships with OpenSSL 1.0.2g (https://packages.ubuntu.com/xenial/openssl).

This doesn't mean that we can't include newer OpenSSL versions in the windows build but in terms of what features we can implement into Mumble, this is the restriction.

[toby63]

@mumble-voip @Krzmbrzl @davidebeatrici
Is also a problem in the linux static server builds:

SSL: OpenSSL version is 'OpenSSL 1.0.2k  26 Jan 2017'
[...]
Murmur 1.3.1 (1.3.1-rc1) running on X11: Debian GNU/Linux bullseye/sid: Booting servers

I don't know if thats possible, but could you at least include "OpenSSL 1.0.2u" into the static builds (asap, for 1.3.1 release) (Linux static server and windows as well)?
See Website:

20-Dec-2019 | OpenSSL 1.0.2u is now available, including security fixes

Also a note for downloaders of a static build would be nice.
Using EOL software depencies is considered a security issue by default.

Edit: Should also have a priority label, at least priority/P1 - Critical.

[toby63]

@Krzmbrzl @mumble-voip @Kissaki @davidebeatrici
You should at least switch to openssl v.1.0.2u (best bad option) and that should already be implemented in Release 1.3.1.
So I recommend a milestone change (1.3.1) and setting the priority (at least P1, maybe even P0-blocker).

For those arguing that it should wait till #3918 is solved, I disagree.
You are providing an openssl version from 2017(!), so you should at least switch all windows users and static server build users on linux to the newest available version (for 1.0.2), which is 1.0.2u (from December 2019) asap.

Update:
Found this Issue #3053, does that mean that mumble can already be compiled with newer openssl (Issue metioned 1.1.0e)?

[davidebeatrici]

Yes, Mumble can already be compiled with OpenSSL 1.1, but doing that for 1.3.1 would require us to update Qt as well, dropping support for Windows XP.

1.4 will, of course, be released with up-to-date dependencies.

[toby63]

@davidebeatrici @mumble-voip @Kissaki @Krzmbrzl

Yes, Mumble can already be compiled with OpenSSL 1.1, but doing that for 1.3.1 would require us to update Qt as well, dropping support for Windows XP.

So you prefer delivering EOL (and by now extremly outdated) openssl software (which is crucial, because encryption is one of the most important parts), to support an OS that is EOL for 6(!) years?
I very much hope you reconsider that.

• And am I understading correctly that mumble could just use openssl v.1.1.1 (on windows builds) right now?
• And why is the static server build for linux not using the newest openssl?
  (and don't give me "support for super old distros" as an explanation, they can use their super outdated packages if they want to)

I want to add two arguments:

1. Windows XP is only used by ~1% of users (according to wikipedia )
2. You put all users who either communicate with users of old openssl versions or use servers that use old openssl versions at potential risk. And don't argue with the "dynamic library"-situation on distros, that is out of your hands, but this is directly in your hands.

[Kissaki]

None of what you want is trivial. We do what we reasonably can and what makes sense. We do not intend to support XP going forward. But we do value stability and not taking in unnecessary risks. The more we change the higher the risk something breaks. There’s good reasons for the split between minor and patch release being conceptually different.

EOL does not necessarily mean there are issues. Obviously it’s a pressing manner. But that does not mean it’s clearly the best choice to upgrade, or that it is simple or practical.

[davidebeatrici]

Also, we already dropped support for Windows XP in master (1.4).

Minor and patch releases should NEVER drop support for a platform, even if EOL.

Since OpenSSL is the most important component (in terms of security), we are going to build 1.3.1 with OpenSSL 1.0.2u: mumble-voip/mumble-releng#90.

[toby63]

@Kissaki:

None of what you want is trivial.

I would like to hear a bit of detail.
Thus the question I asked above:

And am I understading correctly that mumble could just use openssl v.1.1.1 (on windows builds) right now?

Because #3053 indicates to me, that it is not such a big problem.
But maybe I'm wrong, a little detail would be useful to better understand the situation.

• Is a new API the problem? (and there is no compatibility?)
  Does not seem to be the case, because why can I then build mumble against OpenSSL 1.1.1f?
• Is Windows the problem?
  Why then can't the static server build for linux be using openssl 1.1.1?
• And a change to OpenSSL 1.0.2u is not trivial?

We do what we reasonably can and what makes sense.

I saw that in the completely outdated openssl version.

But we do value stability and not taking in unnecessary risks.

Yeah very plausibel by using an openssl version from 2017 :facepalm: .

EOL does not necessarily mean there are issues.

Once again I wonder if thats also your approach at work. I hope not.

@davidebeatrici

Minor and patch releases should NEVER drop support for a platform, even if EOL.

So thats important to keep, but not updating the "most important" part. Very logical.
I don't understand that, version numbers are mostly arbitrary, so what does it matter, if its 1.3.1 or 1.4 or 2.5?
If it matter so much, rename things. The question here is, what is more important?
My answer is clear.

Since OpenSSL is the most important component (in terms of security), we are going to build 1.3.1 with OpenSSL 1.0.2u: mumble-voip/mumble-releng#90.

The best bad solution.

I guess the other depedencies are also outdated beyond belief?
You really should change that. :pray:

[davidebeatrici]

• Is a new API the problem? (and there is no compatibility?)
  Does not seem to be the case, because why can I then build mumble against OpenSSL 1.1.1f?

Right.

• Is Windows the problem?
  Why then can't the static server build for linux be using openssl 1.1.1?

We would like to keep our dependencies synced across all platforms.

• And a change to OpenSSL 1.0.2u is not trivial?

I'm pretty sure @Kissaki was referring to OpenSSL 1.1, which (as I said in my previous message), requires updating Qt.

And we're fortunate we didn't update to the latest LTS (5.12.5): https://bugreports.qt.io/browse/QTBUG-83450

So thats important to keep, but not updating the "most important" part. Very logical.
I don't understand that, version numbers are mostly arbitrary, so what does it matter, if its 1.3.1 or 1.4 or 2.5?
If it matter so much, rename things. The question here is, what is more important?
My answer is clear.

The best bad solution.

I guess the other depedencies are also outdated beyond belief?
You really should change that. :pray:

@Kissaki can probably explain our versioning scheme clearly.

As for security: it's more important than keeping support for an EOL platform. However:

1. OpenSSL 1.0.2 is not EOL.
2. OpenSSL 1.1 brings TLS 1.3 to the table, but TLS 1.2 is not considered insecure yet.
3. Maintaining support for Windows XP doesn't automatically make the software insecure.

Our mistake was using outdated dependencies, which happened because 1.3.0 was supposed to be released in 2017 rather than 2019...

Please inform us whether there are other dependencies we should update for 1.3.1.

[streaps]

• OpenSSL 1.0.2 is not EOL.

It is EOL:
https://www.openssl.org/blog/blog/2018/05/18/new-lts/
https://www.openssl.org/blog/blog/2019/11/07/3.0-update/
_Note that as previously announced OpenSSL 1.0.2 will be End Of Life at the end of this year [2019]. This means there will not be any further public updates or security fixes to the 1.0.2 branch from then. This gives another strong reason for existing 1.0.2 users to upgrade to 1.1.1 as soon as possible._

Releasing 1.3.0 with openssl 1.0.2 only a few month before 1.0.2's EOL was an oversight. I would consider it a bug, that should be fixed in 1.3.1.

• Maintaining support for Windows XP doesn't automatically make the software insecure.

What about another binary for legacy Windows (XP) and update the regular versions for Windows to openssl 1.1.1? Or build the 64-bit version with recent libs and the 32-bit with the older ones?

I do think that supporting older platform has some value, especially for an essential software that does work well on low-end devices. But Shipping an EOL openssl version to every Windows user is maybe too conservative?

[davidebeatrici]

It is EOL:
https://www.openssl.org/blog/blog/2018/05/18/new-lts/
https://www.openssl.org/blog/blog/2019/11/07/3.0-update/
_Note that as previously announced OpenSSL 1.0.2 will be End Of Life at the end of this year [2019]. This means there will not be any further public updates or security fixes to the 1.0.2 branch from then. This gives another strong reason for existing 1.0.2 users to upgrade to 1.1.1 as soon as possible._

Sorry, I missed that.

Thank you for pointing it out.

Releasing 1.3.0 with openssl 1.0.2 only a few month before 1.0.2's EOL was an oversight. I would consider it a bug, that should be fixed in 1.3.1.

Why do you think it's an oversight? OpenSSL 1.0.2 received the last security update in December.

Are there known vulnerabilities in 1.0.2u?

What about another binary for legacy Windows (XP) and update the regular versions for Windows to openssl 1.1.1? Or build the 64-bit version with recent libs and the 32-bit with the older ones?

I do think that supporting older platform has some value, especially for an essential software that does work well on low-end devices. But Shipping an EOL openssl version to every Windows user is maybe too conservative?

Right now we are using an older SDK and the v141_xp toolchain for the x86 build, in order to support Windows XP.

We could:

• Update OpenSSL and Qt to the latest version for the x64 build.
• Update OpenSSL and Qt to the latest version for the x64 and x86 builds and release a special build for XP.

However, it's something that requires quite a bit of time (e.g. updating our patches), which can potentially end up being wasted since we're working on a brand new build environment for 1.4.x: https://github.com/mumble-voip/mumble-releng-experimental

Honestly, I don't think it's worth it if there aren't known vulnerabilities in the latest OpenSSL 1.0.2 release.

We just want to move forward with 1.4.x at this point.

[toby63]

@davidebeatrici

Honestly, I don't think it's worth it if there aren't known vulnerabilities in the latest OpenSSL 1.0.2 release.

And who is going to check for vulnerabilities?
This is exactly the wrong approach, you don't use EOL-software because it is considered insecure by default.

We just want to move forward with 1.4.x at this point.

The deciding question is, when will 1.4.0 be released?
And please don't start a discussion again about how much there is to do etc.
The point here is not for you to hurry to 1.4.0, the point here is, you should fix the EOL software first, if 1.4.0 is still long away.
And yes even a release of 1.4.0 in a few months (means more than 3 months at maximum) is to far away imo.

Edit: Removed bold text at request of @bendem.

[bendem]

@toby63, please cut down on the bold and exclamation marks. Your communication is coming across as aggressive towards the team that's actually trying to fix things. Maintaining multi-platform software is hard, special cases are hard, and the team is already working on making it easier to update dependencies.

Unless you provide patches (and even if you provide patches), please don't act entitled toward a team that's already doing everything it can.

Pointing out solutions and outlining potential holdups is more productive than pointing out mistakes and attacking decisions. Everybody here is painfully aware that things are not ideal.

[toby63]

@bendem Even though it might look different, I only wanted to point out the most important points by using bold marks, because people read over things very fast.

But if it is seen as offensive, I will stop using that.

And to add one point:
I would be more silent, if there was less refractoriness in the answers of the team.
It should be standard not to use EOL software, that is what every software security expert will tell you.
Still there is pointless argumentation about how this is not bad and even about keeping it this way for longer time.

[toby63]

@davidebeatrici
Now that https://github.com/mumble-voip/mumble-releng/issues/90 is fixed, could you release a snapshot with newer openssl?

[davidebeatrici]

We were actually thinking about directly releasing 1.3.1, rather than 1.3.1 RC2.

[toby63]

@davidebeatrici
Ok,
I was just thinking about a development snapshot, that are listed on the bottom of the Downloads page.

[davidebeatrici]

The next snapshot will be based on 1.4.

[Krzmbrzl]

The update has been performed in the background (for our internal builders)

[tylerszabo]

@Krzmbrzl Did you update the builders for murmurd as well? In https://github.com/mumble-voip/mumble/releases/download/1.3.2/murmur-static_x86-1.3.2.tar.bz2 I'm still seeing the following:

SSL: OpenSSL version is 'OpenSSL 1.0.2k 26 Jan 2017'
...
Murmur 1.3.1 (1.3.2) running on X11: Linux 4.19.0-10-cloud-amd64: Booting servers

[Krzmbrzl]

No the environment for the static Linux server has not been updated.

The problem is that the only person who knew the internals of the static linux builder is no longer an active member of our team. Therefore updating it was impossible.

[tylerszabo]

In that case could this bug remain open as not all binaries are updated or the static build discontinued?

Shipping a build with outdated OpenSSL can cause downstream issues. I saw this in @PHLAK's Docker image; so there are downstream effects.

Also, do you have any context on the build options that produce the static build? Perhaps I could take a look and we could fix this with a PR.

[toby63]

Also, do you have any context on the build options that produce the static build?

They use an ancient linux distro with an ancient software for building the static stuff (I forgot the details, but the software doesn't seem to exist any longer etc.).

The Team considers alternatives for that (for Linux there are multiple things like Flatpak, Appimage, Snap etc.), but right now it seems they want to stick with this.

Thx for pointing this out, I didn't check and believed they had solved this.
Missleading.

[Kissaki]

In that case could this bug remain open as not all binaries are updated or the static build discontinued?

We should document it as a known issue/restriction. No need to keep it open if we can not or do not plan to fix it.

1.4 will have updated environments and stuff, including static server build. We are focusing on moving that forward.

[tylerszabo]

Noting the limitation in documentation sounds like a good solution - avoids lots of extra work; I was fiddling with the build indeed it's non-trivial.

The murmur packages in the Alpine and Fedora repositories and mumble-server packages in the default Debian and Ubuntu repositories are indeed using updated OpenSSL versions (it seems to be the shared lib, but I didn't verify beyond checking the reported version) so there are ample workarounds.

Thank you, both, for your quick response to a necro'd bug :)

[toby63]

@Kissaki

We should document it as a known issue/restriction.

You should simply take it down until there is a recent version.
You now talk about 1.4 for a year (or even more).

[Kissaki]

I would appreciate it if you would be less pissy, demanding, dismissive, disrespectful.

Did you run a security assessment? We have no indications of actual or even reported theoretical issues. Just because an upstream library fixes issues does not necessarily mean we use those things and that they apply to us.

Without an alternative removal is not an option.

We do what we reasonably can. So be concrete and practical, or stay silent.

[Kissaki]

This is our project, and our decision. You can voice disagreement, but your continued passive aggressive tone is very inappropriate and annoying. And it does nothing except hinder us from actually doing work, or having motivation to do so. You’re actively sabotaging what you are asking for - a solution.

[toby63]

https://security.archlinux.org/CVE-2020-1968

Your behaviour ist irresponsible.
You make people believe they use a secure and encrypted communication software.
But thats not the case, if you don't want to solve the situation, then at least be honest to people.

It's behaviour like yours that leads to the devastating IT security situation that we have now.

[toby63]

Without an alternative removal is not an option.

There are lots of alternatives, for linux it is no problem at all to simply use packages or copy the contents of packages etc.
And even for windows (because I have the feeling it isn't fixed there either, is it?) users can now use linux versions easily over wsl.

[Krzmbrzl]

The libraries are updated for all targets but the static server binary.
And no using packages is not an option since it's a static binary. All Linux packages are based on shared libraries though

[PHLAK]

The vulnerability linked isn't even practical to exploit. From the linked reference:

The attacker needs particular circumstances for the Raccoon attack to work. He needs to be close to the target server to perform high precision timing measurements. He needs the victim connection to use DH(E) and the server to reuse ephemeral keys. And finally, the attacker needs to observe the original connection. For a real attacker, this is a lot to ask for.