Nixpkgs: Flatpak SSL Certificate problem

[piotr-szegda]

Describe the bug
Flatpak app using SSL don't work on NixOS 20.03. Continue of discussion from https://github.com/NixOS/nixpkgs/pull/80698

Example error:

Discord:
[WebContents] did-fail-load -202 ERR_CERT_AUTHORITY_INVALID retry in 20000 ms
Steam:
[0312/163655.227845:ERROR:ssl_client_socket_impl.cc(935)] handshake failed; returned -1, SSL error code 1, net_error -202
[0312/163655.700394:ERROR:cert_verify_proc_nss.cc(1011)] CERT_PKIXVerifyCert for steam-chat.com failed err=-8172
[0312/163655.700678:ERROR:ssl_client_socket_impl.cc(935)] handshake failed; returned -1, SSL error code 1, net_error -202

Maybe related https://github.com/NixOS/nixpkgs/issues/81284?
To Reproduce
Steps to reproduce the behavior:
1.Enable Flatpak.

1. Install Steam.
2. Run in.

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

• system: "x86_64-linux"
• host os: Linux 5.5.8, NixOS, 20.03beta529.da92e0566d1 (Markhor)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3.2
• channels(root): "nixos-20.03beta529.da92e0566d1"
• channels(piotrek): "home-manager, nixos-20.03beta529.da92e0566d1, nixos-unstable-20.09pre215947.82b54d49066"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

[xvello]

I can reproduce on unstable too as of today

[piotr-szegda]

@xvello: Also on KDE? Maybe it is correlated.

[buckley310]

I am experiencing SSL Cert errors in the flathub version of steam on NixOS 20.03 + Gnome.

[shazow]

FWIW I'm running on i3 without a desktop manager and able to reproduce this on latest unstable (nixos-20.09pre216896.0729b8c55e0, flatpak 1.6.2).

Using Discord:

...
[Modules] Checking for host updates.
Error downloading with electron net: net::ERR_CERT_AUTHORITY_INVALID
Falling back to node net library..
[Modules] Host is up to date.
[Modules] Checking for module updates at https://discordapp.com/api/modules/stable/versions.json
Error downloading with electron net: net::ERR_ABORTED
Falling back to node net library..
[Modules] No module updates available.
[WebContents] did-fail-load -202 ERR_CERT_AUTHORITY_INVALID retry in 1000 ms

as well as Steam:

...
[0315/152328.895335:ERROR:nss_util.cc(283)] After loading Root Certs, loaded==false: /home/shazow/.pki/nssdb/libnssckbi.so: cannot open shared object file: No such file or directory
BuildCompleteAppOverviewChange: 385
RegisterForAppOverview 1: 20ms
RegisterForAppOverview 2: 21ms
[0315/152329.129678:ERROR:cert_verify_proc_nss.cc(1011)] CERT_PKIXVerifyCert for steam-chat.com failed err=-8172
[0315/152329.129963:ERROR:ssl_client_socket_impl.cc(935)] handshake failed; returned -1, SSL error code 1, net_error -202
[0315/152329.163350:ERROR:ssl_client_socket_impl.cc(935)] handshake failed; returned -1, SSL error code 1, net_error -202
[0315/152329.191193:ERROR:ssl_client_socket_impl.cc(935)] handshake failed; returned -1, SSL error code 1, net_error -202
[0315/152329.193567:ERROR:frame_sink_video_capturer_impl.cc(206)] Invalid resolutions constraints: 0x0 must not be greater than 0x0; and also within media::limits.
[0315/152329.287605:ERROR:cert_verify_proc_nss.cc(1011)] CERT_PKIXVerifyCert for store.steampowered.com failed err=-8179
[0315/152329.287919:ERROR:ssl_client_socket_impl.cc(935)] handshake failed; returned -1, SSL error code 1, net_error -202
[0315/152329.993134:ERROR:cert_verify_proc_nss.cc(1011)] CERT_PKIXVerifyCert for steamcdn-a.akamaihd.net failed err=-8179
[0315/152329.994069:ERROR:ssl_client_socket_impl.cc(935)] handshake failed; returned -1, SSL error code 1, net_error -202
[0315/152330.068502:ERROR:ssl_client_socket_impl.cc(935)] handshake failed; returned -1, SSL error code 1, net_error -202
[0315/152336.699474:INFO:crash_reporting.cc(270)] Crash reporting enabled for process: renderer
[0315/152336.764430:ERROR:ssl_client_socket_impl.cc(935)] handshake failed; returned -1, SSL error code 1, net_error -20

My nixfiles are here if that's useful: https://github.com/shazow/nixfiles

[shazow]

As far as I can tell from other distros, p11-kit 0.23.20 should fix the issue which nixpkgs already has. Could it be related to how nss is packaged in nixpkgs? (As hinted by the Steam output, though I'm not sure if that error is new.)

An older bug but maybe related: https://github.com/p11-glue/p11-kit/issues/278

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/flatpak-error-the-name-org-freedesktop-accounts-was-not-provided-by-any-service-file/6000/22

[shazow]

In case it's any help, confirmed that the same issue still persists in e8701fe486266edc600abb1f7399a48779cab995.

Also this is what I'm using as a workaround to pin my NixOS to an earlier version that still has a working flatpak (running flatpak v1.4.x), a21c2fa3ea2b88e698db6fc151d9c7259ae14d96:

$ nix-channel --add https://github.com/NixOS/nixpkgs/archive/a21c2fa3ea2b88e698db6fc151d9c7259ae14d96.tar.gz nixos
$ nix-channel --update
$ nixos-rebuild switch

[piotr-szegda]

There is also this way: https://github.com/NixOS/nixpkgs/issues/41212. Anyone tried? I am not sure how to do this correctly for flatpak.

[jtojnar]

Do you have a simple sequence of commands I could try to reproduce this with?

I used nixos-rebuild build-vm -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/d96bd3394b734487d1c3bfbac0e8f17465e03afe.tar.gz -I nixos-config=../flatpak-xterm.nix && env QEMU_NET_OPTS="hostfwd=tcp::2222-:22" ./result/bin/run-*-vm with the following config to get as bare bones environment as possible:

{ pkgs, config, ... }: {
  environment.systemPackages = with pkgs; [
    strace
    gdb
  ];

  services.xserver = {
    enable = true;
    layout = "cz";
    xkbVariant = "qwerty";

    desktopManager.xterm.enable = true;
    displayManager.lightdm.enable = true;
  };


  xdg.portal.enable = true;
  services.openssh.enable = true;
  services.flatpak.enable = true;

  users.extraUsers.jtojnar = {
    isNormalUser = true;
    uid = 1000;
    extraGroups = [ "wheel" "networkmanager" ];
    password = "";
    openssh.authorizedKeys.keys = ["ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYbOlZydfRRCGCT08wdtPcpfSrgxMc6weDx3NcWrnMpVgxnMs3HozzkaS/hbcZUocn7XbCOyaxEd1O8Fuaw4JXpUBcMetpPXkQC+bZHQ3YsZZyzVgCXFPRF88QQj0nR7YVE1AeAifjk3TCODstTxit868V1639/TVIi5y5fC0/VbYG2Lt4AadNH67bRv8YiO3iTsHQoZPKD1nxA7yANHCuw38bGTHRhsxeVD+72ThbsYSZeA9dBrzACpEdnwyXclaoyIOnKdN224tu4+4ytgH/vH/uoUfL8SmzzIDvwZ4Ba2yHhZHs5iwsVjTvLe7jjE6I1u8qY7X8ofnanfNcsmz/ jtojnar@kaiser"];
  };

  environment.enableDebugInfo = true;
  virtualisation.memorySize = 2048;
}

And inside the VM:

flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
flatpak install flathub com.valvesoftware.Steam
flatpak run com.valvesoftware.Steam

But Steam started downloading just fine:

[shazow]

@jtojnar For me, Steam doesn't start failing until after the store/library views have opened. Not sure if the rest of the flow uses a different TLS mechanism (maybe cert pinning?).

Other flatpak apps like Discord seem to fail earlier. Maybe try that as a more convenient test case?

[jtojnar]

Hmm, I can actually confirm it in GNOME too. The initial update download works but then it is stuck with ERR_CERT_AUTHORITY_INVALID.

[piotr-szegda]

I have workaround of this problem (NOT A SOLUTION). It's pretty simple, in log there is info about missing file "~/.pki/nssdb/libnssckbi.so". Copying this file to real home directory doesn't change anything (probably because it is error from inside of flatpak sandbox) so I copied this file to ~/.var/app/com.valvesoftware.Steam/.pki/nssdb/ and it's working.
I used version from steam runtime but maybe nix version also work. In my case I used command:
cp ~/.var/app/com.valvesoftware.Steam/.local/share/Steam/ubuntu12_32/steam-runtime/usr/lib/x86_64-linux-gnu/nss/libnssckbi.so ~/.var/app/com.valvesoftware.Steam/.pki/nssdb/
It also may work for other apps.
Maybe related or usefule:
https://wiki.gentoo.org/wiki/Certificates (NSS)
https://github.com/NixOS/nixpkgs/issues/8247

EDIT: Version from Nix also work.

[shazow]

Ah interesting, thanks for sharing that @piotr-szegda. Sounds like it is indeed just a packaging issue (rather than a legit bug in flatpak). That's comforting.

[jtojnar]

I tried the following overlay:

(self: super: {
  flatpak = super.flatpak.override {
    p11-kit = super.p11-kit.overrideAttrs (attrs: rec {
      version = "0.23.18.1";
      src = super.fetchFromGitHub {
        owner = "p11-glue";
        repo = attrs.pname;
        rev = version;
        sha256 = "0l4ndipcbikblfjz729zkscvni5lzdzvbvsar2538pylw556rx73";
      };
    });
  };
})

but the simple reproduction still fails:

https://github.com/p11-glue/p11-kit/issues/275#issue-556837079

Did it work for you with flatpak 1.4?

[shazow]

Yes, I'm currently using a pinned nix-channel that still has flatpak 1.4, everything works great. Let me know if I can help test anything.

[jtojnar]

One thing that would be nice to find out if it the issue was caused by flatpak update, or p11-kit update.

Apparently, Flatpak uses p11-kit to make the certificates from host available in the containers but it does not look like the code was touched in Flatpak 1.6 as far as I could tell. That would point to p11-kit issue but when I tried downgrading it for Flatpak using overlay, it did not seem to fix it – maybe some global service depends on it as well? What p11-kit version do you have on your system?

[shazow]

I have p11-kit-0.23.18.1.

My nixos nix-channel is pinned to https://github.com/NixOS/nixpkgs/archive/a21c2fa3ea2b88e698db6fc151d9c7259ae14d96.tar.gz.

[jtojnar]

It would be nice to see if it happens between b57eabee9fdb1674282650e8a9a0793bf78914d5 and 8bcc32a684303fd32bf99d90e2e839b5fc18d29f.

[shazow]

Using nixos https://github.com/NixOS/nixpkgs/archive/b57eabee9fdb1674282650e8a9a0793bf78914d5.tar.gz, which runs flatpak v1.6.3, it still fails on Steam and Discord. Same errors.

Let me know if you want me to check any other specific commit. :)

[piotr-szegda]

Flatpaked Firefox work. Mostly it's problem with Chromium base app.

[piotr-szegda]

This is strace from my start of discord : https://gist.github.com/piotr-szegda/5d21a53d0bf185e3a18095467c7ee718

Maybe we can compare it to working one?

[jtojnar]

Running the steps to reproduce from https://github.com/p11-glue/p11-kit/issues/275#issue-556837079 and stracing the pkcs11 helper using strace -f -p $(pgrep -f pkcs11-flatpak) might be cleaner.

[piotr-szegda]

When I try run
strace -f flatpak run --command=bash --share=network runtime/org.freedesktop.Platform/x86_64/19.08
It's stuck on loop of:
[pid 11591] fcntl(4, F_SETFD, FD_CLOEXEC) = 0
[pid 11591] fcntl(5, F_SETFD, FD_CLOEXEC) = 0
[pid 11591] fcntl(6, F_SETFD, FD_CLOEXEC) = 0
[pid 11591] fcntl(7, F_SETFD, FD_CLOEXEC) = 0
[pid 11591] fcntl(8, F_SETFD, FD_CLOEXEC) = 0
[pid 11591] fcntl(9, F_SETFD, FD_CLOEXEC) = 0
...
A bit more of log:

[pid 11233] clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLDstrace: Process 11234 attached
, child_tidptr=0x7f3c6975bbd0) = 11234
[pid 11234] set_robust_list(0x7f3c6975bbe0, 24 [pid 11233] write(36, "\342+\0\0", 4 [pid 11234] <... set_robust_list resumed>) = 0
[pid 11233] <... write resumed>) = 4
[pid 11233] close(36) = 0
[pid 11233] exit_group(0 [pid 11234] close(36 [pid 11233] <... exit_group resumed>) = ?
[pid 11234] <... close resumed>) = 0
[pid 11234] fcntl(34, F_SETFD, FD_CLOEXEC) = 0
[pid 11234] openat(AT_FDCWD, "/dev/null", O_RDONLY) = 33
[pid 11234] dup2(33, 0) = 0
[pid 11234] close(33) = 0
[pid 11234] prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=10241024, rlim_max=10241024}) = 0
[pid 11234] fcntl(3, F_SETFD, FD_CLOEXEC) = 0
[pid 11234] fcntl(4, F_SETFD, FD_CLOEXEC) = 0
[pid 11233] +++ exited with 0 +++
[pid 11227] <... wait4 resumed>[{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 11233
[pid 11234] fcntl(5, F_SETFD, FD_CLOEXEC [pid 11227] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=11233, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
[pid 11234] <... fcntl resumed>) = 0
[pid 11227] read(33, [pid 11234] fcntl(6, F_SETFD, FD_CLOEXEC) = 0

[jtojnar]

First thing to check would be reverting the p11-kit update to see if the issue was actually caused by it:

git checkout nixos-unstable
git revert bbbb49c
git revert 6fbba7

You can build a VM using nixos-rebuild build-vm -I nixpkgs=$HOME/Projects/nixpkgs -I nixos-config=flatpak-xterm.nix but it requires to rebuild a lot:

{ pkgs, config, ... }: {
  environment.systemPackages = with pkgs; [
    strace
    gdb
  ];

  services.xserver = {
    enable = true;
    layout = "cz";
    xkbVariant = "qwerty";

    desktopManager.xterm.enable = true;
    displayManager.lightdm.enable = true;
  };


  xdg.portal.enable = true;
  services.openssh.enable = true;

  services.flatpak.enable = true;

  users.extraUsers.jtojnar = {
    isNormalUser = true;
    uid = 1000;
    extraGroups = [ "wheel" "networkmanager" ];
    password = "";
    openssh.authorizedKeys.keys = ["ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYbOlZydfRRCGCT08wdtPcpfSrgxMc6weDx3NcWrnMpVgxnMs3HozzkaS/hbcZUocn7XbCOyaxEd1O8Fuaw4JXpUBcMetpPXkQC+bZHQ3YsZZyzVgCXFPRF88QQj0nR7YVE1AeAifjk3TCODstTxit868V1639/TVIi5y5fC0/VbYG2Lt4AadNH67bRv8YiO3iTsHQoZPKD1nxA7yANHCuw38bGTHRhsxeVD+72ThbsYSZeA9dBrzACpEdnwyXclaoyIOnKdN224tu4+4ytgH/vH/uoUfL8SmzzIDvwZ4Ba2yHhZHs5iwsVjTvLe7jjE6I1u8qY7X8ofnanfNcsmz/ jtojnar@kaiser"];
  };

  environment.enableDebugInfo = true;

  nixpkgs.config.allowUnfree = true;

  virtualisation.memorySize = 2048;
  virtualisation.diskSize = 8192;
}

[piotr-szegda]

Even after reverting commits it's still not work in VM. I am out of idea. It is complicated problem on normal Linux distro and extremely complicated on NixOS.

[mat8913]

Problem is caused by https://github.com/NixOS/nixpkgs/blob/fcf547d0e23b4be19fdd877a1e31dff0d0b0498d/pkgs/development/libraries/p11-kit/default.nix#L29.

Changing it to "--with-trust-paths=/etc/ssl/certs/ca-certificates.crt" works for me. Alternatively, you can just disable flatpak's p11 server by reverting 02213fdff268d2296e7e495954e8decaa8b2825e.

[piotr-szegda]

How to easy and fast test this solution on current system? Fork of nixpkgs and rebuild with -I nixpkgs= take a lot of time. Overlays will work?
I can confirm problem is solved in VM by adding "--with-trust-paths=/etc/ssl/certs/ca-certificates.crt".
Also would you kindly take a look at that @worldofpeace and @hedning.

[worldofpeace]

@piotr-szegda Sure, wow.

[worldofpeace]

As a rule, most issues in NixOS can be packaging issues :grin:

[mat8913]

@piotr-szegda Try adding this to your nixos config:

  nixpkgs.overlays = [
    (self: super: {
      flatpak = super.flatpak.override {
        p11-kit = self.p11-kit.overrideAttrs (attrs: {
          configureFlags = [
            "--sysconfdir=/etc"
            "--localstatedir=/var"
            "--with-trust-paths=/etc/ssl/certs/ca-certificates.crt"
          ];
        });
      };
    })
  ];