Nixpkgs: Docker + Kubernetes = no default bridge, commands fail

[jamescostian]

Describe the bug
I set up k8s and docker, and noticed that when running in a docker container, I could not connect to the internet. If I set --net=host for docker run or --network=host for docker build, then I can access the internet within containers. When I disabled k8s, accessing the internet from docker worked without any extra configuration. It is worth noting that the docker0 interface appears in ifconfig when k8s is disabled, but not when k8s is enabled, however, this appears to be on purpose.

To Reproduce
In configuration.nix you need to have docker and k8s:

  services.kubernetes = {
    roles = [ "master" "node" ];
    masterAddress = "localhost";
  };
  virtualisation.docker.enable = true;

Then nixos-rebuild switch and run sudo docker run -it --rm debian /usr/bin/ping 1.1.1.1 and you will see connect: Network is unreachable

Expected behavior
I expect kubernetes and docker to both work when they are both enabled

Additional context
I tried export DOCKER_OPTS="--net=host" but it didn't fix things. I also looked through all the relevant GitHub issues and the online manual, none helped. Google wasn't helpful either. The best I could find was a slide shown in a youtube video, which seemed to acknowledge the lack of a bridge interface for docker. Its config was outdated, so I tried updating it like so:

  services.kubernetes = {
    roles = [ "master" "node" ];
    masterAddress = "localhost";
  };
  networking.bridges = {
    cbr0.interfaces = [];
  };
  networking.interfaces = {
    cbr0.ipv4.addresses = [ {
      address = "192.168.86.64";
      prefixLength = 24;
    } ];
  };
  virtualisation.docker.enable = true;
  virtualisation.docker.extraOptions = "--iptables=false --ip-masq=false -b cbr0";

Unfortunately, this did not fix docker (in fact, it broke my normal internet connection until I rolled back)

Metadata

 - system: `"x86_64-linux"`
 - host os: `Linux 4.19.78, NixOS, 19.03.173582.df7e351af91 (Koi)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.2.2`
 - channels(james): `""`
 - channels(root): `"nixos-19.03.173582.df7e351af91"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[jamescostian]

Just read the options part of the manual more closely, and fixed one part of my configuration:

   networking.bridges = {
-    cbr0.interfaces = [];
+    cbr0 = {
+      interfaces = [ "wlo0" ];
+    };
   };

That allows my normal internet access to work, but prevents docker and kubernetes from starting. Here are my journalctl -xe logs.

[stale[bot]]

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the
   related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
   irc.freenode.net.

[schneefux]

still relevant

[mikroskeem]

Any updates on this?

Using --net=host is not a proper solution at all, the root cause should be figured out instead. Enabling k8s service breaks networking completely inside Docker and containers set up by k8s can't access the internet either.
_Edit:_ Turns out flannel broke before, making k8s containers' network nonfunctional.

I've tried with firewall on/off, ensured that net.ipv4.ip_forward and net.ipv4.conf.<intf>.forwarding are set to 1.

This happens in 20.03

 - system: `"x86_64-linux"`
 - host os: `Linux 5.4.47-xanmod1, NixOS, 20.03.2310.fb6c3a6831c (Markhor)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.6`
 - channels(root): `"nixos-20.03.2310.fb6c3a6831c, nixpkgs-unstable-20.09pre228384.c27e54de99d"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

... and unstable

 - system: `"x86_64-linux"`
 - host os: `Linux 5.4.47, NixOS, 20.09pre231796.22a81aa5fc1 (Nightingale)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.6`
 - channels(root): `"nixos-20.09pre231796.22a81aa5fc1"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`