Nixpkgs: Implement NixOS container networking with networkd

[arianvp]

We are in the process to moving to networkd based networking for 20.03.

nixos-container module neither uses scripted networking nor systemd-based networking,
but its own set of scripts. We should move nixos-container to use networkd for setting up the network.

Currently if you enable networking.useNetworkd = true; , nixos-container will stop working because the default rules that networkd ships sets up DHCP on the ve-* and vz-* interfaces as it assumes these are created by systemd-nspawn and that's the documented behaviour for the tool (See https://github.com/NixOS/systemd/tree/nixos-v243/network)

the matches look like this:

# 80-container-ve.network
[Match]
Name=ve-*
Driver=veth
[Network]
# Default to using a /24 prefix, giving up to 253 addresses per virtual network.
Address=0.0.0.0/24
LinkLocalAddressing=yes
DHCPServer=yes
IPMasquerade=yes
LLDP=yes
EmitLLDP=customer-bridge

# 80-container-vz.network
[Match]
Name=vz-*
Driver=bridge
[Network]
# Default to using a /28 prefix, giving up to 13 addresses per container.
Address=0.0.0.0/28
LinkLocalAddressing=yes
DHCPServer=yes
IPMasquerade=yes
LLDP=yes
EmitLLDP=customer-bridge

We should come up with our own systemd.network files that implement all the features that nixos-container supports.

However, I would also like to have vanilla systemd-nspawn work as documented in the systemd manpages.

This means that the .network rules that nixos-container should generate should be:

1) more specific: e.g. Match=ve-mycontainer vs Match=ve-*
2) have a lower number than 80- such that they're executed before the ones shipped with systemd

Implementing this is a blocker for switching to useNetworkd=true by default.

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

cc @fpletz @flokli @andir @Mic92

[arianvp]

I'm not sure why this was labelled kind: bug. could someone remove that for me? :)

[ck3d]

If host and container uses networkd the network is configured correctly by networkd.
The the test nixos/tests/systemd-machinectl.nix tests that configuration.
I think there is no need for a special set of configuration files.

[arianvp]

I'm pretty sure that we need to take a more careful look especially since networkd takes ownership of the veth-pairs nixos-container makes by default. Weird things might happen that we currently don't test for. E.g. DHCP will be on by default and networkd is handing out IPs on the container interface, eventhough nixos-container sets useDHCP = false;

[arianvp]

At best, all the code that calls ip link up and ip link down can just go once we've switched to networkd

[Ma27]

@elseym discussed some ideas in the past and started an initial attempt at NixCon. We'll probably be able to create a proof of concept on the networkd-sprint next weekend :)

[bqv]

Is there a workaround for this?

[Ma27]

In #84608 is an example how to use the nspawn module itself for "containers". You could just configure your networkd inside the container/machine-config.

[Ma27]

FYI I'm actively working on building a useful proposal for networkd-based nspawn-units that can be configured using NixOS. I'll do my best to publish a first draft to discuss during the next weeks :)

[Ma27]

I'm very sorry for the long time this issue was stale. Since I had a brief discussion with
@fpletz last weekend I'd like to share the results with you to gather even further feedback.
At the end of this comment I'll post the next steps I've planned on this.

__Note:__ as this comment became way longer and more detailed than I anticipated, I'm wondering
if the whole process of replacing the current container implementation should go through an RFC.

Summary

• We'd like to get rid of the containers.* module and the nixos-container package in the
  long run, however we know that it's very hard to get rid of it, so we'd wait at least 3
  NixOS releases after those proposed changes landed in a stable release.

  The deprecation is from our PoV necessary for the following reasons:

  • nixos-container is mostly a wrapper around
    machinectl(1), when
    using .nspawn-units for containers, we can use machinectl for most of the tasks

  • nspawn-machines usually expect to get configured using networkd. We work around this
    by configuring the host-side network interfaces after the container has booted in an
    ExecStartPost= hook. Because of that, containers don't have any uplink at boot-time
    which can lead to unbootable containers (because of timed-out services) in the worst case.

• We want to introduce a layer on top of systemd.nspawn (which is a simple module to generate
  nspawn files) that can create a machine from a NixOS config and optionally configures some basic
  networking.

Networking

• Automatically assigned addresses:

  • By default, a container should get a local address from 192.168/16 (as it is currently
    the case in upstream systemd).

  • Additionally an IPv6 ULA-prefix for containers should be generated
    to assign an IPv6 address (or prefix) to a container. This isn't implemented yet in upstream
    systemd which is why I started writing a patch a while ago (which needs a lot of work though
    before getting usable).

  • When using this approach, a container can optionally get internet-access using NAT.

• Static network configuration:

  • Optionally it should be possible to statically declare the container's network configuration
    to e.g. assign public addresses to a container.

  • We want to keep this pretty thin however and make it rather simple for administrators
    to customize the network configuration for their containers.

Additional Features

• nixos-container can be replaced by machinectl and a new tool which generates a rootfs
  and places it into /var/lib/machines using a NixOS configuration for a container.

• Only mount store-paths into the container that are actually needed (I've written a draft for this,
  I mostly have to figure out the performance implications of creating lots of bind-mounts).

• switch-to-configuration should be able to decide whether to restart a container or whether to
  only activate the new configuration when deploying a host.

Migration

As mentioned above, we'd like to replace the current container feature with a very long transition
period.

I'm not sure yet how a migraiton-path should look in detail, but since we're not switchting
technologies, it should be possible to write down a detailed migration recipe (in theory at least).

Roadmap

Please note though that I'll have a few exams at the end of July and in August, so this won't
make it into 20.09 (I'd target 21.03 for now) to get extensive community feedback on this.

I'll try to publish a usable draft until the end of August/September including IPv6 support.
Then we'd have a basic proof-of-concept to work with.

I planned further steps on this in this year's fall.

[arianvp]

This sounds like an excellent plan. I wonder if we should turn it into an
RFC to collect feedback and assess fallout?

On Mon, Jul 13, 2020, 20:10 Maximilian Bosch [e-mail]
wrote:

I'm very sorry for the long time this issue was stale. Since I had a brief
discussion with
@fpletz https://github.com/fpletz last weekend I'd like to share the
results with you to gather even further feedback.
At the end of this comment I'll post the next steps I've planned on this.

Note: as this comment became way longer and more detailed than I
anticipated, I'm wondering
if the whole process of replacing the current container implementation
should go through an RFC.
Summary

-

We'd like to get rid of the containers.* module and the nixos-container
package in the
long run, however we know that it's very hard to get rid of it, so
we'd wait at least 3
NixOS releases after those proposed changes landed in a stable release.

The deprecation is from our PoV necessary for the following reasons:
-

  nixos-container is mostly a wrapper around
  machinectl(1)
  <https://www.freedesktop.org/software/systemd/man/machinectl.html>,
  when
  using .nspawn-units for containers, we can use machinectl for most
  of the tasks
  -

  nspawn-machines usually expect to get configured using networkd. We
  work around this
  by configuring the host-side network interfaces *after* the
  container has booted in an
  ExecStartPost= hook. Because of that, containers don't have any
  uplink at boot-time
  which can lead to unbootable containers (because of timed-out
  services) in the worst case.
  -

We want to introduce a layer on top of systemd.nspawn (which is a
simple module to generate
nspawn files) that can create a machine from a NixOS config and
optionally configures some basic
networking.

Networking

-

Automatically assigned addresses:
-

  By default, a container should get a local address from 192.168/16
  (as it is currently
  the case in upstream systemd).
  -

  Additionally an IPv6 ULA-prefix for containers should be generated
  to assign an IPv6 address (or prefix) to a container. This isn't
  implemented yet in upstream
  systemd which is why I started writing a patch a while ago (which
  needs a lot of work though
  before getting usable).
  -

  When using this approach, a container can optionally get
  internet-access using NAT.
  -

Static network configuration:
-

  Optionally it should be possible to statically declare the
  container's network configuration
  to e.g. assign public addresses to a container.
  -

  We want to keep this pretty thin however and make it rather simple
  for administrators
  to customize the network configuration for their containers.

Additional Features

-

nixos-container can be replaced by machinectl and a new tool which
generates a rootfs
and places it into /var/lib/machines using a NixOS configuration for a
container.
-

Only mount store-paths into the container that are actually needed
(I've written a draft for this,
I mostly have to figure out the performance implications of creating
lots of bind-mounts).
-

switch-to-configuration should be able to decide whether to restart a
container or whether to
only activate the new configuration when deploying a host.

Migration

As mentioned above, we'd like to replace the current container feature
with a very long transition
period.

I'm not sure yet how a migraiton-path should look in detail, but since
we're not switchting
technologies, it should be possible to write down a detailed migration
recipe (in theory at least).
Roadmap

Please note though that I'll have a few exams at the end of July and in
August, so this won't
make it into 20.09 (I'd target 21.03 for now) to get extensive community
feedback on this.

I'll try to publish a usable draft until the end of August/September
including IPv6 support.
Then we'd have a basic proof-of-concept to work with.

I planned further steps on this in this year's fall.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/NixOS/nixpkgs/issues/69414#issuecomment-657711139,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAEZNI2WSTMYXKRDZHVOXZLR3NE3FANCNFSM4I2K5BSQ
.

[bqv]

Thanks, @Ma27 !

[Ma27]

RFC to collect feedback and assess fallout

Yeah, I had a similar feeling when I realized that my comment slightly escalated :sweat_smile:

[arianvp]

When using this approach, a container can optionally get internet-access using NAT.

Only for the ipv4 part yet? the ipv6 just needs routes.

Note that we can use networkd to configure both

[Ma27]

Only for the ipv4 part yet? the ipv6 just needs routes.

Are you talking about public prefixes or ULA prefixes? Not sure if I'm misunderstanding you, but when using an ULA address for a container, we'd need something like v6 NAT or NPTv6, right?

Btw, I'll try to write a draft for an RFC at one of the next weekends :)

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/issues-using-nixos-container-to-set-up-an-etcd-cluster/8438/2