Nixpkgs: Tinc -netdev systemd service can fail with ioctl(TUNSETIFF): Device or resource busy

[nh2]

The tinc nixos service currently has these lines:

    networking.interfaces = flip mapAttrs' cfg.networks (network: data: nameValuePair
      ("tinc.${network}")
      ({
        virtual = true;
        virtualType = "${data.interfaceType}";
      })
    );

I have found that in rare cases this can lead to the error ioctl(TUNSETIFF): Device or resource busyon startup

In particular, the *-netdev systemd service created by createTunDevice in network-interfaces-scripted.nix can fail; it runs this code:

            script = ''
              ip tuntap add dev "${i.name}" \
              ${optionalString (i.virtualType != null) "mode ${i.virtualType}"} \
              user "${i.virtualOwner}"
            '';

I suspect that this is because it races with the tinc systemd service's tincd startup, which itself creates the tun device, so if tinc creates the interface first and starts sending data on it, the *-netdev service can't do its ip tuntap/mode setting on the running device.

Note I'm using tinc 1.1 pre here (this nixpkgs package).

As per the Tinc 1.1 manual (just before section 9.2 Routes), tinc is OK with the user creating the interface first (with ip tuntap add dev).

However, I think the problem here is that our nixos service's systemd configuration doesn't ensure it runs after that -netdev script has finished.

Also, it is not clear to me why networking.interfaces was used in the first place. As the manual says:

On Linux, it is possible to create a persistent tun/tap interface which will continue to exist even
if tinc quit, although this is normally not required. It can be useful to set up a tun/tap interface
owned by a non-root user, so tinc can be started without needing any root privileges at all.

The typical motivation here is to have tinc run without root priviliges. But our nixos service seem to need that. It adds a tinc user and uses tincd -U theuser to drop privileges to that user after it has created the VPN tun network device as root.

So I don't think the networking.interfaces is needed, and it seems I can run it all fine without that entry (so far; the error only happens on some of my nixops deploys). @wkennington, could you let me know why you added it in the original commit?

---

CC the tinc service commiters @ericsagnes @fadenb @joachifm @tg-x

[Mic92]

You can probably just drop networking.interfaces from the tinc service.

[florianjacob]

For me, this still happens on NixOS 18.03 and tinc 1.0.

[Mic92]

backport: cde886e7c62

[Ekleog]

@Mic92 Upon a reboot of 18.09 machines tinc no longer links at startup here, so this is breakage entering 18.09 :/ unless it's the bump to 1.1pre17, but the failure mode makes me think it's a bug in the updated nixos module.

I'm getting tincd[1456]: Can't write to Linux tun/tap device (tun mode) /dev/net/tun: Input/output error and the route to the interface not being set.

Currently investigating further.

[Mic92]

@Ekleog we also notice this yesterday. One needs to use ip link set $INTERFACE up in tinc-up as shown here https://wiki.archlinux.org/index.php/Tinc#Configuration_of_alpha
Sorry for breaking configuration I did not notice that because networkd is doing this automatically.
In the long run this is the better approach because currently it is very racy and delays the boot significantly.
It would be great if we can add this to the release notes as well.

[Ekleog]

@Mic92 OK so I think this is due to using networking.interfaces to configure the tinc address: tinc creates the interface long after sys-subsystem-net-devices-tinc.${name}.device has failed.

As a consequence, somehow the device never gets up (as shown by ip a) and tinc fails to actually exchange data, even though tinc shows as correctly linked everywhere.

I suggest reverting this change on 18.09 and documenting as a backwards-compatibility-breaking change for 19.03 that using tinc with static addresses set via networking.interfaces now requires a manual setting for the virtual and virtualType parameters.

I can confirm that adding these two parameters to my local configuration fixes tinc here.

[Mic92]

I reverted back to the old behavior: ea30555a03dd7a506e174440583ebbaf255915ff

[Mic92]

I need a release note entry or a better fix for 19.03.

[Ekleog]

@Mic92 Do you have an idea of what to put as release note entry? AFAIU it's the last thing that prevents this issue from being closed, and ideally it'd be made before a 19.03 cutoff, but I'm not sure what exactly can be written.

Maybe something along the lines of “The tinc module now waits until tinc starts up to create its interface, you may need to set virtual and virtualType if you relied on it being available at boot”?

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[Mic92]

It's too late for release notes now... This is part of nixos for quite a while