Nixpkgs: Cannot install Calico on Kubernetes

[kliu128]

Issue description

Installing Calico for policy only, to get NetworkPolicy support, gives no error but doesn't work.

I believe this is because kubelet doesn't look in /etc/cni/net.d by default, which is where Calico puts its config files, and it also clears /opt/cni/bin.

The better solution here is honestly probably to package Calico; I saw that #19023 had an implementation, but it seems to have gotten lost in a force-push.

Steps to reproduce

Install calico following the above instructions on a flannel-enabled cluster, with flannel enabled in NixOS.

Technical details

• system: "x86_64-linux"
• host os: Linux 4.19.12, NixOS, 19.03.git.55dfae9 (Koi)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.1.3
• channels(root): ""
• nixpkgs: /etc/nixos/nixpkgs

[q3k]

We've managed to hack around it by:

• mounting /opt/cni/bin into the calico-node daemonset so that it can write its plugins there
• overriding systemd.services.kubelet.preStart to remove the current nix CNI plugin installation

Once Calico DaemonSets are scheduled on nodes, those nodes will then correctly start pods that want to use Calico.

But agreed, we'd be better off packaging Calico in Nix.

[kliu128]

Nice workaround! Just tried it and seems to work. The best solution is packaging, but maybe it would also be nice if there was a "use external CNI" option to do this automatically.

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[duckfullstop]

Still important, IMO. Having a not-completely-insane and well documented (for morons like me) way of initialising a cluster with Calico would go a long way to making the k8s derivations a lot more usable for more advanced scenarios, like IPv6 / IPv6DualStack.

(This comment brought to you by smashing my head against a cluster for about 4 weeks total at this point trying to get DualStack to work properly)