Nomad: CNI plugin version 0.7.x support

[thanhy]

For reporting security vulnerabilities please refer to the website.

If you have a question, prepend your issue with [question] or preferably use the nomad mailing list.

If filing a bug please include the following:

Nomad version

Output from nomad version
Nomad v0.10.0 (25ee121d951939504376c70bf8d7950c1ddb6a82)

Operating system and Environment details

Photon OS 3

Issue

We’re using Photon OS 3 which currently only has support for CNI plugin version 0.7.5.3. Nomad does not seem to work with that CNI plugin version.
Below are the nomad error log messages when I deploy the job with bridge network using CNI plugin version 0.7.5.3.

failed to setup alloc: pre-run hook "network" failed:
failed to configure networking for alloc:
failed to configure bridge network:
incompatible CNI versions;
config is "0.4.0", plugin supports ["0.1.0" "0.2.0" "0.3.0" "0.3.1"]

Here’s the error when I tried to deploy the same nomad job using CNI plugin version 0.8.1 as suggested:
https://www.nomadproject.io/guides/integrations/consul-connect/index.html#cni-plugins

failed to setup alloc: pre-run hook "network"
failed: failed to configure networking for alloc:
failed to configure bridge network:
failed to create bridge "nomad":
could not add "nomad":
operation not supported

It’s unlikely we’re able to upgrade CNI plugin version anytime soon. Do you have a workaround? Is it possible for nomad to support CNI plugin version 0.7.x?

Reproduction steps

https://www.nomadproject.io/guides/integrations/consul-connect/index.html#cni-plugins

Job file (if appropriate)

Nomad Client logs (if appropriate)

If possible please post relevant logs in the issue.

Logs and other artifacts may also be sent to: [email protected]

Please link to your Github issue in the email and reference it in the subject
line:

To: [email protected]

Subject: GH-1234: Errors garbage collecting allocs

Emails sent to that address are readable by all HashiCorp employees but are not publicly visible.

Nomad Server logs (if appropriate)

[tgross]

Hi @thanhy! Unfortunately the reason we're requiring CNI version 0.8.2 (and probably higher once https://github.com/containernetworking/plugins/pull/408 is merged and released) is because those versions of CNI were released to fix bugs we found while building support into Nomad. In particular, races around the creation of interfaces and the network bridge. If we allowed the earlier version our support would not work.

Is PhotonOS publishing a package for CNI? Is it not possible to get the plugins directly from the CNI releases page?

[thanhy]

We've already making request for CNI plugin version upgrade with Photon OS team.
https://github.com/vmware/photon/issues/952

[tgross]

Gave that my :+1:, hopefully they're able to help you out there.

[idrennanvmware]

@tgross - Same issue here (I actually work with tmather) but I went through the steps of upgrading the CNI plugin's but this is now what Photon Reports

failed to setup alloc: pre-run hook "network" failed: failed to configure networking for alloc: failed to configure bridge network: failed to create bridge "nomad": could not add "nomad": operation not supported

Nomad functions as expected but the second a "side car service/consul connect aware" attempts to start with a bridge network, the above error recurs repeatedly

[tgross]

Hi @idrennanvmware! Unfortunately in #6567 we discovered more bugs in the CNI plugins around idempotent creation of the bridge, and that fix was merged into CNI as https://github.com/containernetworking/plugins/pull/408. That's on deck to be released as CNI 0.8.4... if they keep on their usual monthly cadence for releases that should be released in the next few days.

[idrennanvmware]

Thanks @tgross - we will keep an eye on that merge and try again

[tgross]

https://github.com/containernetworking/plugins/releases/tag/v0.8.4 was just released. I'm going to get that into testing and then we'll update the docs to recommend that version.

[idrennanvmware]

Awesome - ty. Excited to get this tested on Photon OS3