Libnetwork: flood of 'level=error msg="Failed to deserialize netlink ndmsg: Link not found"'

[salzig]

Hej,
i'm in the situation where a swarm of two nodes writes endlessly "Failed to deserialize netlink ndmsg: Link not found" into the daemon logs.

Sadly https://github.com/docker/libnetwork/blob/master/drivers/overlay/ov_network.go#L746 dosn't provide any further information for debug.

As i'm not a go developer, would it be ok to add the following right after the Error output for further error details?

logrus.Debugf("netlink ndmsg deserialize failed for: %v", msg.Data)

[soakes]

I am also experiencing the same issue. This seems to happen when a node leaves or joins the swarm cluster. This is running on CentOS 7 with the latest stable release of docker.

Server Version: 17.12.0-ce

Jan 10 06:04:10 dmgr01 dockerd[873]: time="2018-01-10T06:04:10.686514236Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Jan 10 06:04:10 dmgr01. dockerd[873]: time="2018-01-10T06:04:10.686582238Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Jan 10 06:04:10 dmgr01 dockerd[873]: time="2018-01-10T06:04:10.686610542Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"

The only pattern that I have found is if a node gets rebooted/restarted or otherwise disconnects from the cluster, then this message is usually spammed to system logs.

These are XEN HVM VM, however, I haven't seen these to my knowledge on KVM yet. The builds themselves are identical. I have also started experiencing some weird issues but unsure if they are related to this. In short, I keep having a random manager stop responding to the ports it should be using. I can confirm with a netstat that the ports are running but if you try and telnet to it, it doesn't connect. I have also tried to remove and replace the ingress network but this hasn't helped either.

I also don't recall ever seeing anything like this on coreos before, so I might switch back later and see if the problem is resolved. This might be something in the rpm build, I don't know anything more at this time. Hope this can help.

[eugeneduvenage]

I am experiencing the same issue on Ubuntu 16.04 servers freshly upgraded Docker version 17.12.0-ce, build c97c6d6. I have a non swarm overlay network setup across multiple machines using consul as the KV store. No other errors are being reported that could point to any cause. The message appears on each of the machines in the cluster.

[shkrid]

Same issue after upgrade swarm cluster from 17.09.1 -> 17.12.0

[soakes]

I can also confirm this is only happening from nodes which are added to swarm cluster. This has been tested with identical OS/Docker versions with the exception that four nodes were not in the cluster but the rest were.

This I think must be something to do with the swarm network (ingress?) even tho I have deleted this and re-set this up, it's not helped.

One thing to note, I am also using Consul, however, I have checked with a colleague at work with almost identical setup to myself and he doesn't run Consul but still suffers this problem, so I think we can rule that out.

Does anything have any ideas? apart from downgrading docker.

[fcrisciani]

This PR most likely is the cause: https://github.com/vishvananda/netlink/pull/263

[fcrisciani]

This PR will remove the issue for swarm mode: https://github.com/docker/libnetwork/pull/2047

[fcrisciani]

Just posted a fix, this will be taken on next release and next patch release for 17.12

[soakes]

Thanks so much for sorting this so quickly. I am impressed.

[vadimslu]

I'm still getting level=error msg="Failed to deserialize netlink ndmsg: Link not found errors after upgrading to Docker version 18.01.0-ce, build 03596f5

[fcrisciani]

@vadimslu the commit is merged into libnetwork master, to be picked has to be vendored into moby/moby. If you want you can revendor this commit into moby/moby master and build your own binary

[fcrisciani]

Closing this one, considering that the code is merged

[salzig]

@fcrisciani still like to know if the provided code would work. (extending logging)

Resolving a problem is nice, but better debugability would be nice too.

[fcrisciani]

@salzig thanks for the proposal, totally agree about the debuggability point!
I put some more details in the PR itself about the reasoning of the issue.

Let's say in this case your suggested change has a couple of problems in my opinion:
1) the line that you indicated is actually not where is the return statement, but the log is coming from: https://github.com/docker/libnetwork/blob/master/drivers/overlay/ov_network.go#L766.
2) simply printing in Debug would only kind of hide the issue, in the sense that if you are not in debug mode you won't realize that is happening, the behavior was still correct.
3) If the data does not properly deserialize can be simply junk binary data so potentially is not super useful for debug anyway and potentially not printable as a string so most likely in case of a real issue printing the binary would not have helped anyway.