Weave: name/mac derivation not unique on some hosted and physical machines

[tmsmr]

This is no real issue - just a hint.

Online.net has cheap dedicated hosts (Dedibox) which seem to share all the same product UUID:
12345678-1234-5678-90AB-CDDEEFAABBCC.

This causes identical peer names, which results in a non-working weave installation.

To run weave on these hosts, the peer names has to be specified manually with e.g. --name ::1.

I suggest an entry in the FAQ section of the documentation regarding this problem. Maybe there are other hosters that do the same thing with their firmware...

[rade]

Interesting. Is there is some other unique ID we could use here?

[brb]

As it can be seen https://botbot.me/freenode/weavenetwork/2016-07-06/?msg=69184491&page=1 , the user got "cannot connect to ourself, retry: never". Perhaps, we could improve error reporting for this case.

[tmsmr]

I don't think there is any better unique (hardware-dependant) value. I can send you the full dmidecode output if you want.

As @brb suggested already, better error reporting would be sufficient. The confusing part for me was:

# weave status connections -> $PEERIP:6783 failed cannot connect to ourself, retry: never

while

# weave status connections -> $PEERIP:6783 (Peer ca:dd:16:be:df:42) failed cannot connect to ourself, retry: never

or

# weave status connections -> $PEERIP:6783 failed cannot connect to ourself (Me: ca:dd:16:be:df:42, Peer: ca:dd:16:be:df:42), retry: never

would be better in this case.

[rade]

We could actually detect this situation very precisely by transmitting a per-run id in the handshake. Then when detect the same peer id but different run id we know we must have a peer id clash.

Note that we already have such a run id - the peer uid, which is distinct from the name. And it is transmitted in the handshake.

[rade]

So there is really two sides to this issue:

1. improve the detection / error message
2. come up with a better way of deriving a UUID

Re 2... the questions whether there is anything sufficiently durable (i.e. survives reboots) _and_ unique that we could derive a UUID from.

[rade]

@opthomas-prime what specific Dedibox flavour are you running?

[tmsmr]

It's the "DEDIBOX® XC 2016 SSD" (https://www.online.net/en/dedicated-server/dedibox-xc).
These are custom-built, blade-style servers with their own custom firmware:

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
    Vendor: Online Labs
    Version: 00.00.00.0011
    Release Date: 03/29/2016
...
Handle 0x0001, DMI type 1, 27 bytes
System Information
    Manufacturer: Online Labs
    Product Name: SR
    Version: (^_^)
    Serial Number: 42
    UUID: 12345678-1234-5678-90AB-CDDEEFAABBCC
    Wake-up Type: Power Switch
    SKU Number: (^_^)
    Family: Avoton
...

EDIT: While the given values are the same on all hosts!

If you need further information or want me to test something, feel free to ask.

[brb]

On unique IDs: http://0pointer.de/blog/projects/ids.html

[rade]

On unique IDs: http://0pointer.de/blog/projects/ids.html

Yes. We have gone through much deliberation and research here. Poke around github, the comments and the commit messages.

[tmsmr]

What about using the mac address of the first ethernet interface as prefix for the product UUID instead of a static string (https://github.com/weaveworks/weave/blob/master/weave#L531)?

Something like:

NIC=`find /sys/class/net/ -maxdepth 1 -type l -name "en*" -or -name "eth*" | sort | head -n 1`
if [ -e ${NIC} ]; then
  ADDR=`cat ${NIC}/address`
  UUID="$ADDR-$UUID"
fi

[rade]

AFAIK interface order is not stable on modern linuxes. And MAC addresses may not be stable either.

[tmsmr]

Is the CPUID/EAX=3 (As Lennart P. describes in http://0pointer.de/blog/projects/ids.html) stable in virtualized environments?

Anyway, i don't want to interfer your discussion.

Thx for caring about this problem!

[gmessner]

We also have the same issue where the product_uuid is the same on multiple hosts. For example, we have 12 x Dell C6145 servers (each with 2 mainboards), out of those 24 mainboards there are only 6 unique product_uuid. This has caused us many headaches.

How about prefixing product_uuid with the contents of /var/lib/dbus/machine-id (or /etc/machine-id) if present?

Another possible solution would to add an option and allow the file containing a unique ID to be specified.

[rade]

How about prefixing product_uuid with the contents of /var/lib/dbus/machine-id (or /etc/machine-id) if present?

Is /etc/machine-id a) distinct on all your machines, and b) constant across reboots?

[rade]

To run weave on these hosts, the peer names has to be specified manually with e.g. --name ::1.

That this isn't the only issue. The UUID is used to generate the MAC for the weave bridge, which in turn is used as the peer name. If the UUIDs aren't unique then both the weave bridge MACs and peer names aren't unique. Setting the peer names with --name fixes the latter but not the former. Having multiple peers with the same weave bridge MAC will cause MAC collisions, which in turn prevents weave expose from working properly.

[gmessner]

The UUID being used for the weave bridge MAC is exactly we we are having our issues. In my example of 24 servers, 18 of the 24 had a non-unique MAC on the weave bridge because of this.

As far as your question:

Is /etc/machine-id a) distinct on all your machines, and b) constant across reboots?

It usually is, but not 100%, I have seen cases where it was specifically set up to regenerate on every boot. This is why I suggested a command line option so that a file containing a UUID could be specified when launching weave. This would allow the user to control what file is being used and if needed create their own UUID and store in their own file that is persisted across reboots. Would be a way to 100% guarantee the UUID.

[rade]

@opthomas-prime same question to you, i.e. is /etc/machine-id a) distinct on all your machines, and b) constant across reboots?

[tmsmr]

@rade both, a) and b) are true on the problematic online.net hosts!

This is not surprising, since this ID is random-generated during the OS installation (https://www.freedesktop.org/software/systemd/man/machine-id.html).

BUT: Only on systemd boxes!

[rade]

@brb we should also fix https://github.com/weaveworks/go-checkpoint/blob/master/checkpoint.go#L342

[bboreham]

Note we are getting more reports of duplicate IDs from users using Kubernetes.
https://github.com/weaveworks/weave-kube/issues/19 relates

[awh]

we should also fix https://github.com/weaveworks/go-checkpoint/blob/master/checkpoint.go#L342

We should factor this out (along with its copy in the weave script) to a single place so it can't get out of sync.

[jalberto]

I am hitting same issue in scaleway servers on weave-1.7.2 (deployed using kontena)

[jakolehm]

How about adding an option to provide uuid via env?

[bboreham]

@jakolehm you can set the peer-name with --name; is it much easier for you to use an environment variable?

[jakolehm]

@bboreham peer-name does not affect mac? (we are using expose)

[bboreham]

@jakolehm ok, I guess that was mentioned at https://github.com/weaveworks/weave/issues/2427#issuecomment-232256346

Have you tried it? I am not entirely convinced that the mac clash will cause a problem in practice.

[bboreham]

@rade points out that packets entering the Weave network with a source NAT is one example where it is likely to cause a problem.

[jalberto]

What about using a combinaiton of sources?

Find N things permanent in the system (productID, machine-ID, FooEnvVar, etc), use X of N to generate the needed UUID (eg: 3 of 5) thrown error is there is not enough sources.

[bboreham]

@jalberto we already have two sources; #2460 added two more but then we realised it needs to be backwards-compatible otherwise an upgrade will break an existing cluster. And making it backwards-compatible is proving to be tricky.

[jalberto]

@bboreham yep back-compatibility is hell some times.

Then the decision that must be taken is, keep it back-compatible but a blocker for new deploys in baremetal/datacenters, or broke in some point with older versions but be able to use weave in this areas.

a classic startegy will be to use a deprecaiton flag that be enable/disable for few versions.

I think the multi-source approach is more future-proof

whatever decision is taken it must be clear in the documents, so people don't spend time implementing something that will not work for some uses-cases and that will affect projects based on weave in a way or another.

[rade]

What's wrong with using the bridge mac / persisted peer id if they exist, and, if absent, generate new IDs according to the expanded formula?

It would mean that a machine which is restarted after an upgrade to the new version, and lost its persisted data, will get a new ID. That strikes me as a worthwhile sacrifice to make.

[gajus]

@jalberto Have you managed to solve this on Scaleway?

How?

/sys/class/dmi/id/product_uuid cannot be overwritten. Therefore, the only possible solution is to use a custom image of weave that uses /etc/machine-id. This is sub-optimal to say the least.

As for a backwards compatible solution, you could check for existence of a custom file (e.g. /var/weave-machine-uuid) and if it exists, use its content as a unique identifier instead of /sys/class/dmi/id/product_uuid.

Then all thats left to do is to document the existing issue with /sys/class/dmi/id/product_uuid in the troubleshooting section of the documentation and suggest a solution: populate /var/weave-machine-uuid with a random value.

[gajus]

I needed a temporary workaround to make this work on Scaleway.

Here is how to use it:

# On every node run
mkdir /etc/weave
uuidgen > /etc/weave/machine-uuid

And to setup weave-daemonset:

kubectl create -f https://gist.githubusercontent.com/gajus/ce8fd5ffae4583e4943c199245d3e317/raw/46b666e97b515b4f6126f433774764bb7a85bc04/weave-daemonset.yaml

Note:

Don't use it in production. The image is not going to be maintained. This is a quick workaround that enables setting up development environment on Scaleway (and other providers that do not ensure unique /sys/class/dmi/id/product_uuid UUID value).

In terms of the code changes, the only thing I have changed in ./weave is:

+     MAC=$(mac_from_file /etc/weave/machine-uuid) || \
        MAC=$(mac_from_file /sys/class/dmi/id/product_uuid) || \
        MAC=$(mac_from_file /sys/hypervisor/uuid) || \
        MAC=$(random_mac)

[bboreham]

What's wrong with using the bridge mac / persisted peer id if they exist

I believe the original objection to this was that a user creating cloned VM images after running Weave Net would get duplicated IDs. But that's probably not a likely scenario.

It does need some revamping in the code, to introduce something which can read the persisted ID at the point it is needed. Also to cope with the different locations of persisted data for weave and weave-kube.