Gluon: respondd: duplicate ipv6 address in nodeinfo

[mweinelt]

For a while routers seem to be reporting some of their addresses twice in their nodeinfo response. The issue stems from the nodeinfo response on the router, as can be seen by calling gluon-neighbour-info locally on an affected router.

An excerpt given by Jason from Freifunk Frankfurt:

root@avm4020:~# gluon-neighbour-info -d ::1 -p 1001 -r nodeinfo -c 1
{
  "software": {
    "autoupdater": {
      "branch": "test",
      "enabled": true
    },
    "batman-adv": {
      "version": "2013.4.0",
      "compat": 14
    },
    "fastd": {
      "version": "v18",
      "enabled": false
    },
    "firmware": {
      "base": "gluon-2018.2",
      "release": "v2.5.1-test-0102"
    }
  },
  "network": {
    "addresses": [
      "2a06:8187:fbba:1234:ca0e:14ff:feb5:e1ad",
      "2a06:8187:fbba:1234:ca0e:14ff:feb5:e1ad",
      "fddd:5d16:b5dd::ca0e:14ff:feb5:e1ad",
      "fe80::ca0e:14ff:feb5:e1ad"
    ],
    "mesh": {
      "bat0": {
        "interfaces": {
          "wireless": [
            "be:2e:7d:2c:32:42"
          ],
          "other": [
            "be:2e:7d:2c:32:44",
            "be:2e:7d:2c:32:43",
            "be:2e:7d:2c:32:40"
          ]
        }
      }
    },
    "mac": "c8:0e:14:b5:e1:ad"
  },
  "location": {
    "latitude": 50.12771951,
    "longitude": 8.59648837
  },
  "owner": {
    "contact": "[email protected]"
  },
  "system": {
    "site_code": "ffffm"
  },
  "node_id": "c80e14b5e1ad",
  "hostname": "avm4020",
  "hardware": {
    "model": "AVM FRITZ!Box 4020",
    "nproc": 1
  }
}

root@avm4020:~# cat /proc/net/if_inet6
2a068187fbba1234ca0e14fffeb5e1ad 09 40 00 00 br-client
fddd5d16b5dd0000ca0e14fffeb5e1ad 09 40 00 00 br-client
fe80000000000000bc2e7dfffe2c3243 0a 40 20 80 primary0
fe80000000000000bc2e7dfffe2c3240 11 40 20 80  client0
fe80000000000000bc2e7dfffe2c3240 06 40 20 80   br-wan
fe80000000000000ca0e14fffeb5e1ad 0b 40 20 80     bat0
fe80000000000000ca0e14fffeb5e1ad 09 40 20 80 br-client
fe80000000000000144195fffe40f7dc 08 40 20 80 local-node
fe80000000000000bc2e7dfffe2c3242 10 40 20 80    ibss0
00000000000000000000000000000001 01 80 10 80       lo
fddd5d16b5dd00000000000000000001 08 80 00 a0 local-node
fe80000000000000bc2e7dfffe2c3244 04 40 20 80     eth1

The issue could have possibly been introduced in https://github.com/freifunk-gluon/gluon/commit/7408f046057f91daf5c78262d5ea9069bc843213 and is not always present. I've only seen it in roughly ~15% of our routers with a recent master build.

[NeoRaider]

I don't see how the code could possibly malfunction in a way that would turn 3 matching entries in if_inet6 into 4.

Is the behaviour stable - do the same nodes always exhibit the issue, or does it disappear with a reboot, or at random?

I guess we should switch from /proc/net/if_inet6 to the Netlink-based getifaddrs() in any case - I think the current code is still a remnant of the previous Lua-based respondd implementation.

[blocktrron]

Just some input - another node based on Gluon master:

root@64xxx-5c49793001b4:~# gluon-neighbour-info -d ::1 -p 1001 -r nodeinfo -c 1
{"software":{"autoupdater":{"branch":"stable","enabled":false},"batman-adv":{"version":"openwrt-2018.1-5","compat":15},"fastd":{"version":"v18","enabled":true,"public_key":"692f60178199654162151ddda08d3731778fe85de7698521096e8c5ca1608a92"},"firmware":{"base":"gluon-v2018.1-180-gc739e481","release":"1.3~20190109"}},"network":{"addresses":["2001:67c:2ed8:100e:5e49:79ff:fe30:1b4","2001:67c:2ed8:100e:5e49:79ff:fe30:1b4","fe80::5e49:79ff:fe30:1b4","fd01:67c:2ed8:100e:5e49:79ff:fe30:1b4"],"mesh":{"bat0":{"interfaces":{"wireless":["4a:f8:07:87:6e:c9"],"tunnel":["4a:f8:07:87:6e:cf"],"other":["4a:f8:07:87:6e:cb"]}}},"mac":"5c:49:79:30:01:b4"},"system":{"site_code":"ffda","domain_code":"ffda_64367"},"node_id":"5c49793001b4","hostname":"64xxx-5c49793001b4","hardware":{"model":"AVM FRITZ!Box 4020","nproc":1}}
root@64xxx-5c49793001b4:~# cat /proc/net/if_inet6
2001067c2ed8100e5e4979fffe3001b4 06 40 00 00 br-client
fe8000000000000048f807fffe876ecf 0f 40 20 80 mesh-vpn
fe800000000000005e4979fffe3001b4 0c 40 20 80     bat0
fe800000000000005e4979fffe3001b4 06 40 20 80 br-client
00000000000000000000000000000001 01 80 10 80       lo
fe8000000000000048f807fffe876ec9 0d 40 20 80    mesh0
fe80000000000000d8ff14fffe00ffff 0a 40 20 80 local-node
fd01067c2ed8100e5e4979fffe3001b4 06 40 00 00 br-client
fe8000000000000048f807fffe876ecb 0b 40 20 80 primary0
fe8000000000000048f807fffe876ec8 0e 40 20 80  client0
fd01067c2ed8100e0000000000010001 0a 80 00 a0 local-node
root@64xxx-5c49793001b4:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether ea:ef:ee:a6:d9:e6 brd ff:ff:ff:ff:ff:ff
3: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master br-wan state DOWN qlen 1000
    link/ether 5c:49:79:30:01:b3 brd ff:ff:ff:ff:ff:ff
4: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN qlen 1000
    link/ether 5c:49:79:30:01:b2 brd ff:ff:ff:ff:ff:ff
6: br-client: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 5c:49:79:30:01:b4 brd ff:ff:ff:ff:ff:ff
    inet6 2001:67c:2ed8:100e:5e49:79ff:fe30:1b4/64 scope global dynamic 
       valid_lft 86399sec preferred_lft 14399sec
    inet6 fd01:67c:2ed8:100e:5e49:79ff:fe30:1b4/64 scope global dynamic 
       valid_lft 86399sec preferred_lft 14399sec
    inet6 fe80::5e49:79ff:fe30:1b4/64 scope link 
       valid_lft forever preferred_lft forever
7: eth1.1@eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-client state LOWERLAYERDOWN qlen 1000
    link/ether 5c:49:79:30:01:b2 brd ff:ff:ff:ff:ff:ff
8: br-wan: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
    link/ether 4a:f8:07:87:6e:c8 brd ff:ff:ff:ff:ff:ff
9: local-port@local-node: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-client state UP qlen 1000
    link/ether 5c:49:79:30:01:b4 brd ff:ff:ff:ff:ff:ff
10: local-node@local-port: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether da:ff:14:00:ff:ff brd ff:ff:ff:ff:ff:ff
    inet 10.84.239.254/20 brd 10.84.239.255 scope global local-node
       valid_lft forever preferred_lft forever
    inet6 fd01:67c:2ed8:100e::1:1/128 scope global deprecated 
       valid_lft forever preferred_lft 0sec
    inet6 fe80::d8ff:14ff:fe00:ffff/64 scope link 
       valid_lft forever preferred_lft forever
11: primary0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1532 qdisc noqueue master bat0 state UNKNOWN qlen 1000
    link/ether 4a:f8:07:87:6e:cb brd ff:ff:ff:ff:ff:ff
    inet6 fe80::48f8:7ff:fe87:6ecb/64 scope link 
       valid_lft forever preferred_lft forever
12: bat0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-client state UNKNOWN qlen 1000
    link/ether 5c:49:79:30:01:b4 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5e49:79ff:fe30:1b4/64 scope link 
       valid_lft forever preferred_lft forever
13: mesh0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1532 qdisc noqueue master bat0 state UP qlen 1000
    link/ether 4a:f8:07:87:6e:c9 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::48f8:7ff:fe87:6ec9/64 scope link 
       valid_lft forever preferred_lft forever
14: client0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-client state UP qlen 1000
    link/ether 4a:f8:07:87:6e:c8 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::48f8:7ff:fe87:6ec8/64 scope link 
       valid_lft forever preferred_lft forever
15: mesh-vpn: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1312 qdisc fq_codel master bat0 state UNKNOWN qlen 1000
    link/ether 4a:f8:07:87:6e:cf brd ff:ff:ff:ff:ff:ff
    inet6 fe80::48f8:7ff:fe87:6ecf/64 scope link 
       valid_lft forever preferred_lft forever
root@64xxx-5c49793001b4:~# 

[kaechele]

I can confirm that on affected nodes we do see the same ULA address on br-client twice in ifconfig:

br-client Link encap:Ethernet  HWaddr 14:CC:20:C8:14:90  
          inet6 addr: fddf:ebfd:a801:214:16cc:20ff:fec8:1490/64 Scope:Global
          inet6 addr: fddf:ebfd:a801:214:16cc:20ff:fec8:1490/64 Scope:Global
          inet6 addr: fe80::16cc:20ff:fec8:1490/64 Scope:Link

but not in ip:

6: br-client: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 14:cc:20:c8:14:90 brd ff:ff:ff:ff:ff:ff
    inet6 fddf:ebfd:a801:214:16cc:20ff:fec8:1490/64 scope global dynamic 
       valid_lft 86382sec preferred_lft 14382sec
    inet6 fe80::16cc:20ff:fec8:1490/64 scope link 
       valid_lft forever preferred_lft forever

[rotanid]

interesting finding, maybe there's an upstream bug, too...

[mweinelt]

The issue seems to be device (not model) specific. I have a UAP-AC-M that even after multiple reboots always comes back with the problem.

root@64283-cccda-w:~# cat /proc/net/if_inet6 
fd01067c2ed81001f29fc2fffedec4c5 09 40 00 00 br-client
fe80000000000000f29fc2fffedec4c5 0b 40 20 80     bat0
fe80000000000000f29fc2fffedec4c5 09 40 20 80 br-client
fe80000000000000b8f560fffedd3973 0a 40 20 80 primary0
00000000000000000000000000000001 01 80 10 80       lo
fd01067c2ed810010000000000010001 08 80 00 a0 local-node
fe80000000000000b8f560fffedd3970 0e 40 20 80  client0
fe80000000000000b8f560fffedd3970 0f 40 20 80 vx_mesh_wan
fe80000000000000b8f560fffedd3970 06 40 20 80   br-wan
fe80000000000000d8ff01fffe00ffff 08 40 20 80 local-node
fe80000000000000b8f560fffedd3975 0c 40 20 80    mesh1
fe80000000000000f29fc2fffeddc4c5 0d 40 20 80  client1
2001067c2ed81001f29fc2fffedec4c5 09 40 00 00 br-client

the busybox ifconfig applet, using /proc/net/if_inet6 as well, also exhibits this issue:

root@64283-cccda-w:~# ifconfig -a br-client
br-client Link encap:Ethernet  HWaddr F0:9F:C2:DE:C4:C5  
          inet6 addr: fd01:67c:2ed8:1001:f29f:c2ff:fede:c4c5/64 Scope:Global
          inet6 addr: fd01:67c:2ed8:1001:f29f:c2ff:fede:c4c5/64 Scope:Global
          inet6 addr: fe80::f29f:c2ff:fede:c4c5/64 Scope:Link
          inet6 addr: 2001:67c:2ed8:1001:f29f:c2ff:fede:c4c5/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:26231 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5786 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2741459 (2.6 MiB)  TX bytes:815270 (796.1 KiB)

root@64283-cccda-w:~# strace ifconfig -a br-client
execve("/sbin/ifconfig", ["ifconfig", "-a", "br-client"], 0x7f806e98 /* 12 vars */) = 0
set_thread_area(0x77917dc0)             = 0
set_tid_address(0x77910d28)             = 10905
open("/etc/ld-musl-mips-sf.path", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/libgcc_s.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=78096, ...}) = 0
read(3, "\177ELF\1\2\1\0\0\0\0\0\0\0\0\0\0\3\0\10\0\0\0\1\0\0(p\0\0\0004"..., 936) = 936
mmap2(NULL, 147456, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x77848000
mmap2(0x7786b000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x13000) = 0x7786b000
close(3)                                = 0
mprotect(0x45b000, 4096, PROT_READ)     = 0
prctl(PR_SET_NAME, "ifconfig")          = 0
getuid()                                = 0
open("/proc/net/dev", O_RDONLY|O_LARGEFILE) = 3
readv(3, [{iov_base="", iov_len=0}, {iov_base="Inter-|   Receive               "..., iov_len=1024}], 2) = 1024
readv(3, [{iov_base="", iov_len=0}, {iov_base="945492    6896    0    0    0   "..., iov_len=1024}], 2) = 805
_llseek(3, -750, [1079], SEEK_CUR)      = 0
close(3)                                = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 3
ioctl(3, SIOCGIFFLAGS, {ifr_name="br-client", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MULTICAST}) = 0
ioctl(3, SIOCGIFHWADDR, {ifr_name="br-client", ifr_hwaddr={sa_family=ARPHRD_ETHER, sa_data=f0:9f:c2:de:c4:c5}}) = 0
ioctl(3, SIOCGIFMETRIC, {ifr_name="br-client", ifr_metric=0}) = 0
ioctl(3, SIOCGIFMTU, {ifr_name="br-client", ifr_mtu=1500}) = 0
ioctl(3, SIOCGIFMAP, {ifr_name="br-client", ifr_map={mem_start=0, mem_end=0, base_addr=0, irq=0, dma=0, port=0}}) = 0
ioctl(3, SIOCGIFTXQLEN, {ifr_name="br-client", ifr_qlen=1000}) = 0
ioctl(3, SIOCGIFADDR, {ifr_name="br-client"}) = -1 EADDRNOTAVAIL (Address not available)
close(3)                                = 0
ioctl(1, TIOCGWINSZ, 0x7fad84a0)        = 0
writev(1, [{iov_base="br-client Link encap:Ethernet  H"..., iov_len=57}, {iov_base="\n", iov_len=1}], 2br-client Link encap:Ethernet  HWaddr F0:9F:C2:DE:C4:C5  
) = 58
open("/proc/net/if_inet6", O_RDONLY|O_LARGEFILE) = 3
readv(3, [{iov_base="", iov_len=0}, {iov_base="fd01067c2ed81001f29fc2fffedec4c5"..., iov_len=1024}], 2) = 767
writev(1, [{iov_base="          inet6 addr: fd01:67c:2"..., iov_len=76}, {iov_base="\n", iov_len=1}], 2          inet6 addr: fd01:67c:2ed8:1001:f29f:c2ff:fede:c4c5/64 Scope:Global
) = 77
writev(1, [{iov_base="          inet6 addr: fd01:67c:2"..., iov_len=76}, {iov_base="\n", iov_len=1}], 2          inet6 addr: fd01:67c:2ed8:1001:f29f:c2ff:fede:c4c5/64 Scope:Global
) = 77
writev(1, [{iov_base="          inet6 addr: fe80::f29f"..., iov_len=61}, {iov_base="\n", iov_len=1}], 2          inet6 addr: fe80::f29f:c2ff:fede:c4c5/64 Scope:Link
) = 62
readv(3, [{iov_base="", iov_len=0}, {iov_base="", iov_len=1024}], 2) = 0
writev(1, [{iov_base="          inet6 addr: 2001:67c:2"..., iov_len=76}, {iov_base="\n", iov_len=1}], 2          inet6 addr: 2001:67c:2ed8:1001:f29f:c2ff:fede:c4c5/64 Scope:Global
) = 77
close(3)                                = 0
writev(1, [{iov_base="          UP BROADCAST RUNNING M"..., iov_len=60}, {iov_base="\n", iov_len=1}], 2          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
) = 61
writev(1, [{iov_base="          RX packets:28120 error"..., iov_len=64}, {iov_base="\n", iov_len=1}], 2          RX packets:28120 errors:0 dropped:0 overruns:0 frame:0
) = 65
writev(1, [{iov_base="          TX packets:6896 errors"..., iov_len=65}, {iov_base="\n", iov_len=1}], 2          TX packets:6896 errors:0 dropped:0 overruns:0 carrier:0
) = 66
writev(1, [{iov_base="          collisions:0 txqueuele"..., iov_len=39}, {iov_base="\n", iov_len=1}], 2          collisions:0 txqueuelen:1000 
) = 40
writev(1, [{iov_base="          RX bytes:3887581 (3.7 "..., iov_len=65}, {iov_base="\n", iov_len=1}], 2          RX bytes:3887581 (3.7 MiB)  TX bytes:945492 (923.3 KiB)
) = 66
writev(1, [{iov_base="", iov_len=0}, {iov_base="\n", iov_len=1}], 2
) = 1
exit_group(0)                           = ?
+++ exited with 0 +++

We did a single device testrun with https://github.com/blocktrron/gluon/commit/71eb8ba53ce607488e186739d961639db02a7289 yesterday and it looked promising. Tonight we will roll this out to another 30 nodes and see if the issue is gone entirely.

[collimas]

What are the results from your test? Is the problem solved?

[blocktrron]

@collimas see #1616

[Byggvir]

Updated yesterday a TP-Link TL-WDR3600 v1 to 2018.2. The device has now 2 identical IPv6 addresses on br-client. Add does not show on the map anymore. Also the connections don't to the WAN don't work.
If I delete the duplicate address bot entries are deleted. If I add the address the interface gets two identical addresses.

 OpenWrt 18.06-SNAPSHOT, r7629+12-eef6bd3393
 -----------------------------------------------------
root@su-rhb-zingsheim-31:~# ifconfig br-client
br-client Link encap:Ethernet  HWaddr C4:E9:84:F9:CB:AE  
          inet6 addr: 2a03:2260:3017:1400:c6e9:84ff:fef9:cbae/64 Scope:Global
          inet6 addr: 2a03:2260:3017:1400:c6e9:84ff:fef9:cbae/64 Scope:Global
          inet6 addr: fe80::c6e9:84ff:fef9:cbae/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:14 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1096 (1.0 KiB)  TX bytes:4052 (3.9 KiB)

root@su-rhb-zingsheim-31:~# ip a del 2a03:2260:3017:1400:c6e9:84ff:fef9:cbae/64 dev br-client
root@su-rhb-zingsheim-31:~# ifconfig br-client
br-client Link encap:Ethernet  HWaddr C4:E9:84:F9:CB:AE  
          inet6 addr: fe80::c6e9:84ff:fef9:cbae/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:18 errors:0 dropped:0 overruns:0 frame:0
          TX packets:51 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1208 (1.1 KiB)  TX bytes:5248 (5.1 KiB)

root@su-rhb-zingsheim-31:~# ip a add 2a03:2260:3017:1400:c6e9:84ff:fef9:cbae/64 dev br-client
root@su-rhb-zingsheim-31:~# ifconfig br-client
br-client Link encap:Ethernet  HWaddr C4:E9:84:F9:CB:AE  
          inet6 addr: 2a03:2260:3017:1400:c6e9:84ff:fef9:cbae/64 Scope:Global
          inet6 addr: 2a03:2260:3017:1400:c6e9:84ff:fef9:cbae/64 Scope:Global
          inet6 addr: fe80::c6e9:84ff:fef9:cbae/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:19 errors:0 dropped:0 overruns:0 frame:0
          TX packets:55 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1236 (1.2 KiB)  TX bytes:5750 (5.6 KiB)

[mweinelt]

The device has now 2 identical IPv6 addresses on br-client.

Because the busybox applet ifconfig has the same issue, but #1616 only patched respondd.

Add does not show on the map anymore.

Add meaning Additional address? If you merged #1616 into your build that is expected.

Also the connections don't to the WAN don't work.

No idea what that means.

If I delete the duplicate address bot entries are deleted.

Because they're not actually there twice. Use ip addr instead of ifconfig.

If I add the address the interface gets two identical addresses.

See before.

[Byggvir]

Today I installed two virgin AVM 450E.

On device shows duplicate addresses with ifconfig, the other one only one address.

[blocktrron]

@Byggvir ifconfig is out of scope for gluon. Is you image built with #1616 and shows respondd a duplicate address behavior?