I was waiting for the official release of VRRP into FRR to replace Keepalived, but I've noticed now that it uses macvlan
"_In macvlan, a single interface has multiple IP and MAC._"
In virtual switch (as VMware) this is allowed only with promiscuous mode with a big impact on infrastructure performance.
Is it possible to implement VRRP with unique MAC address?
smarcosmarco
All 23 comments
@smarcosmarco macvlan devices are netdevices like any other interface device in Linux. They are slaved to a physical interface. I am confused on whether you think you have to enable promiscuous mode (due to a misunderstanding of how macvlan works) or if you are referring to some specific issue in a vmware appliance (not clear to me which product you're talking about) where macvlan devices require promiscuous mode on the base interface.
Finally, I believe keepalived uses macvlans as well. Do correct me if I'm wrong.
qlyoung
on 22 Nov 2019
@polychaeta autoclose in 1 day
qlyoung
on 23 Nov 2019
Hello @qlyoung
Keepalived add "mobile IP address" to the same interface and MAC.
In this case promiscuous mode is not needed.
Here an example where XXX.XXX.32.1/28 is managed by Keepalived:
# ip addr
ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:ba:19:ee brd ff:ff:ff:ff:ff:ff
inet XXX.XXX.32.2/28 brd XXX.XXX.32.15 scope global ens192
valid_lft forever preferred_lft forever
inet XXX.XXX.32.1/28 brd XXX.XXX.32.15 scope global secondary ens192
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:feba:19ee/64 scope link
valid_lft forever preferred_lft forever
# ifconfig
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet XXX.XXX.32.2 netmask 255.255.255.240 broadcast XXX.XXX.32.15
inet6 fe80::250:56ff:feba:19ee prefixlen 64 scopeid 0x20<link>
ether 00:50:56:ba:19:ee txqueuelen 1000 (Ethernet)
RX packets 727051638 bytes 408802970966 (408.8 GB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1017453444 bytes 1021439960799 (1.0 TB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Instead FRR need to use a different MAC as the documentation:
ip link add vrrp4-2-1 link eth0 addrgenmode random type macvlan mode bridge
ip link set dev vrrp4-2-1 address 00:00:5e:00:01:05
ip addr add 10.0.2.16/24 dev vrrp4-2-1
ip link set dev vrrp4-2-1 up
The result is that the virtual interface has two different MAC addresses and not just one.
This doesn't work without promiscuous mode.
smarcosmarco
on 25 Nov 2019
@polychaeta how to reopen the issue?
smarcosmarco
on 25 Nov 2019
@smarcosmarco did you actually try it and see it not work? Or are you inferring based on the documentation?
The virtual interface does not have two MAC addresses, it has one.
qlyoung
on 25 Nov 2019
@smarcosmarco did you actually try it and see it not work? Or are you inferring based on the documentation?
I tried with two VM (FRR 7.2 + VRRP) in our environment without a good result.
A simple tcpdump show that packet are announced by the VMs but do not arrive on the network/virtual network.
I have not tried to enable promiscuous mode to not impact our infrastructure. I will try tomorrow in a testing environment. I'm pretty sure that's the reason...
Productions VMs with FRR 7.1 and Keepalived work correctly.
smarcosmarco
on 25 Nov 2019
I tried with two VM (FRR 7.2 + VRRP) in our environment without a good result.
On which kernel?
What is the macvlan device mode?
What are your iptables rules, if any?
Which device are you capturing packets on?
Are you using any VRFs?
Etc, etc.
Would you like to find out what the problem with your configuration is, or do you just want to know whether promiscuous mode is required for FRR VRRP to operate? If the latter - it's not. If the former - let's take a look at your configuration and see what the actual problem is here.
qlyoung
on 25 Nov 2019
test1 and test2 are two virgins VM (basic installation without any settings)
root@test1:~# lsb_release -a
Description: Ubuntu 18.04.3 LTS
root@test1:~# uname -a
Linux test1 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 10:36:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
root@test1:~# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Here the interfaces where VRRP-4-2-1 have different MAC (00:00:5e:00:01:01) from ens160
root@test1:~# ifconfig
ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.202.101 netmask 255.255.255.0 broadcast 192.168.202.255
inet6 fe80::250:56ff:feba:3192 prefixlen 64 scopeid 0x20<link>
ether 00:50:56:ba:31:92 txqueuelen 1000 (Ethernet)
RX packets 1449532 bytes 240540384 (240.5 MB)
RX errors 0 dropped 1311977 overruns 0 frame 0
TX packets 6209978 bytes 286857639 (286.8 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 428 bytes 38692 (38.6 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 428 bytes 38692 (38.6 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vrrp4-2-1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.202.100 netmask 255.255.255.0 broadcast 0.0.0.0
inet6 fe80::200:5eff:fe00:101 prefixlen 64 scopeid 0x20<link>
ether 00:00:5e:00:01:01 txqueuelen 1000 (Ethernet)
RX packets 1385289 bytes 147526923 (147.5 MB)
RX errors 0 dropped 1311764 overruns 0 frame 0
TX packets 6173023 bytes 283959466 (283.9 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
root@test1:~# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:ba:31:92 brd ff:ff:ff:ff:ff:ff
inet 192.168.202.101/24 brd 192.168.202.255 scope global ens160
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:feba:3192/64 scope link
valid_lft forever preferred_lft forever
3: vrrp4-2-1@ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:00:5e:00:01:01 brd ff:ff:ff:ff:ff:ff
inet 192.168.202.100/24 scope global vrrp4-2-1
valid_lft forever preferred_lft forever
inet6 fe80::200:5eff:fe00:101/64 scope link
This is the status from FRR
test1# sh running-config
Building configuration...
Current configuration:
!
frr version 7.2
frr defaults traditional
hostname test1
log syslog informational
service integrated-vtysh-config
!
interface ens160
vrrp 1
vrrp 1 advertisement-interval 100
vrrp 1 priority 200
vrrp 1 ip 192.168.202.100
!
line vty
!
end
test1# sh vrrp
Virtual Router ID 1
Protocol Version 3
Autoconfigured No
Shutdown No
Interface ens160
VRRP interface (v4) vrrp4-2-1
VRRP interface (v6) None
Primary IP (v4) 192.168.202.101
Primary IP (v6) ::
Virtual MAC (v4) 00:00:5e:00:01:01
Virtual MAC (v6) 00:00:5e:00:02:01
Status (v4) Master
Status (v6) Initialize
Priority 200
Effective Priority (v4) 200
Effective Priority (v6) 200
Preempt Mode Yes
Accept Mode Yes
Advertisement Interval 100 ms
Master Advertisement Interval (v4) 100 ms
Master Advertisement Interval (v6) 0 ms
Advertisements Tx (v4) 6175020
Advertisements Tx (v6) 0
Advertisements Rx (v4) 0
Advertisements Rx (v6) 0
Gratuitous ARP Tx (v4) 1
Neigh. Adverts Tx (v6) 0
State transitions (v4) 2
State transitions (v6) 0
Skew Time (v4) 20 ms
Skew Time (v6) 0 ms
Master Down Interval (v4) 320 ms
Master Down Interval (v6) 0 ms
IPv4 Addresses 1
.................................. 192.168.202.100
IPv6 Addresses 0
VMware Policy Settings:
Security
--
Promiscuous mode | Reject
MAC address changes | Reject
Forged transmits | Reject
And here you can see VRRP packets with different MAC (00:00:5e:00:01:01) from ens160 (00:50:56:ba:31:92)
root@test1:~# tcpdump -i ens160 -en not port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
13:31:25.122743 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 46: 192.168.202.101 > 224.0.0.18: VRRPv3, Advertisement, vrid 1, prio 200, intvl 10cs, length 12
13:31:25.222835 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 46: 192.168.202.101 > 224.0.0.18: VRRPv3, Advertisement, vrid 1, prio 200, intvl 10cs, length 12
13:31:25.322907 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 46: 192.168.202.101 > 224.0.0.18: VRRPv3, Advertisement, vrid 1, prio 200, intvl 10cs, length 12
13:31:25.422986 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 46: 192.168.202.101 > 224.0.0.18: VRRPv3, Advertisement, vrid 1, prio 200, intvl 10cs, length 12
13:31:25.523063 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 46: 192.168.202.101 > 224.0.0.18: VRRPv3, Advertisement, vrid 1, prio 200, intvl 10cs, length 12
VM 2 can't see VRRP packets
root@test2:~# tcpdump -i ens160 -en vrrp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
7 packets dropped by interface
Running FRR on test2:
test2# sh vrrp
Virtual Router ID 1
Protocol Version 3
Autoconfigured No
Shutdown No
Interface ens160
VRRP interface (v4) vrrp4-2-1
VRRP interface (v6) None
Primary IP (v4) 192.168.202.102
Primary IP (v6) ::
Virtual MAC (v4) 00:00:5e:00:01:01
Virtual MAC (v6) 00:00:5e:00:02:01
Status (v4) Master
Status (v6) Initialize
Priority 100
Effective Priority (v4) 100
Effective Priority (v6) 100
Preempt Mode Yes
Accept Mode Yes
Advertisement Interval 100 ms
Master Advertisement Interval (v4) 100 ms
Master Advertisement Interval (v6) 0 ms
Advertisements Tx (v4) 120
Advertisements Tx (v6) 0
Advertisements Rx (v4) 0 <------
Advertisements Rx (v6) 0
Gratuitous ARP Tx (v4) 1
Neigh. Adverts Tx (v6) 0
State transitions (v4) 2
State transitions (v6) 0
Skew Time (v4) 60 ms
Skew Time (v6) 0 ms
Master Down Interval (v4) 360 ms
Master Down Interval (v6) 0 ms
IPv4 Addresses 1
.................................. 192.168.202.100
IPv6 Addresses 0
Now, changing VMware settings to:
Security
--
Promiscuous mode | Accept
MAC address changes | Accept
Forged transmits | Accept
this is the result
root@test2:~# tcpdump -i ens160 -en vrrp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
13:41:21.615932 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 60: 192.168.202.101 > 224.0.0.18: VRRPv3, Advertisement, vrid 1, prio 200, intvl 10cs, length 12
13:41:21.715965 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 60: 192.168.202.101 > 224.0.0.18: VRRPv3, Advertisement, vrid 1, prio 200, intvl 10cs, length 12
13:41:21.815997 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 60: 192.168.202.101 > 224.0.0.18: VRRPv3, Advertisement, vrid 1, prio 200, intvl 10cs, length 12
13:41:21.916044 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 60: 192.168.202.101 > 224.0.0.18: VRRPv3, Advertisement, vrid 1, prio 200, intvl 10cs, length 12
13:41:22.016170 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 60: 192.168.202.101 > 224.0.0.18: VRRPv3, Advertisement, vrid 1, prio 200, intvl 10cs, length 12
13:41:22.116218 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 60: 192.168.202.101 > 224.0.0.18: VRRPv3, Advertisement, vrid 1, prio 200, intvl 10cs, length 12
test2# sh vrrp
Virtual Router ID 1
Protocol Version 3
Autoconfigured No
Shutdown No
Interface ens160
VRRP interface (v4) vrrp4-2-1
VRRP interface (v6) None
Primary IP (v4)
Primary IP (v6) ::
Virtual MAC (v4) 00:00:5e:00:01:01
Virtual MAC (v6) 00:00:5e:00:02:01
Status (v4) Backup
Status (v6) Initialize
Priority 100
Effective Priority (v4) 100
Effective Priority (v6) 100
Preempt Mode Yes
Accept Mode Yes
Advertisement Interval 100 ms
Master Advertisement Interval (v4) 100 ms
Master Advertisement Interval (v6) 0 ms
Advertisements Tx (v4) 707
Advertisements Tx (v6) 0
Advertisements Rx (v4) 98 <------------
Advertisements Rx (v6) 0
Gratuitous ARP Tx (v4) 1
Neigh. Adverts Tx (v6) 0
State transitions (v4) 3
State transitions (v6) 0
Skew Time (v4) 60 ms
Skew Time (v6) 0 ms
Master Down Interval (v4) 360 ms
Master Down Interval (v6) 0 ms
IPv4 Addresses 1
.................................. 192.168.202.100
IPv6 Addresses 0
Keepalived example:
# ip addr
ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:ba:19:ee brd ff:ff:ff:ff:ff:ff
inet XXX.XXX.32.2/28 brd XXX.XXX.32.15 scope global ens192
valid_lft forever preferred_lft forever
inet XXX.XXX.32.1/28 brd XXX.XXX.32.15 scope global secondary ens192
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:feba:19ee/64 scope link
valid_lft forever preferred_lft forever
# ifconfig
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet XXX.XXX.32.2 netmask 255.255.255.240 broadcast XXX.XXX.32.15
inet6 fe80::250:56ff:feba:19ee prefixlen 64 scopeid 0x20<link>
ether 00:50:56:ba:19:ee txqueuelen 1000 (Ethernet)
RX packets 727051638 bytes 408802970966 (408.8 GB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1017453444 bytes 1021439960799 (1.0 TB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
VRRP packets leave the VM with the same MAC (00:50:56:ba:19:ee) and work correctly with VMware default Security Policy
# tcpdump -i ens192 -ne vrrp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
14:48:43.210374 00:50:56:ba:19:ee > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: XXX.XXX.32.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 150, authtype simple, intvl 1s, length 20
14:48:44.211446 00:50:56:ba:19:ee > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: XXX.XXX.32.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 150, authtype simple, intvl 1s, length 20
14:48:45.212528 00:50:56:ba:19:ee > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: XXX.XXX.32.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 150, authtype simple, intvl 1s, length 20
Looking at your post, there's several problems with your configuration.
First, your kernel:
root@test1:~# uname -a
Linux test1 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 10:36:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
This kernel does not support FRR VRRP. This is mentioned in the docs:
The reason 5.1 is the minimum required kernel version is because that version adds support for protodown on macvlan devices. The reason we need to protodown macvlan devices is because that's how we change mastership status without adding and removing IP addresses from interfaces. You can read more about that design choice in the documentation.
That same documentation link also explains your particular problem. The kernel drops traffic with an ethernet source address equal to one it owns on an interface - unless the MAC belongs to a macvlan device slaved to the interface, and said macvlan is in private or bridge mode. So, assuming your macvlans are in one of those modes, then we come to the idiosyncratic macvlan driver behavior, also discussed in the FRR docs linked above as well as the Keepalived docs:
By default MACVLAN interface are in VEPA mode which filters out received packets whose MAC source address matches that of the MACVLAN interface. Setting MACVLAN interface in private mode will not filter based on source MAC address.
(This is also true for bridge mode).
Along with a link to the relevant kernel patch:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=729e72a10930ef765c11a5a35031ba47f18221c4
Because we use protodown as the failover mechanism we need to Rx VRRP traffic on the base interface. The above kernel patch causes ingress traffic with a source mac equal to a private mode macvlan to be placed only on that macvlan. So this is why your macvlans have to be set to bridge mode to use them with FRR VRRP. I can't see whether they are or not - you can check this yourself with ip -d link - but I'm betting they aren't, based on the fact that you also haven't set addrgenmode random (as specified in the docs) and are instead using EUI64 for v6 link local derivation on your macvlans:
3: vrrp4-2-1@ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:00:5e:00:01:01 brd ff:ff:ff:ff:ff:ff
inet 192.168.202.100/24 scope global vrrp4-2-1
valid_lft forever preferred_lft forever
>>>> inet6 fe80::200:5eff:fe00:101/64 scope link
If you ever decide to backup a v6 address, you'll hit a whole new set of confusing behavior because the link locals are used as the source address for VRRP advertisements, and EUI64 will produce the same link locals on both of your virtual routers.
Anyhow, if your macvlans aren't set to bridge mode, that is why your routers aren't seeing each others VRRP traffic without promiscuous mode turned on. Try using the instructions given in the docs for creating VRRP macvlans and check if you start seeing traffic. Of course, without the correct kernel version, master-backup failover still won't work, because protodown isn't implemented in 4.14.
As for what Keepalived is doing in your working setup - in your case, you are using Keepalived in its non-RFC compliant mode of operation, that is, without a VMAC. Unique mac addresses are not discussed in the VRRP RFC. Keepalived does this itself, as an optional change to the protocol, to work around exactly the problems discussed above. This comes at the cost of downstream neighbor table churn in a failover scenario, which the RFC tries to avoid. FRR implements the RFC - nothing more. If you can't upgrade your kernel you'll need to continue using Keepalived. Here is the relevant RFC:
https://tools.ietf.org/html/rfc5798
Hopefully I've answered your questions now. If not, let me know and I'll try to clarify further.
qlyoung
on 26 Nov 2019
Thank you for the important information and time dedicated to this case.
Sorry if I missed the kernel version before opening this case. I assumed it was correct.
I am away from the office to work in another location and country until Sunday.
I will be able to do a new test with a kernel upgrade just next Monday.
I don't know if you can keep the case open for a little time...
Regards
smarcosmarco
on 27 Nov 2019
Sure can, let me know how it goes.
qlyoung
on 27 Nov 2019
Hello!
I've updated my test VMs with the last kernel
root@test1:~# uname -a
Linux test1 5.4.1-050401-generic #201911290555 SMP Fri Nov 29 11:03:47 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
root@test1:~# ip -d link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:50:56:ba:31:92 brd ff:ff:ff:ff:ff:ff promiscuity 1 addrgenmode eui64 numtxqueues 2 numrxqueues 2 gso_max_size 65536 gso_max_segs 65535
3: vrrp4-2-1@ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 00:00:5e:00:01:01 brd ff:ff:ff:ff:ff:ff promiscuity 0
macvlan mode bridge addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
-----
root@test2:~# uname -a
Linux test2 5.4.1-050401-generic #201911290555 SMP Fri Nov 29 11:03:47 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
root@test2:~# ip -d link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:50:56:ba:6c:0e brd ff:ff:ff:ff:ff:ff promiscuity 1 addrgenmode eui64 numtxqueues 2 numrxqueues 2 gso_max_size 65536 gso_max_segs 65535
3: vrrp4-2-1@ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 00:00:5e:00:01:01 brd ff:ff:ff:ff:ff:ff promiscuity 0
macvlan mode bridge addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
But I've the same result...
About the "bridge" mode , is this correct?
ip link add vrrp4-2-1 link ens160 addrgenmode random type macvlan mode bridge
ip link set dev vrrp4-2-1 address 00:00:5e:00:01:01
ip addr add 192.168.202.100/24 dev vrrp4-2-1
ip link set dev vrrp4-2-1 up
Your interface configuration looks correct to me. What is show vrrp indicating, and are you seeing VRRP traffic inbound on ens160 with promiscuous mode off (you've got it turned on in your example there)?
tcpdump --no-promisc -i ens160 --direction in 'vrrp'
If you turn on debug vrrp what are you seeing in the logs?
qlyoung
on 2 Dec 2019
VMware Policy Settings:
Security
----
Promiscuous mode | Reject
MAC address changes | Reject
Forged transmits | Reject
VM TEST1:
root@test1:~# tcpdump --no-promisc -i ens160 --direction in 'vrrp'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
59 packets received by filter
0 packets dropped by kernel
-----
test1# show vrrp
Virtual Router ID 1
Protocol Version 3
Autoconfigured No
Shutdown No
Interface ens160
VRRP interface (v4) vrrp4-2-1
VRRP interface (v6) None
Primary IP (v4) 192.168.202.101
Primary IP (v6) ::
Virtual MAC (v4) 00:00:5e:00:01:01
Virtual MAC (v6) 00:00:5e:00:02:01
Status (v4) Master
Status (v6) Initialize
Priority 200
Effective Priority (v4) 200
Effective Priority (v6) 200
Preempt Mode Yes
Accept Mode Yes
Advertisement Interval 100 ms
Master Advertisement Interval (v4) 100 ms
Master Advertisement Interval (v6) 0 ms
Advertisements Tx (v4) 2533
Advertisements Tx (v6) 0
Advertisements Rx (v4) 0
Advertisements Rx (v6) 0
Gratuitous ARP Tx (v4) 1
Neigh. Adverts Tx (v6) 0
State transitions (v4) 2
State transitions (v6) 0
Skew Time (v4) 20 ms
Skew Time (v6) 0 ms
Master Down Interval (v4) 320 ms
Master Down Interval (v6) 0 ms
IPv4 Addresses 1
.................................. 192.168.202.100
IPv6 Addresses 0
VM TEST2:
root@test2:~# tcpdump --no-promisc -i ens160 --direction in 'vrrp'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
48 packets received by filter
0 packets dropped by kernel
-----
test2# sh vrrp
Virtual Router ID 1
Protocol Version 3
Autoconfigured No
Shutdown No
Interface ens160
VRRP interface (v4) vrrp4-2-1
VRRP interface (v6) None
Primary IP (v4) 192.168.202.102
Primary IP (v6) ::
Virtual MAC (v4) 00:00:5e:00:01:01
Virtual MAC (v6) 00:00:5e:00:02:01
Status (v4) Master
Status (v6) Initialize
Priority 100
Effective Priority (v4) 100
Effective Priority (v6) 100
Preempt Mode Yes
Accept Mode Yes
Advertisement Interval 100 ms
Master Advertisement Interval (v4) 100 ms
Master Advertisement Interval (v6) 0 ms
Advertisements Tx (v4) 1262
Advertisements Tx (v6) 0
Advertisements Rx (v4) 0
Advertisements Rx (v6) 0
Gratuitous ARP Tx (v4) 1
Neigh. Adverts Tx (v6) 0
State transitions (v4) 2
State transitions (v6) 0
Skew Time (v4) 60 ms
Skew Time (v6) 0 ms
Master Down Interval (v4) 360 ms
Master Down Interval (v6) 0 ms
IPv4 Addresses 1
.................................. 192.168.202.100
IPv6 Addresses 0
VM1 CONFIG VRRP:
Current configuration:
!
frr version 7.2
frr defaults traditional
hostname test1
log file /var/log/frr/vrrp.log
log syslog
service integrated-vtysh-config
!
debug vrrp arp
debug vrrp autoconfigure
debug vrrp ndisc
debug vrrp packets
debug vrrp protocol
debug vrrp sockets
debug vrrp zebra
!
interface ens160
vrrp 1
vrrp 1 advertisement-interval 100
vrrp 1 priority 200
vrrp 1 ip 192.168.202.100
!
line vty
!
end
VM1 DEBUG VRRP LOG:
2019/12/03 08:55:55 ZEBRA: client 22 says hello and bids fair to announce only static routes vrf=0
2019/12/03 08:55:55 ZEBRA: client 28 says hello and bids fair to announce only vrrp routes vrf=0
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] [IPv4] Interface: vrrp4-2-1
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] [IPv6] Interface: None (no interface found w/ MAC 00:00:5e:00:02:01)
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] [IPv4] 'Startup' event
2019/12/03 08:55:55 VRRP: [ARP] Initialized gratuitous ARP socket
2019/12/03 08:55:55 VRRP: [ARP] Initialized gratuitous ARP subsystem
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] [IPv4] Bound Rx socket to ens160
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] [IPv4] Bound Rx socket to VRRP multicast group
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] [IPv4] Joined VRRP multicast group
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] [IPv4] Set vrrp4-2-1 as outgoing multicast interface
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] [IPv4] Bound Tx socket to primary IP address 192.168.202.101
2019/12/03 08:55:55 VRRP: [ZEBRA] Requesting Zebra to set vrrp4-2-1 protodown on
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] [IPv4] Initialize -> Backup
2019/12/03 08:55:55 ZEBRA: Setting interface vrrp4-2-1 (3): protodown on
2019/12/03 08:55:55 VRRP: vrrp_zebra_if_state_down: vrrp4-2-1 index 0(2) parent 3 mac 00:00:5e:00:01:01 flags 4099 metric 0 mtu 1500 operative 0
2019/12/03 08:55:55 VRRP: [CORE] Searching for instances that could use interface vrrp4-2-1
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] <-- This instance can probably use interface vrrp4-2-1
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] Interface vrrp4-2-1 down
2019/12/03 08:55:55 VRRP: vrrp_zebra_if_address_del: vrrp4-2-1 index 0(2) parent 3 mac 00:00:5e:00:01:01 flags 4099 metric 0 mtu 1500 operative 0
2019/12/03 08:55:55 VRRP: vrrp_zebra_if_address_del: interface vrrp4-2-1 addresses:
2019/12/03 08:55:55 VRRP: vrrp_zebra_if_address_del: interface vrrp4-2-1 address 254.128.0.0 primary
2019/12/03 08:55:55 VRRP: vrrp_zebra_if_address_del: vrrp4-2-1 index 0(2) parent 3 mac 00:00:5e:00:01:01 flags 4099 metric 0 mtu 1500 operative 0
2019/12/03 08:55:55 VRRP: vrrp_zebra_if_address_del: interface vrrp4-2-1 addresses:
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] [IPv4] Master_Down_Timer expired
2019/12/03 08:55:55 VRRP: [ZEBRA] Requesting Zebra to set vrrp4-2-1 protodown off
2019/12/03 08:55:55 ZEBRA: Setting interface vrrp4-2-1 (3): protodown off
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] [IPv4] Delaying VRRP advertisement until interface is up
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] [IPv4] Delaying VRRP gratuitous ARPs until interface is up
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] [IPv4] Backup -> Master
2019/12/03 08:55:55 VRRP: vrrp_zebra_if_state_down: vrrp4-2-1 index 0(2) parent 3 mac 00:00:5e:00:01:01 flags 69635 metric 0 mtu 1500 operative 0
2019/12/03 08:55:55 VRRP: [CORE] Searching for instances that could use interface vrrp4-2-1
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] <-- This instance can probably use interface vrrp4-2-1
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] Interface vrrp4-2-1 down
2019/12/03 08:55:55 VRRP: vrrp_zebra_if_state_up: vrrp4-2-1 index 0(2) parent 3 mac 00:00:5e:00:01:01 flags 69699 metric 0 mtu 1500 operative 1
2019/12/03 08:55:55 VRRP: [CORE] Searching for instances that could use interface vrrp4-2-1
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] <-- This instance can probably use interface vrrp4-2-1
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] [IPv4] Interface up; sending pending advertisement
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] [IPv4] Bound Tx socket to primary IP address 192.168.202.101
2019/12/03 08:55:55 VRRP:
0x00005571c47c4270: 31 01 c8 01 00 0a 10 48 1......H
0x00005571c47c4278: c0 a8 ca 64 ...d
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] [IPv4] Interface up; sending pending gratuitous ARP
2019/12/03 08:55:55 VRRP: [ARP] [VRID 1] [IPv4] Sending gratuitous ARP on vrrp4-2-1 for 192.168.202.100
2019/12/03 08:55:55 VRRP:
0x00007ffd6c8e26c0: ff ff ff ff ff ff 00 00 ........
0x00007ffd6c8e26c8: 5e 00 01 01 08 06 00 01 ^.......
0x00007ffd6c8e26d0: 08 00 06 04 00 01 00 00 ........
0x00007ffd6c8e26d8: 5e 00 01 01 c0 a8 ca 64 ^......d
0x00007ffd6c8e26e0: ff ff ff ff ff ff c0 a8 ........
0x00007ffd6c8e26e8: ca 64 .d
2019/12/03 08:55:55 VRRP: vrrp_zebra_if_address_add: vrrp4-2-1 index 0(2) parent 3 mac 00:00:5e:00:01:01 flags 69699 metric 0 mtu 1500 operative 1
2019/12/03 08:55:55 VRRP: vrrp_zebra_if_address_add: interface vrrp4-2-1 addresses:
2019/12/03 08:55:55 VRRP: vrrp_zebra_if_address_add: interface vrrp4-2-1 address 192.168.202.100 primary
2019/12/03 08:55:55 VRRP: vrrp_zebra_if_address_add: vrrp4-2-1 index 0(2) parent 3 mac 00:00:5e:00:01:01 flags 69699 metric 0 mtu 1500 operative 1
2019/12/03 08:55:55 VRRP: vrrp_zebra_if_address_add: interface vrrp4-2-1 addresses:
2019/12/03 08:55:55 VRRP: vrrp_zebra_if_address_add: interface vrrp4-2-1 address 192.168.202.100 primary
2019/12/03 08:55:55 VRRP: vrrp_zebra_if_address_add: interface vrrp4-2-1 address 254.128.0.0 primary
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] [IPv4] Adver_Timer expired
2019/12/03 08:55:55 VRRP:
0x00005571c47c28f0: 31 01 c8 01 00 0a 10 48 1......H
0x00005571c47c28f8: c0 a8 ca 64 ...d
[..]
2019/12/03 08:55:55 VRRP: [CORE] [VRID 1] [IPv4] Adver_Timer expired
2019/12/03 08:55:55 VRRP:
0x00005571c47c4270: 31 01 c8 01 00 0a 10 48 1......H
0x00005571c47c4278: c0 a8 ca 64 ...d
VM2
2019/12/03 09:00:24 ZEBRA: client 15 says hello and bids fair to announce only static routes vrf=0
2019/12/03 09:00:24 ZEBRA: client 28 says hello and bids fair to announce only vrrp routes vrf=0
2019/12/03 09:00:24 VRRP: [CORE] [VRID 1] [IPv4] Interface: vrrp4-2-1
2019/12/03 09:00:24 VRRP: [CORE] [VRID 1] [IPv6] Interface: None (no interface found w/ MAC 00:00:5e:00:02:01)
2019/12/03 09:00:24 VRRP: [CORE] [VRID 1] [IPv4] 'Startup' event
2019/12/03 09:00:24 VRRP: [ARP] Initialized gratuitous ARP socket
2019/12/03 09:00:24 VRRP: [ARP] Initialized gratuitous ARP subsystem
2019/12/03 09:00:24 VRRP: [CORE] [VRID 1] [IPv4] Bound Rx socket to ens160
2019/12/03 09:00:24 VRRP: [CORE] [VRID 1] [IPv4] Bound Rx socket to VRRP multicast group
2019/12/03 09:00:24 VRRP: [CORE] [VRID 1] [IPv4] Joined VRRP multicast group
2019/12/03 09:00:24 VRRP: [CORE] [VRID 1] [IPv4] Set vrrp4-2-1 as outgoing multicast interface
2019/12/03 09:00:24 VRRP: [CORE] [VRID 1] [IPv4] Bound Tx socket to primary IP address 192.168.202.102
2019/12/03 09:00:24 VRRP: [ZEBRA] Requesting Zebra to set vrrp4-2-1 protodown on
2019/12/03 09:00:24 VRRP: [CORE] [VRID 1] [IPv4] Initialize -> Backup
2019/12/03 09:00:24 ZEBRA: Setting interface vrrp4-2-1 (3): protodown on
2019/12/03 09:00:24 VRRP: vrrp_zebra_if_state_down: vrrp4-2-1 index 0(2) parent 3 mac 00:00:5e:00:01:01 flags 4099 metric 0 mtu 1500 operative 0
2019/12/03 09:00:24 VRRP: [CORE] Searching for instances that could use interface vrrp4-2-1
2019/12/03 09:00:24 VRRP: [CORE] [VRID 1] <-- This instance can probably use interface vrrp4-2-1
2019/12/03 09:00:24 VRRP: [CORE] [VRID 1] Interface vrrp4-2-1 down
2019/12/03 09:00:24 VRRP: vrrp_zebra_if_address_del: vrrp4-2-1 index 0(2) parent 3 mac 00:00:5e:00:01:01 flags 4099 metric 0 mtu 1500 operative 0
2019/12/03 09:00:24 VRRP: vrrp_zebra_if_address_del: interface vrrp4-2-1 addresses:
2019/12/03 09:00:24 VRRP: vrrp_zebra_if_address_del: interface vrrp4-2-1 address 254.128.0.0 primary
2019/12/03 09:00:24 VRRP: vrrp_zebra_if_address_del: vrrp4-2-1 index 0(2) parent 3 mac 00:00:5e:00:01:01 flags 4099 metric 0 mtu 1500 operative 0
2019/12/03 09:00:24 VRRP: vrrp_zebra_if_address_del: interface vrrp4-2-1 addresses:
2019/12/03 09:00:25 VRRP: [CORE] [VRID 1] [IPv4] Master_Down_Timer expired
2019/12/03 09:00:25 VRRP: [ZEBRA] Requesting Zebra to set vrrp4-2-1 protodown off
2019/12/03 09:00:25 VRRP: [CORE] [VRID 1] [IPv4] Delaying VRRP advertisement until interface is up
2019/12/03 09:00:25 VRRP: [CORE] [VRID 1] [IPv4] Delaying VRRP gratuitous ARPs until interface is up
2019/12/03 09:00:25 VRRP: [CORE] [VRID 1] [IPv4] Backup -> Master
2019/12/03 09:00:25 ZEBRA: Setting interface vrrp4-2-1 (3): protodown off
2019/12/03 09:00:25 VRRP: vrrp_zebra_if_state_down: vrrp4-2-1 index 0(2) parent 3 mac 00:00:5e:00:01:01 flags 69635 metric 0 mtu 1500 operative 0
2019/12/03 09:00:25 VRRP: [CORE] Searching for instances that could use interface vrrp4-2-1
2019/12/03 09:00:25 VRRP: [CORE] [VRID 1] <-- This instance can probably use interface vrrp4-2-1
2019/12/03 09:00:25 VRRP: [CORE] [VRID 1] Interface vrrp4-2-1 down
2019/12/03 09:00:25 VRRP: vrrp_zebra_if_state_up: vrrp4-2-1 index 0(2) parent 3 mac 00:00:5e:00:01:01 flags 69699 metric 0 mtu 1500 operative 1
2019/12/03 09:00:25 VRRP: [CORE] Searching for instances that could use interface vrrp4-2-1
2019/12/03 09:00:25 VRRP: [CORE] [VRID 1] <-- This instance can probably use interface vrrp4-2-1
2019/12/03 09:00:25 VRRP: [CORE] [VRID 1] [IPv4] Interface up; sending pending advertisement
2019/12/03 09:00:25 VRRP: [CORE] [VRID 1] [IPv4] Bound Tx socket to primary IP address 192.168.202.102
2019/12/03 09:00:25 VRRP:
0x000055e5fea7bdf0: 31 01 64 01 00 0a 74 47 1.d...tG
0x000055e5fea7bdf8: c0 a8 ca 64 ...d
2019/12/03 09:00:25 VRRP: [CORE] [VRID 1] [IPv4] Interface up; sending pending gratuitous ARP
2019/12/03 09:00:25 VRRP: [ARP] [VRID 1] [IPv4] Sending gratuitous ARP on vrrp4-2-1 for 192.168.202.100
2019/12/03 09:00:25 VRRP:
0x00007ffd356e9bd0: ff ff ff ff ff ff 00 00 ........
0x00007ffd356e9bd8: 5e 00 01 01 08 06 00 01 ^.......
0x00007ffd356e9be0: 08 00 06 04 00 01 00 00 ........
0x00007ffd356e9be8: 5e 00 01 01 c0 a8 ca 64 ^......d
0x00007ffd356e9bf0: ff ff ff ff ff ff c0 a8 ........
0x00007ffd356e9bf8: ca 64 .d
2019/12/03 09:00:25 VRRP: vrrp_zebra_if_address_add: vrrp4-2-1 index 0(2) parent 3 mac 00:00:5e:00:01:01 flags 69699 metric 0 mtu 1500 operative 1
2019/12/03 09:00:25 VRRP: vrrp_zebra_if_address_add: interface vrrp4-2-1 addresses:
2019/12/03 09:00:25 VRRP: vrrp_zebra_if_address_add: interface vrrp4-2-1 address 192.168.202.100 primary
2019/12/03 09:00:25 VRRP: vrrp_zebra_if_address_add: vrrp4-2-1 index 0(2) parent 3 mac 00:00:5e:00:01:01 flags 69699 metric 0 mtu 1500 operative 1
2019/12/03 09:00:25 VRRP: vrrp_zebra_if_address_add: interface vrrp4-2-1 addresses:
2019/12/03 09:00:25 VRRP: vrrp_zebra_if_address_add: interface vrrp4-2-1 address 192.168.202.100 primary
2019/12/03 09:00:25 VRRP: vrrp_zebra_if_address_add: interface vrrp4-2-1 address 254.128.0.0 primary
2019/12/03 09:00:25 VRRP: [CORE] [VRID 1] [IPv4] Adver_Timer expired
2019/12/03 09:00:25 VRRP:
0x000055e5fea81200: 31 01 64 01 00 0a 74 47 1.d...tG
0x000055e5fea81208: c0 a8 ca 64 ...d
[..]
2019/12/03 09:00:25 VRRP: [CORE] [VRID 1] [IPv4] Adver_Timer expired
2019/12/03 09:00:25 VRRP:
0x000055e5fea81200: 31 01 64 01 00 0a 74 47 1.d...tG
0x000055e5fea81208: c0 a8 ca 64 ...d
Changing VMware Policy Settings to:
Security
----
Promiscuous mode | Accept
MAC address changes | Accept
Forged transmits | Accept
..this is the log for VM2:
2019/12/03 09:03:09 VRRP: [CORE] [VRID 1] [IPv4] Adver_Timer expired
2019/12/03 09:03:09 VRRP:
0x000055e5fea81170: 31 01 64 01 00 0a 74 47 1.d...tG
0x000055e5fea81178: c0 a8 ca 64 ...d
2019/12/03 09:03:09 VRRP: [CORE] [VRID 1] [IPv4] Adver_Timer expired
2019/12/03 09:03:09 VRRP:
0x000055e5fea81170: 31 01 64 01 00 0a 74 47 1.d...tG
0x000055e5fea81178: c0 a8 ca 64 ...d
2019/12/03 09:03:09 VRRP: [CORE] [VRID 1] [IPv4] Adver_Timer expired
2019/12/03 09:03:09 VRRP:
0x000055e5fea81170: 31 01 64 01 00 0a 74 47 1.d...tG
0x000055e5fea81178: c0 a8 ca 64 ...d
2019/12/03 09:03:09 VRRP: [CORE] [VRID 1] [IPv4] Datagram rx:
2019/12/03 09:03:09 VRRP:
0x000055e5fea81ddc: 45 c0 00 20 00 45 40 00 E.. .E@.
0x000055e5fea81de4: ff 70 0f 48 c0 a8 ca 65 .p.H...e
0x000055e5fea81dec: e0 00 00 12 31 01 c8 01 ....1...
0x000055e5fea81df4: 00 0a 10 48 c0 a8 ca 64 ...H...d
2019/12/03 09:03:09 VRRP: [CORE] [VRID 1] [IPv4] Received VRRP Advertisement from 192.168.202.101:
version 3, type 1 (ADVERTISEMENT), vrid 1, priority 200, #1 addresses, max adver int 10, checksum 1048
2019/12/03 09:03:09 VRRP: [CORE] [VRID 1] [IPv4] Received advertisement from 192.168.202.101 w/ priority 200; switching to Backup
2019/12/03 09:03:09 VRRP: [ZEBRA] Requesting Zebra to set vrrp4-2-1 protodown on
2019/12/03 09:03:09 ZEBRA: Setting interface vrrp4-2-1 (3): protodown on
2019/12/03 09:03:09 VRRP: [CORE] [VRID 1] [IPv4] Master -> Backup
2019/12/03 09:03:09 VRRP: vrrp_zebra_if_state_up: vrrp4-2-1 index 0(2) parent 3 mac 00:00:5e:00:01:01 flags 4163 metric 0 mtu 1500 operative 1
2019/12/03 09:03:09 VRRP: [CORE] Searching for instances that could use interface vrrp4-2-1
2019/12/03 09:03:09 VRRP: [CORE] [VRID 1] <-- This instance can probably use interface vrrp4-2-1
2019/12/03 09:03:09 VRRP: vrrp_zebra_if_state_down: vrrp4-2-1 index 0(2) parent 3 mac 00:00:5e:00:01:01 flags 4099 metric 0 mtu 1500 operative 0
2019/12/03 09:03:09 VRRP: [CORE] Searching for instances that could use interface vrrp4-2-1
2019/12/03 09:03:09 VRRP: [CORE] [VRID 1] <-- This instance can probably use interface vrrp4-2-1
2019/12/03 09:03:09 VRRP: [CORE] [VRID 1] Interface vrrp4-2-1 down
2019/12/03 09:03:09 VRRP: vrrp_zebra_if_address_del: vrrp4-2-1 index 0(2) parent 3 mac 00:00:5e:00:01:01 flags 4099 metric 0 mtu 1500 operative 0
2019/12/03 09:03:09 VRRP: vrrp_zebra_if_address_del: interface vrrp4-2-1 addresses:
2019/12/03 09:03:09 VRRP: vrrp_zebra_if_address_del: interface vrrp4-2-1 address 254.128.0.0 primary
2019/12/03 09:03:09 VRRP: vrrp_zebra_if_address_del: vrrp4-2-1 index 0(2) parent 3 mac 00:00:5e:00:01:01 flags 4099 metric 0 mtu 1500 operative 0
2019/12/03 09:03:09 VRRP: vrrp_zebra_if_address_del: interface vrrp4-2-1 addresses:
2019/12/03 09:03:09 VRRP: [CORE] [VRID 1] [IPv4] Datagram rx:
2019/12/03 09:03:09 VRRP:
0x000055e5fea81ddc: 45 c0 00 20 00 5e 40 00 E.. .^@.
0x000055e5fea81de4: ff 70 0f 2f c0 a8 ca 65 .p./...e
0x000055e5fea81dec: e0 00 00 12 31 01 c8 01 ....1...
0x000055e5fea81df4: 00 0a 10 48 c0 a8 ca 64 ...H...d
2019/12/03 09:03:09 VRRP: [CORE] [VRID 1] [IPv4] Received VRRP Advertisement from 192.168.202.101:
version 3, type 1 (ADVERTISEMENT), vrid 1, priority 200, #1 addresses, max adver int 10, checksum 1048
2019/12/03 09:03:09 VRRP: [CORE] [VRID 1] [IPv4] Datagram rx:
2019/12/03 09:03:09 VRRP:
0x000055e5fea81ddc: 45 c0 00 20 00 77 40 00 E.. .w@.
0x000055e5fea81de4: ff 70 0f 16 c0 a8 ca 65 .p.....e
0x000055e5fea81dec: e0 00 00 12 31 01 c8 01 ....1...
0x000055e5fea81df4: 00 0a 10 48 c0 a8 ca 64 ...H...d
2019/12/03 09:03:09 VRRP: [CORE] [VRID 1] [IPv4] Received VRRP Advertisement from 192.168.202.101:
version 3, type 1 (ADVERTISEMENT), vrid 1, priority 200, #1 addresses, max adver int 10, checksum 1048
2019/12/03 09:03:09 VRRP: [CORE] [VRID 1] [IPv4] Datagram rx:
2019/12/03 09:03:09 VRRP:
0x000055e5fea81ddc: 45 c0 00 20 00 7e 40 00 E.. .~@.
0x000055e5fea81de4: ff 70 0f 0f c0 a8 ca 65 .p.....e
0x000055e5fea81dec: e0 00 00 12 31 01 c8 01 ....1...
0x000055e5fea81df4: 00 0a 10 48 c0 a8 ca 64 ...H...d
2019/12/03 09:03:09 VRRP: [CORE] [VRID 1] [IPv4] Received VRRP Advertisement from 192.168.202.101:
version 3, type 1 (ADVERTISEMENT), vrid 1, priority 200, #1 addresses, max adver int 10, checksum 1048
2019/12/03 09:03:09 VRRP: [CORE] [VRID 1] [IPv4] Datagram rx:
2019/12/03 09:03:09 VRRP:
0x000055e5fea81ddc: 45 c0 00 20 00 84 40 00 E.. ..@.
0x000055e5fea81de4: ff 70 0f 09 c0 a8 ca 65 .p.....e
0x000055e5fea81dec: e0 00 00 12 31 01 c8 01 ....1...
0x000055e5fea81df4: 00 0a 10 48 c0 a8 ca 64 ...H...d
2019/12/03 09:03:09 VRRP: [CORE] [VRID 1] [IPv4] Received VRRP Advertisement from 192.168.202.101:
version 3, type 1 (ADVERTISEMENT), vrid 1, priority 200, #1 addresses, max adver int 10, checksum 1048
2019/12/03 09:03:09 VRRP: [CORE] [VRID 1] [IPv4] Datagram rx:
2019/12/03 09:03:09 VRRP:
0x000055e5fea81ddc: 45 c0 00 20 00 8c 40 00 E.. ..@.
0x000055e5fea81de4: ff 70 0f 01 c0 a8 ca 65 .p.....e
0x000055e5fea81dec: e0 00 00 12 31 01 c8 01 ....1...
0x000055e5fea81df4: 00 0a 10 48 c0 a8 ca 64 ...H...d
Note: the problem occurs even if the 2 VMs run on the same VMware ESXi host
smarcosmarco
on 3 Dec 2019
Sounds like a VMware issue to me, since your kernel & FRR configuration is correct at this point. I'm not familiar with whatever VMware product this is (ESXi?), but from a quick google:
Forget Transmits – affects traffic that is transmitted from a virtual machine. When set to Reject, the virtual NIC drops frames that the guest operating system sends if the source MAC address is different than the one assigned to the virtual NIC. Set to Accept by default.
Wouldn't this drop outbound VRRP traffic? Maybe you can try turning on each of the 3 policies you mentioned, one at a time, and report back what you find. Then we can add it to the docs. It might be the case that you only need Forged Transmits turned on.
In any case I think we agree this is not an FRR bug at this point.
qlyoung
on 3 Dec 2019
I had already done these tests. The only way to make VRRP work with FRR is to enable all 3 options.
Sure, this is not a FRR bug, but since it is a product for enterprise use it can be said that it is not efficient and usable in virtual environment.
You cannot imagine the impact in terms of security and performance on the infrastructure enabling these 3 options.
I will be forced to continue using Keepalived or other tools that work properly in multiple professional environments.
smarcosmarco
on 4 Dec 2019
I guess that behavior makes sense from a virtualization security standpoint in that ESXi wouldn't allow guests to propagate virtual MACs up to the hypervisor VNIC, since that's effectively allowing promiscuous mode to be configured from within the guest, which I suppose is a security risk in some scenarios if you planned to run untrusted software in the guest. So I guess what we can conclude here is that you can't run any RFC compliant VRRP implementation from within an ESXi guest without promiscuous mode.
At the same time, I feel like you shouldn't be running untrusted software on a box that you're using as a gateway router, so in this case I don't really see the issue with enabling promiscuous mode for your VRRP VM. Purely for my own education, I'd be interested to know why that's not a good idea in your setup.
Edit: someone also pointed out to me that this is relevant for container networking, which also uses macvlan devices, e.g.:
https://forums.docker.com/t/why-is-promiscuous-mode-needed-for-macvlan-driver/37416
So unfortunately it does seem like VRRP implementations that break rfc and use the virtual nic mac address are your only option here. Perhaps this is something we can revisit in the future. I appreciate your responsiveness on this issue.
qlyoung
on 4 Dec 2019
Actually, gonna reopen this to remind myself to put a note in the docs about this scenario.
qlyoung
on 4 Dec 2019
@smarcosmarco Someone on the FRR slack informed me that ESXi 6.7 has a new mac learning feature that you can use in lieu of promiscuous mode. If you can upgrade that should solve your problem.
https://www.virtuallyghetto.com/2018/04/native-mac-learning-in-vsphere-6-7-removes-the-need-for-promiscuous-mode-for-nested-esxi.html
qlyoung
on 5 Dec 2019
I already have ESXi version 6.7. Let me look at the document carefully and I'll let you know.
Thanks
smarcosmarco
on 9 Dec 2019
@qlyoung ...hit in full ;-)
PS C:\WINDOWS\system32> Get-MacLearn -DVPortgroupName @("dpgTEST")
DVPortgroup : dpgTEST
MacLearning : True
NewAllowPromiscuous : False
NewForgedTransmits : True
NewMacChanges : False
Limit : 4096
LimitPolicy : DROP
LegacyAllowPromiscuous : False
LegacyForgedTransmits : False
LegacyMacChanges : False
[..]
16:01:02.710385 IP 192.168.202.101 > 224.0.0.18: VRRPv3, Advertisement, vrid 1, prio 250, intvl 10cs, length 12
16:01:02.810578 IP 192.168.202.101 > 224.0.0.18: VRRPv3, Advertisement, vrid 1, prio 250, intvl 10cs, length 12
16:01:02.910748 IP 192.168.202.101 > 224.0.0.18: VRRPv3, Advertisement, vrid 1, prio 1, intvl 10cs, length 12
16:01:03.010949 IP 192.168.202.101 > 224.0.0.18: VRRPv3, Advertisement, vrid 1, prio 1, intvl 10cs, length 12
16:01:03.111160 IP 192.168.202.101 > 224.0.0.18: VRRPv3, Advertisement, vrid 1, prio 1, intvl 10cs, length 12
16:01:03.132852 IP 192.168.202.102 > 224.0.0.18: VRRPv3, Advertisement, vrid 1, prio 200, intvl 10cs, length 12
16:01:03.231587 IP 192.168.202.102 > 224.0.0.18: VRRPv3, Advertisement, vrid 1, prio 200, intvl 10cs, length 12
16:01:03.331736 IP 192.168.202.102 > 224.0.0.18: VRRPv3, Advertisement, vrid 1, prio 200, intvl 10cs, length 12
[..]
...you need to update the documentation about this scenario and solution.
Regards!
smarcosmarco
on 10 Dec 2019
@smarcosmarco yes, I intended to update the docs. Although seeing as I've never used ESXi, I might not be the best person - and remember everyone can contribute to FRR! ;-)
qlyoung
on 10 Dec 2019
Related issues
hansengao
·
5Comments
thilse
·
7Comments
jpds
·
4Comments
blakedot
·
5Comments
Chen0Yang
·
3Comments