Frr: Zebra not refreshing rib on linux address deletion
Hello
First, I'd like to thank you for this awesome project !
Then, I'm opening this issue cause I think I faced a bug.
I know the environment is a little exotic but I hope we'll determine the root cause :-)
(Sorry in advance if the issue is not perfectly formatted, it's my first one)
Things you may try first
(put "x" in "[ ]" if you already tried following)
- [X] Did you check if this is a duplicate issue?
- [X] Did you test it on the latest FRRouting/frr master branch?
Description
(Attachements are at the end)
I have a linux interface (type "dummy") hosting some /32 IP addresses.
Addresses are added/removed automatically by Kube-proxy (Kubernetes).
Internally, they use the Go Netling Library (https://github.com/vishvananda/netlink).
When new addresses are added to this interface, no problem, I see those new addresses in FRR as /32 connected routes (by using sh ip route connected).
However, when addresses are removed from the linux interface, FRR still sees the /32 connected routes. On the linux side everything looks good (the removed addresses are removed from everywhere and the associated routes are also removed)
I tried to down/up the interface.
At down, FRR doesn't see the routes.
At up, FRR sees the routes again, and are even duplicated (new ones with '>', the old ones that shouldn't have reappeared without '>').
I also tried to reload FRR. No luck.
The only way to refresh the routing table of FRR is to Stop and then Start FRR (via systemctl).
(IOS-like clear ip route would be interesting to have for those kind of issues by the way ;-) )
I checked the debug logs (thanks to @donaldsharp ) and Zebra seems to be properly notified of the deletion but it doesn't take the action of deleting the route.
Interestingly :
- If I add again the removed address via Kube-proxy, a new route is added on FRR (and the old ones are NOT removed so there are multiple identical /32 connected routes)
- If I add again the removed address manually, FRR take the action of deleting the old one and add the new one.
Steps to Reproduce
- Add a service in Kubernetes with Kube-proxy in IPVS mode which adds a /32 address on a dummy interface.
- Check on FRR that those /32 are seen as connected /32 routes.
- Delete the service which removes the /32 address and the associated routes.
- Check on FRR to see that this /32 is still there.
If you don't have an environment to test :
- Create a blank workbench on https://cumulusnetworks.com/products/cumulus-in-the-cloud
- Connect to the oob-mgt-server
wget https://raw.githubusercontent.com/jpmondet/Fully-Automated-K8s-deployment/master/deploy_all.sh ; chmod u+x deploy_all.sh ; bash deploy_all.shssh server01 sudo kubectl run nginx --image=nginx- Check connected routes on a node :
ssh server03 "sudo vtysh -c 'sh ip route connected'" - Introducing a new /32 connected route :
ssh server01 sudo kubectl expose deploy nginx --port 80 - Check connected routes on a node :
ssh server03 "sudo vtysh -c 'sh ip route connected'" - Suppress this /32 route :
ssh server01 sudo kubectl delete svc nginx - Check connected routes on a node :
ssh server03 "sudo vtysh -c 'sh ip route connected'"
Expected behavior:
The /32 connected route should be deleted and not visible at all in FRR.
Actual behavior:
The /32 connected route is not deleted even if Zebra receives the event.
Components
[zebra, bgpd]
Versions
- OS: CentOS 7.5.1804
- Kernel: kernel-3.10.0-862.el7. Also tried with a new stable kernel (4.18.5)
- FRR: First on frr-3.2+cl3u4-2018022701 (last from cumulus repo) and then on the latest release 5.0.1 from here
Attachments
Linux interface before deletion :
11: kube-ipvs0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether 9a:95:b9:ab:6a:1a brd ff:ff:ff:ff:ff:ff
inet 10.9.20.168/32 brd 10.9.20.168 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.32.160/32 brd 10.9.32.160 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.45.242/32 brd 10.9.45.242 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.20.129/32 brd 10.9.20.129 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.36.81/32 brd 10.9.36.81 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.8.91/32 brd 10.9.8.91 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.0.3/32 brd 10.9.0.3 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.25.171/32 brd 10.9.25.171 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.55.54/32 brd 10.9.55.54 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.11.165/32 brd 10.9.11.165 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.26.72/32 brd 10.9.26.72 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.21.200/32 brd 10.9.21.200 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.41.238/32 brd 10.9.41.238 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.1.207/32 brd 10.9.1.207 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.12.38/32 brd 10.9.12.38 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.34.114/32 brd 10.9.34.114 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.22.244/32 brd 10.9.22.244 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.38.106/32 brd 10.9.38.106 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.12.45/32 brd 10.9.12.45 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.0.1/32 brd 10.9.0.1 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.52.115/32 brd 10.9.52.115 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.28.158/32 brd 10.9.28.158 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.4.248/32 brd 10.9.4.248 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.38.99/32 brd 10.9.38.99 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.59.197/32 brd 10.9.59.197 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.8.236/32 brd 10.9.8.236 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.36.196/32 brd 10.9.36.196 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.60.100/32 brd 10.9.60.100 scope global kube-ipvs0
valid_lft forever preferred_lft forever
----------> 10.9.34.114/32 & 10.9.60.100/32 are on the interface before deletion
And we can see them on FRR :
# sh ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel,
> - selected route, * - FIB route
C>* 10.9.0.1/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.0.3/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.1.207/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.4.248/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.8.91/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.8.236/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.11.165/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.12.38/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.12.45/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.20.129/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.20.168/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.21.200/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.22.244/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.25.171/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.26.72/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.28.158/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.32.160/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.34.114/32 is directly connected, kube-ipvs0, 01:00:18
C>* 10.9.36.81/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.36.196/32 is directly connected, kube-ipvs0, 00:51:46
C>* 10.9.38.99/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.38.106/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.41.238/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.45.242/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.52.115/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.55.54/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.59.197/32 is directly connected, kube-ipvs0, 2d22h34m
C>* 10.9.60.100/32 is directly connected, kube-ipvs0, 01:16:50
Linux interface after deletion :
11: kube-ipvs0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether 9a:95:b9:ab:6a:1a brd ff:ff:ff:ff:ff:ff
inet 10.9.20.168/32 brd 10.9.20.168 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.32.160/32 brd 10.9.32.160 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.45.242/32 brd 10.9.45.242 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.20.129/32 brd 10.9.20.129 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.36.81/32 brd 10.9.36.81 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.8.91/32 brd 10.9.8.91 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.0.3/32 brd 10.9.0.3 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.25.171/32 brd 10.9.25.171 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.55.54/32 brd 10.9.55.54 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.11.165/32 brd 10.9.11.165 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.26.72/32 brd 10.9.26.72 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.21.200/32 brd 10.9.21.200 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.41.238/32 brd 10.9.41.238 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.1.207/32 brd 10.9.1.207 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.12.38/32 brd 10.9.12.38 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.22.244/32 brd 10.9.22.244 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.38.106/32 brd 10.9.38.106 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.12.45/32 brd 10.9.12.45 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.0.1/32 brd 10.9.0.1 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.52.115/32 brd 10.9.52.115 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.28.158/32 brd 10.9.28.158 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.4.248/32 brd 10.9.4.248 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.38.99/32 brd 10.9.38.99 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.59.197/32 brd 10.9.59.197 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.8.236/32 brd 10.9.8.236 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.9.36.196/32 brd 10.9.36.196 scope global kube-ipvs0
valid_lft forever preferred_lft forever
----------> 10.9.34.114/32 & 10.9.60.100/32 are deleted from the interface.
But we still see them on FRR :
# sh ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel,
> - selected route, * - FIB route
C>* 10.9.0.1/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.0.3/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.1.207/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.4.248/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.8.91/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.8.236/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.11.165/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.12.38/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.12.45/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.20.129/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.20.168/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.21.200/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.22.244/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.25.171/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.26.72/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.28.158/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.32.160/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.34.114/32 is directly connected, kube-ipvs0, 01:13:41
C>* 10.9.36.81/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.36.196/32 is directly connected, kube-ipvs0, 01:05:09
C>* 10.9.38.99/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.38.106/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.41.238/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.45.242/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.52.115/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.55.54/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.59.197/32 is directly connected, kube-ipvs0, 2d22h47m
C>* 10.9.60.100/32 is directly connected, kube-ipvs0, 01:30:13
When I add a /32 IP, here are the Zebra debug logs :
2018/08/28 17:30:11 ZEBRA: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=76, seq=0, pid=0
2018/08/28 17:30:11 ZEBRA: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=76, seq=0, pid=0
2018/08/28 17:30:12 ZEBRA: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=76, seq=0, pid=0
2018/08/28 17:30:18 ZEBRA: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWADDR(20), len=92, seq=90847, pid=1
2018/08/28 17:30:18 ZEBRA: netlink_interface_addr RTM_NEWADDR kube-ipvs0 flags 0x80:
2018/08/28 17:30:18 ZEBRA: IFA_LOCAL 10.9.51.232/32
2018/08/28 17:30:18 ZEBRA: IFA_ADDRESS 10.9.51.232/32
2018/08/28 17:30:18 ZEBRA: IFA_BROADCAST 10.9.51.232/32
2018/08/28 17:30:18 ZEBRA: IFA_CACHEINFO pref -1, valid -1
2018/08/28 17:30:18 ZEBRA: MESSAGE: ZEBRA_INTERFACE_ADDRESS_ADD 10.9.51.232/32 on kube-ipvs0
2018/08/28 17:30:18 ZEBRA: rib_add_multipath: 0:10.9.51.232/32: Inserting route rn 0x1793480, re 0x1793300 (type 2) existing (nil)
2018/08/28 17:30:18 ZEBRA: rib_add_multipath: 0:10.9.51.232/32 (MRIB): Inserting route rn 0x17936d0, re 0x1793550 (type 2) existing (nil)
2018/08/28 17:30:18 ZEBRA: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
2018/08/28 17:30:18 ZEBRA: RTM_NEWROUTE ipv4 proto kernel vrf 0
2018/08/28 17:30:18 ZEBRA: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
2018/08/28 17:30:18 ZEBRA: RTM_NEWROUTE ipv4 proto kernel vrf 0
2018/08/28 17:30:18 ZEBRA: 0:10.9.51.232/32: Adding route rn 0x1793480, re 0x1793300 (type 2)
2018/08/28 17:30:18 ZEBRA: 0:10.9.51.232/32: Redist update re 0x1793300 (type 2), old (nil) (type -1)
2018/08/28 17:30:18 ZEBRA: 0:10.9.51.232/32: Adding route rn 0x17936d0, re 0x1793550 (type 2)
2018/08/28 17:30:18 ZEBRA: 0:10.9.51.232/32: Redist update re 0x1793550 (type 2), old (nil) (type -1)
And when I remove a /32 IP :
2018/08/28 17:35:41 ZEBRA: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=76, seq=0, pid=0
2018/08/28 17:35:41 ZEBRA: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=76, seq=0, pid=0
2018/08/28 17:36:00 ZEBRA: netlink_parse_info: netlink-listen (NS 0) type RTM_DELADDR(21), len=92, seq=87307, pid=1
2018/08/28 17:36:00 ZEBRA: netlink_interface_addr RTM_DELADDR kube-ipvs0 flags 0x80:
2018/08/28 17:36:00 ZEBRA: IFA_LOCAL 10.9.51.231/32
2018/08/28 17:36:00 ZEBRA: IFA_ADDRESS 10.9.51.231/32
2018/08/28 17:36:00 ZEBRA: IFA_BROADCAST 10.9.51.231/32
2018/08/28 17:36:00 ZEBRA: IFA_CACHEINFO pref -1, valid -1
2018/08/28 17:36:00 ZEBRA: netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=60, seq=0, pid=0
2018/08/28 17:36:00 ZEBRA: RTM_DELROUTE ipv4 proto kernel vrf 0
2018/08/28 17:36:00 ZEBRA: netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=60, seq=0, pid=0
2018/08/28 17:36:00 ZEBRA: RTM_DELROUTE ipv4 proto kernel vrf 0
2018/08/28 17:36:05 ZEBRA: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=76, seq=0, pid=0
The message ZEBRA: MESSAGE: ZEBRA_INTERFACE_ADDRESS_DELETE is not happening.
jpmondet
All 17 comments
I just tried this on Linux; it works fine there. I did not use Kubernetes but did the same process you describe w/ a dummy interface via iproute2. Will test on CentOS in the near future.
qlyoung
on 24 Sep 2018
Works on CentOS 7 as well for me:
[vagrant@localhost ~]$ uname -a
Linux localhost.localdomain 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
https://asciinema.org/a/2TuRr7DZAhcVeWw2IGO6PrIrD
@jpmondet can you try the same and see if this works for you? If so, then at least we know it's not a problem with Zebra <-> kernel
qlyoung
on 25 Sep 2018
Hello @qlyoung,
Thank you for looking into this issue :+1:
Yeah, manually, via iproute2, it looks ok, Zebra do the ZEBRA_INTERFACE_ADDRESS_DELETE.
However, via kube-proxy, Zebra also receives exactly the same netlink_interface_addr RTM_DELADDR but does nothing. This is what I don't understand.
And I can't incriminate kube-proxy since everything else on the Linux side is in a good state.
I'll try to create the same environment running on CITC for you to see/test the issue.
jpmondet
on 25 Sep 2018
Hello @qlyoung,
Here is the procedure to have a working environment and to check that the routes are not deleted :
- Create a blank workbench on https://cumulusnetworks.com/products/cumulus-in-the-cloud
- Connect to the oob-mgt-server
wget https://raw.githubusercontent.com/jpmondet/Fully-Automated-K8s-deployment/master/deploy_all.sh ; chmod u+x deploy_all.sh ; bash deploy_all.shssh server01 sudo kubectl run nginx --image=nginx- Check connected routes on a node :
ssh server03 "sudo vtysh -c 'sh ip route connected'" - Introducing a new /32 connected route :
ssh server01 sudo kubectl expose deploy nginx --port 80 - Check connected routes on a node :
ssh server03 "sudo vtysh -c 'sh ip route connected'" - Suppress this /32 route :
ssh server01 sudo kubectl delete svc nginx - Check connected routes on a node :
ssh server03 "sudo vtysh -c 'sh ip route connected'"
jpmondet
on 29 Sep 2018
FYI, I also got the issue with Ubuntu and with the last version of Kubernetes (1.12).
Should I try with frr-6.1-dev ?
jpmondet
on 2 Oct 2018
If you could please, that would be helpful.
qlyoung
on 5 Oct 2018
If you could please, that would be helpful.
Ok, done it just now with the most recent commits on the master branch : git build of frr-6.1-dev-122-g1c7e6ef
I then followed the items 4 through 9 described here : https://github.com/FRRouting/frr/issues/3053#issuecomment-425627015
The issue is still there :
#ip a show dev kube-ipvs0
8: kube-ipvs0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether 2a:87:4c:df:29:ac brd ff:ff:ff:ff:ff:ff
inet 85.85.85.85/32 brd 85.85.85.85 scope global kube-ipvs0
valid_lft forever preferred_lft forever
#sh ip route connected
C>* 85.85.85.85/32 is directly connected, kube-ipvs0, 00:01:44
C>* 85.85.90.225/32 is directly connected, kube-ipvs0, 00:01:18
exactly same issue. I found the reason is that
ip -4 addr add 10.193.0.13/32 dev qg-0bd30c4f-0a brd 10.193.0.13
brd <- here
if i delete the broadcasting address, it was deleted normally. I don't know why though..
Xeite
on 28 Nov 2018
I don't understand what the exact purpose of the code below is, maybe it results from
struct connected *connected_check_ptp(struct interface *ifp,
│ │ union prefixconstptr pu,
│ │ union prefixconstptr du)
{
const struct prefix *p = pu.p;
const struct prefix *d = du.p;
struct connected *ifc;
struct listnode *node;
/* ignore broadcast addresses */
if (p->prefixlen != IPV4_MAX_PREFIXLEN)
d = NULL;
for (ALL_LIST_ELEMENTS_RO(ifp->connected, node, ifc)) {
if (!prefix_same(ifc->address, p))
continue;
if (!CONNECTED_PEER(ifc) && !d)
return ifc;
if (CONNECTED_PEER(ifc) && d
│ && prefix_same(ifc->destination, d))
return ifc;
}
return NULL;
}
This function just return NULL since 'd' is NULL and ifc does not have peer address. I think it's bug that assume that address with 32 prefix is for broadcast.
Xeite
on 28 Nov 2018
@qlyoung Hello, did you have a little time to look further into this issue ? @Xeite saw something similar and a potential hint (thanks a lot by the way ! :+1: ).
The brd address could definitively be an explication, however, since I don't add addresses manually (kube-proxy does it for me), I can't try and confirm.
EDIT: Actually, I was able to confirm. The brd address specifically set to the same address as the /32 is annoying FRR somehow. I also tried on eth, lo & dummy -> same issue.
For the last comment on the code snippet, I can't really tell. Maybe someone with a deep knowledge on the FRR codebase could ? @donaldsharp maybe ?
jpmondet
on 9 Dec 2018
Seems duplicate https://github.com/FRRouting/frr/issues/1967
gmshake
on 14 Jan 2019
Same issue here with frr 7.0.1
HaveFun83
on 26 Mar 2019
Note that this is a kernel misconfiguration by kube-proxy.
inet 10.9.20.168/32 brd 10.9.20.168 scope global kube-ipvs0
It's a single-IP subnet (no problem there) but the IP address is also set as broadcast address. While we could work around this, this needs to be properly fixed in Kubernetes. (Kubernetes should simply not configure any broadcast address; it's just plain wrong to do so and completely unnecessary.)
eqvinox
on 23 Apr 2019
Note that this is a kernel misconfiguration by kube-proxy.
inet 10.9.20.168/32 brd 10.9.20.168 scope global kube-ipvs0It's a single-IP subnet (no problem there) but the IP address is also set as broadcast address. While we could work around this, this needs to be properly fixed in Kubernetes. (Kubernetes should simply not configure any broadcast address; it's just plain wrong to do so and completely unnecessary.)
Hmm, I understand your point of view. Going further with your point of view, Linux shouldn't let you configure this, right ?
jpmondet
on 24 Apr 2019
This issue should now be fixed in master with the merge of #4812. @jpmondet can you verify this with your setup?
pogojotz
on 27 Aug 2019
Awesome @pogojotz ! I'll try this and give you a feedback asap :D
jpmondet
on 28 Aug 2019
This issue should now be fixed in master with the merge of #4812. @jpmondet can you verify this with your setup?
This is perfect, it works flawlessly for me, thanks a lot @pogojotz ! :1st_place_medal:
We can close this issue ! :smile:
jpmondet
on 29 Aug 2019
Related issues
andylemin
·
3Comments
Chen0Yang
·
3Comments
netravnen
·
6Comments
hansengao
·
5Comments
pguibert6WIND
·
5Comments
Most helpful comment
exactly same issue. I found the reason is that
brd <- here
if i delete the broadcasting address, it was deleted normally. I don't know why though..