Frr: VRF support broken in contrast with quagga 1.1 implementation

[pguibert6WIND]

One difference with quagga 1.1 is that the frrouting feature of VRF has changed, and brings two issues:

• the VRF feature in frrouting is lost for kernels not supporting [1], whereas it was working on Quagga.
• it is not possible to rely on kernel netns, whereas in Quagga the netns solution was used.

[1] https://www.kernel.org/doc/Documentation/networking/vrf.txt

This issue is here to solve that.

[donaldsharp]

@pguibert6WIND Can you help me to understand what is broken exactly? What do you want the behavior to be?

[pguibert6WIND]

Hi @donaldsharp ,

========
Behind what I say is "broken" means that VRF features does not work like it was working in Quagga 1.1 world.
It is broken, since frrouting users can not do the same thing as they were doing when they were quagga users.
Before going further let me precise some terms:

• VRF kernel feature
• Netns kernel feature
• "VRF quagga" versus "VRF frrouting"

This has some consequences for frrouting users:

• if frrouting users use old kernels not supporting "VRF kernel feature", then "VRF frrouting" is useless.
  in the same conditions, if quagga users use old kernels, then they can use the "VRF quagga"., and it works. This is because "VRF quagga" uses "Netns kernel feature".
  For quagga users wishing to move to frrouting users, if they have an established setup based on "VRF quagga", they may want to still continue to benefit from "Netns kernel feature".

========
Technically speaking :

• the quagga maps "VRF quagga" on "Netns kernel feature"
• the frrouting maps "VRF frrouting" on "VRF kernel feature"

The intention is to do the following:
Give the user the ability to choose between mapping "VRF frrouting" between "VRF kernel feature" or "Netns kernel feature".
The intention is to do the following:

• provide a runtime configuration for frrouting that selects the kernel feature: netns or vrf.
  by default, it could be vrf kernel feature, and if VRF kernel feature is not present, it would fallback to NETNS kernel feature. It could however be possible to map to a vty command the wish for the user to set "NETNS kernel feature.

I dont't know if the problem description is clear ?

[pguibert6WIND]

quagga 1.1.1 enclosed permits adding a static route in netns1
This feature is not available in FRR.
config_vrf.txt

[k0ste]

Thanks @pguibert6WIND
Who doesn't works with "VRF kernel feature", I make example (did this before).

[root@linux01 ~]# ip link add name vlan100 link eth0 type vlan id 100
[root@linux01 ~]# ip link add name vlan3999 link eth0 type vlan id 3999
[root@linux01 ~]# ip link set vlan100 up
[root@linux01 ~]# ip link set vlan3999 up
[root@linux01 ~]# ip a add 10.0.0.2/24 brd 10.0.0.255 dev vlan100
[root@linux01 ~]# ip a add 10.0.0.2/24 brd 10.0.0.255 dev vlan3999
[root@linux01 ~]# ip link add vrf100 type vrf table 100
[root@linux01 ~]# ip link set dev vlan100 master vrf100
[root@linux01 ~]# ip link set vrf100 up
[root@linux01 ~]# ip link add vrf3999 type vrf table 3999
[root@linux01 ~]# ip link set dev vlan3999 master vrf3999
[root@linux01 ~]# ip link set vrf3999 up
[root@linux01 ~]# iptables -t filter -A INPUT -p icmp -i vrf100 -j LOG --log-prefix "from vrf100 "
[root@linux01 ~]# iptables -t filter -A INPUT -p icmp -i vrf3999 -j LOG --log-prefix "from vrf3999 "
[root@linux01 ~]# dmesg -TL | grep vrf
[Sun Jan 15 17:21:30 2017] from vrf100 IN=vrf100 OUT= MAC=08:00:27:92:41:3d:08:00:27:65:f1:17:08:00 SRC=10.0.0.3 DST=10.0.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=34698 DF PROTO=ICMP TYPE=8 CODE=0 ID=391 SEQ=1 
[Sun Jan 15 17:21:31 2017] from vrf100 IN=vrf100 OUT= MAC=08:00:27:92:41:3d:08:00:27:65:f1:17:08:00 SRC=10.0.0.3 DST=10.0.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=34871 DF PROTO=ICMP TYPE=8 CODE=0 ID=391 SEQ=2 
[Sun Jan 15 17:21:35 2017] from vrf3999 IN=vrf3999 OUT= MAC=08:00:27:92:41:3d:08:00:27:65:f1:17:08:00 SRC=10.0.0.3 DST=10.0.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=35672 DF PROTO=ICMP TYPE=8 CODE=0 ID=392 SEQ=1 
[Sun Jan 15 17:21:36 2017] from vrf3999 IN=vrf3999 OUT= MAC=08:00:27:92:41:3d:08:00:27:65:f1:17:08:00 SRC=10.0.0.3 DST=10.0.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=35915 DF PROTO=ICMP TYPE=8 CODE=0 ID=392 SEQ=2 
[root@linux01 ~]# ip ne sh vrf vrf100
10.0.0.3 dev vlan100 lladdr 08:00:27:65:f1:17 REACHABLE
[root@linux01 ~]# ip ne sh vrf vrf3999
10.0.0.3 dev vlan3999 lladdr 08:00:27:65:f1:17 STALE
[root@linux01 ~]# ip ru sh
0:      from all lookup local 
1000:   from all lookup [l3mdev-table] 
32766:  from all lookup main 
32767:  from all lookup default 
[root@linux01 ~]#

[root@linux02 ~]# ip link add name vlan100 link eth0 type vlan id 100
[root@linux02 ~]# ip link add name vlan3999 link eth0 type vlan id 3999
[root@linux02 ~]# ip link set vlan100 up
[root@linux02 ~]# ip link set vlan3999 up
[root@linux02 ~]# ip a add 10.0.0.3/24 brd 10.0.0.255 dev vlan100
[root@linux02 ~]# ip a add 10.0.0.3/24 brd 10.0.0.255 dev vlan3999
[root@linux02 ~]# ip link add vrf100 type vrf table 100
[root@linux02 ~]# ip link set dev vlan100 master vrf100
[root@linux02 ~]# ip link set vrf100 up
[root@linux02 ~]# ip link add vrf3999 type vrf table 3999
[root@linux02 ~]# ip link set dev vlan3999 master vrf3999
[root@linux02 ~]# ip link set vrf3999 up
[root@linux02 ~]# ping -c 2 -I vrf100 10.0.0.2
ping: Warning: source address might be selected on device other than vrf100.
PING 10.0.0.2 (10.0.0.2) from 10.0.0.3 vrf100: 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=1.30 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=0.840 ms

--- 10.0.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1003ms
rtt min/avg/max/mdev = 0.840/1.070/1.300/0.230 ms
[root@linux02 ~]# ping -c 2 -I vrf3999 10.0.0.2
ping: Warning: source address might be selected on device other than vrf3999.
PING 10.0.0.2 (10.0.0.2) from 10.0.0.3 vrf3999: 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=1.15 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=0.747 ms

--- 10.0.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.747/0.951/1.156/0.206 ms1
[root@linux02 ~]# ip ru sh
0:      from all lookup local 
1000:   from all lookup [l3mdev-table] 
32766:  from all lookup main 
32767:  from all lookup default 
[root@linux02 ~]#

[pguibert6WIND]

Hello @k0ste ,
Thanks for pointing out this linux configuration example to enhance "VRF kernel feature".

Before "VRF kernel feature" exists, "NetNs kernel feature" was used by quagga to map "Quagga VRF".
In the same way it is illustrated by you, you can create a netns in the shell;

Then, once done, you can configure in the routing engine.
For netns, you must call function 'ip netns add netns1' then look at this file

=================================
What I want to raise as subject is that frrouting should be able to map "frrouting VRF" to both mecanics: either "VRF kernel feature" or "Netns kernel feature".

This could be done through usage of a default vty command

```shell
zebra# debug vrf kernel-vrf-mode

[k0ste]

I want to note, the most optimal variant of development is the creation VRF interfaces by the frr, instead of manual work in the shell, then, if there was an error, expect the correct operation of the frr ("VRF kernel feature").

For less advanced Linux users, this causes problems that you need to perform a configuration somewhere else besides vtysh. Accordingly, it repels new users to the Linux routing ("I used Cisco CLI all my life", and similar statements have a place to be) of frr as main choice. #340 as example what I mean.

[mTaleqani]

@k0ste - I tried to make a vrf in the way you said (using VRF kernel feature), but encountered the following error :

i have installed IProute2 and my linux kernel is 4.10 (4.10.6-1.el6.elrepo.x86_64) and using centos 6.5 but unable to create vrf.
for the ones like me beginner in linux, it would be great to be able to create VRFs through FRR.

[k0ste]

Check your kernel config:

[k0ste@WorkStation ~]$ gunzip -c /proc/config.gz | grep -i vrf
CONFIG_NET_VRF=m

Check iproute know about vrf:

[k0ste@WorkStation ~]$ ip link help vrf
Usage: ... vrf table TABLEID

[mTaleqani]

@k0ste - there is no file named "config.gz" in /proc. but i found the kernel config in
/boot/config-4.10.6-1.el6.elrepo.x86_64 . then i searched for VRF keyword in kernel config but nothing found.

this is the result of executing the command "ip link help vrf" :

[k0ste]

@mTaleqani, you should build kernel with NET_VRF. Or try use ArchLinux for example, where NET_VRF is enabled by default.

[pguibert6WIND]

Hi @mTaleqani ,

I think you are not the only one that wants to benefit from "frrouting VRF feature", while you don't have the correct kernel ( either by kernel settings or old kernel).
That is why it is proposed to be able to configure and map the "frrouting VRF feature" into the "netns kernel feature". "Netns Kernel feature" should be present in your case.

This issue is just here to :

• collect the +1 approvals on this initiative ( there may be some disagreements on the way the implementation or the frrouting should go
• implement the stuff.

[dsahern]

@k0ste In your VRF commands above you show:

"[root@linux02 ~]# ip ru sh
0: from all lookup local
1000: from all lookup [l3mdev-table]
32766: from all lookup main
32767: from all lookup default
"

You should move the local rule to after the l3mdev-table rule to avoid false hits (e.g, duplicate addresses or overlapping networks between interfaces in the default VRF and another VRF). The l3mdev rule defaults to priority 1000 to allow users to have higher priority policy rules.

[donaldsharp]

Having FRR create the vrf, is a separate issue. If there is interest file a new issue and someone can take up that coding effort.

[dsahern]

I'd be careful about making FRR a swiss army knife. A VRF is used by more than just FRR. Maintaining all of the network configuration is better done (meaning cleaner and less complicated) by a network manager or interface manager.

[rwestphal]

@pguibert6WIND I'm ok with your proposal.

The long term goal is to have an abstraction layer that is capable of handling VRFs on different stacks, like FreeBSD's Jails, OpenBSD's rdomains, etc. So IMO having two different VRF methods for Linux is reasonable considering that we need to have an abstraction layer anyway. Also, a lot of people will be stuck with older kernels for a while, like people dealing with embedded systems, and we want to get these people on board as well.

My only concern is that in the long term the idea is to support namespaces/containers in addition to VRFs in FRR. Theoretically we can use netns's for namespaces and nested netns's for VRFs, but the code can get messy if we decided to go that way...

[pguibert6WIND]

FRR VRF will now be able to handle the following:

• discover netns from system ( equivalent to get 'ip netns list') and map it to VRF
  ex: # ip netns list
  test
  test2
  ./zebra/zebra&
  ip netns list
  test:0 <--- an ID is automatically generated if not available,and used to encode the vrf-id
  test:1
  zebra# show running-config
  ...
  zebra(conf)# vrf test
  zebra(vrf)# netns /var/run/netns
  zebra(conf)# exit
  zebra(conf)# vrf test2
  zebra(vrf)# netns /var/run/netns2
  zebra(conf)# exit
  ...
• discover vrf lite inside netns and map it to VRF
  (zebra)# show vrf
  vrf test, netns test, vrf ID 0
  vrf test2, netns test2, vrf ID 65536 <-- value is 1 << 16
  vrf vrf-test, vrf lite from netns test2, vrf ID 65560 <--- value is 1 << 16 + 24 ( ifindex from iface vrf-test)

zebra will be in charge of doing that discovery, by initing new zns context.

• the vrf support is then extended to netns, in addition to support vrf lite.
  The vrf id 32 bit value permits supporting both VRFs.
  this permits frr to get the VRF information, so as to know if that VRF is a vrf lite or a standard VRF, and then do the appropriate actions.
  example: for BGP, when creating socket server for BGP VRF test, then before calling new socket, an API will be called so as to move to an other NETNS, ( and move back to original NETNS after).

an implementation is in progress. in 3 steps

• the VRF ID is moved from 16 bit to 32 bit so as to be able to handle vrf either mapped on vrf namespaces or vrf lite ( from kernel)
  https://github.com/FRRouting/frr/pull/1574
• a framework is brought in both lib and zebra folder, so as to handle namespaces through vrf.
• support for other daemons, by beginnning with BGP.

[pguibert6WIND]

Hi @louberger , if you happen to be online, some lecture is present in this ticket and may help you on the questioning:
"I guess I don't understand the overall use case for netns. is it just another way to do vrf lite, or something more?"

[louberger]

Okay thanks for the info. Looks interesting / useful. I'll need to look a
little more carefully at the existing VRF feature to fully understand
consequences.

On December 26, 2017 9:19:45 AM Philippe Guibert [e-mail]
wrote:

Hi @louberger , if you happen to be online, some lecture is present in this
ticket and may help you on the questioning:
"I guess I don't understand the overall use case for netns. is it just
another way to do vrf lite, or something more?"

--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/FRRouting/frr/issues/385#issuecomment-353973618

[pguibert6WIND]

discussion with @donaldsharp about this feature:

• memory impact with pim
• also mapping VRF over Netns does not mean that all underlying functionalities in the dataplane are ready ( this is the case of vrf route leaking mechanism that is not available for inter-NS communication).
• VRF concept may be confusing, because it could not only be mapped with vrf lite, but also with netns
• 2nd confusion with the concept of logical-router ( cisco vdc is an illustration), where the netns are mapped with netns.
  The two confusing points may lead to think that in order to clarify, a conditional compilation should be applied to NETNS support.

[vjardin]

@pguibert6WIND @donaldsharp
We should avoid compilation options since it creates some complexity to manage the packaging.

[donaldsharp]

This has been integrated with FRR