Calico: AS number per node is not reflected in calico/node

[m4r10k]

Hi at all! I have a problem during the setting of a different AS number for specific hosts.

Expected Behavior

AS number which is set on per host bases is set in bird.cfg as local as.

Current Behavior

Setting the AS peer number does not get reflected in the bird.cfg in the calico node. This line does not get the node_as_key and sticks to the global AS number:

local as {{if exists $node_as_key}}{{getv $node_as_key}}{{else}}{{getv "/global/as_num"}}{{end}};

results in

local as 63400;

which is obviously wrong:

bgppeer-node-x4141   10.x.x.140   x4141   63500

which results in:

| 10.x.x140  | node specific | start | 12:33:20 | Idle Received: Bad peer A

Possible Solution

$node_as_key should be found if set by bgppeer.

Steps to Reproduce (for bugs)

1. Set node local bgppeer
2. Take a look inside the bird.cfg

Context

I am not able to drive Calico in multiple data center with different BGP edge router with different AS numbers.

Your Environment

• Calico version

Client Version:    v2.0.4
Build date:        2018-04-20T19:51:26+0000
Git commit:        cd12a632
Cluster Version:   v3.1.1
Cluster Type:      k8s,bgp

• Orchestrator version (e.g. kubernetes, mesos, rkt):
  kubeadm, on premise

VERSION   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
v1.10.1   Ubuntu 16.04.4 LTS   4.4.0-116-generic   docker://17.12.1-ce
v1.10.1   Ubuntu 16.04.4 LTS   4.4.0-116-generic   docker://17.12.1-ce
v1.10.1   Ubuntu 16.04.4 LTS   4.4.0-116-generic   docker://17.12.1-ce
v1.10.1   Ubuntu 16.04.4 LTS   4.4.0-116-generic   docker://17.12.1-ce
v1.10.1   Ubuntu 16.04.4 LTS   4.4.0-116-generic   docker://17.12.1-ce
v1.10.1   Ubuntu 16.04.4 LTS   4.4.0-127-generic   docker://17.12.1-ce

[fasaxc]

@m4r10k Please can you explain exactly what you did to set the AS? What config did you apply?

[m4r10k]

Yes, I can.

First, I disabled the node to node mesh.

NAME      LOGSEVERITY   MESHENABLED   ASNUMBER   
default   Info          false         63400 

Next, just for testing, I removed the ASNUMBER in the default config, which leads to a default number of 64123 or so. So I turned it back to this configuration.

Than I added all hosts with a peer configuration:

apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: bgppeer-node-x205
spec:
  peerIP: 10.x.x.211
  node: x205
  asNumber: 63400
---
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: bgppeer-node-x206
spec:
  peerIP: 10.x.x.211
  node: x206
  asNumber: 63400
---
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: bgppeer-nodex207
spec:
  peerIP: 10.x.x.211
  node: x207
  asNumber: 63400
---
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: bgppeer-node-x208
spec:
  peerIP: 10.x.x.211
  node: x208
  asNumber: 63400
---
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: bgppeer-node-x209
spec:
  peerIP: 10.x.x.211
  node: x209
  asNumber: 63400
---
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: bgppeer-node-x4141
spec:
  peerIP: 10.x.x.140
  node: x4141
  asNumber: 63400
---

The last node (bgppeer-node-x4141) should use another asNumber as it is in another data center. The configuration at the moment works, because the bird rr in this data center also uses this numer:

# Configure logging
#log syslog { debug, trace, info, remote, warning, error, auth, fatal, bug };
#log stderr {info, warning, error};
log stderr all;

# Override router ID
router id 10.73.4.140;

filter import_kernel {
if ( net != 0.0.0.0/0 ) then {
   accept;
   }
reject;
}

# Turn on global debugging of all protocols
debug protocols all;

# This pseudo-protocol watches all interface up/down events.
protocol device {
  scan time 2;    # Scan interfaces every 2 seconds
}

protocol kernel {
#  learn;           # Learn all alien routes from the kernel
#  persist;     # Don't remove routes on bird shutdown
  scan time 20;     # Scan kernel routing table every 20 seconds
#   import none;        # Default is import all
  import all;
  export all;       # Default is export none
#  kernel table 5;      # Kernel table to synchronize with (default: main)
}
protocol bgp x4141 {
  description "10.x.x.141";
  local as 63400;
  neighbor 10.x.x.141 as 63400;
  multihop;
  rr client;
  graceful restart;
  import all;
  export all;
  direct;
}

What I think that should work is:

• Change the as node for peer to:

apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: bgppeer-node-x4141
spec:
  peerIP: 10.x.x.140
  node: x4141
  asNumber: 63500

• Change the bird config to:

protocol bgp x4141 {
  description "10.x.x.141";
  local as 63500;
  neighbor 10.x.x.141 as 63500;
  multihop;
  rr client;
  graceful restart;
  import all;
  export all;
  direct;
}

To let the node and the bird rr in this data center use the asnumber 63500.

But as written above, local as {{if exists $node_as_key}}{{getv $node_as_key}}{{else}}{{getv "/global/as_num"}}{{end}}; results in the global as number.

I've just printed the $node_as_key as comment into the config file and this results in the path of the libcalico_api /host/x4141/... and I think there is no value coming back, but I do not know where to look in the ETCD to look after the configuration because /host/x4141/... is not the ETCD key.

[fasaxc]

The BGPPeer should controls the _remote_ AS number.
If you want to control the _local_ AS number, you need to create a BGPConfiguration object for the node.

[m4r10k]

Ok, I totally overlooked this in the documentation node.<nodename>. sorry :blush: ! but:

ApiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: node.x4141
spec:
  asNumber: 63500

Results in:

./calicoctl apply -f calico-bgbconfiguration-node.yaml
Failed to apply 'BGPConfiguration' resource: error with field BGPConfiguration.Spec.ASNumber (Cannot set ASNumber on a non default BGP Configuration.)

[fasaxc]

Docs fix: https://github.com/projectcalico/calico/pull/2025

[fasaxc]

Failed to apply 'BGPConfiguration' resource: error with field BGPConfiguration.Spec.ASNumber (Cannot set ASNumber on a non default BGP Configuration.)

That's odd, i thought that was allowed.

[m4r10k]

:smile: Yes, indeed, that would be the solution I guess - not a real problem at the moment. But currently the different AS number by rack/node example does not work.

[robbrockbank]

The per node AS number is set through the Node resource, so if you do a calicoctl get node <nodename> -o yaml you can edit/add the asNumber field to the BGP configuration in the node and re-apply. You can also set the AS number environment when starting the node - which will overwrite the ASNumber configured in the Node resource each time the node restarts.

[m4r10k]

@robbrockbank Yes great, this does the trick! Thank you! Therefore, this is not a bug but it maybe should get documented in a tutorial? I am currently writing a larger documentation (https://gitlab.com/n0r1sk/kubernetes-the-roguelike-way) where Calico (and the config customization) is a part of it. If you like, I can write something for the Calico documentation too when it is finished...