calico-node never becomes healthy

[thommay]

Expected Behavior

The calico-node should become healthy

Current Behavior

> kubectl describe -n kube-system pods calico-node-8k2tm
  Warning  Unhealthy  25s (x61 over 25m)    kubelet, upsilon   Readiness probe failed: calico/node is not ready: felix is not ready: readiness probe reporting 503

click for kubectl logs output

> kubectl logs --namespace kube-system  calico-node-8k2tm
    /go/src/github.com/projectcalico/node/cmd/calico-node/main.go:100 +0x405
2019-10-02 20:20:13.305 [INFO][25423] logutils.go 82: Early screen log level set to info
2019-10-02 20:20:13.306 [INFO][25423] daemon.go 146: Felix starting up GOMAXPROCS=12 builddate="7a042b762bc3a20b28a712e22fcc72160e4b5ddf" gitcommit="2019-09-26T17:59:26+0000" version="v3.9.1"
2019-10-02 20:20:13.306 [INFO][25423] daemon.go 164: Loading configuration...
2019-10-02 20:20:13.307 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "ipv6support"="false"
2019-10-02 20:20:13.307 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "felixhostname"="upsilon"
2019-10-02 20:20:13.307 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "etcdcertfile"=""
2019-10-02 20:20:13.307 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "logseverityscreen"="info"
2019-10-02 20:20:13.307 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "etcdscheme"=""
2019-10-02 20:20:13.307 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "defaultendpointtohostaction"="ACCEPT"
2019-10-02 20:20:13.307 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "etcdaddr"=""
2019-10-02 20:20:13.308 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "datastoretype"="kubernetes"
2019-10-02 20:20:13.308 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "etcdkeyfile"=""
2019-10-02 20:20:13.308 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "healthenabled"="true"
2019-10-02 20:20:13.308 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "ipinipmtu"="1440"
2019-10-02 20:20:13.308 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "etcdendpoints"=""
2019-10-02 20:20:13.308 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "etcdcafile"=""
2019-10-02 20:20:13.308 [INFO][25423] daemon.go 187: Loading config file: /etc/calico/felix.cfg
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 253: Merging in config from environment variable: map[datastoretype:kubernetes defaultendpointtohostaction:ACCEPT etcdaddr: etcdcafile: etcdcertfile: etcdendpoints: etcdkeyfile: etcdscheme: felixhostname:upsilon healthenabled:true ipinipmtu:1440 ipv6support:false logseverityscreen:info]
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 262: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="etcdendpoints" source=environment variable
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 262: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="etcdcafile" source=environment variable
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 262: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="etcdscheme" source=environment variable
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 262: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="etcdaddr" source=environment variable
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 262: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="etcdkeyfile" source=environment variable
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 262: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="etcdcertfile" source=environment variable
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 334: Parsing value for LogSeverityScreen: info (from environment variable)
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 370: Parsed value for LogSeverityScreen: INFO (from environment variable)
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 334: Parsing value for DatastoreType: kubernetes (from environment variable)
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 370: Parsed value for DatastoreType: kubernetes (from environment variable)
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 334: Parsing value for HealthEnabled: true (from environment variable)
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 370: Parsed value for HealthEnabled: true (from environment variable)
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 334: Parsing value for Ipv6Support: false (from environment variable)
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 370: Parsed value for Ipv6Support: false (from environment variable)
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 334: Parsing value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 370: Parsed value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 334: Parsing value for IpInIpMtu: 1440 (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 370: Parsed value for IpInIpMtu: 1440 (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 334: Parsing value for FelixHostname: upsilon (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 370: Parsed value for FelixHostname: upsilon (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 253: Merging in config from config file: map[LogFilePath:None LogSeverityFile:None LogSeveritySys:None MetadataAddr:None]
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 334: Parsing value for HealthEnabled: true (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 370: Parsed value for HealthEnabled: true (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 334: Parsing value for Ipv6Support: false (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 370: Parsed value for Ipv6Support: false (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 334: Parsing value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 370: Parsed value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 334: Parsing value for IpInIpMtu: 1440 (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 370: Parsed value for IpInIpMtu: 1440 (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 334: Parsing value for FelixHostname: upsilon (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 370: Parsed value for FelixHostname: upsilon (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 334: Parsing value for LogSeverityScreen: info (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 370: Parsed value for LogSeverityScreen: INFO (from environment variable)
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 334: Parsing value for DatastoreType: kubernetes (from environment variable)
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 370: Parsed value for DatastoreType: kubernetes (from environment variable)
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 334: Parsing value for MetadataAddr: None (from config file)
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 370: Parsed value for MetadataAddr:  (from config file)
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 334: Parsing value for LogFilePath: None (from config file)
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 370: Parsed value for LogFilePath:  (from config file)
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 334: Parsing value for LogSeverityFile: None (from config file)
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 370: Parsed value for LogSeverityFile:  (from config file)
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 334: Parsing value for LogSeveritySys: None (from config file)
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 370: Parsed value for LogSeveritySys:  (from config file)
2019-10-02 20:20:13.311 [INFO][25423] health.go 233: Health enabled.  Starting server. host="localhost" port=9099
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 455: Encap disabled, disabling node poll (if KDD is in use).
2019-10-02 20:20:13.311 [INFO][25423] daemon.go 219: Connecting to datastore datastore="kubernetes"
2019-10-02 20:20:13.314 [INFO][25423] k8s.go 228: Using Calico IPAM
2019-10-02 20:20:13.314 [INFO][25423] daemon.go 226: Created datastore client
2019-10-02 20:20:13.328 [INFO][25423] daemon.go 800: Loaded ready flag kind="ClusterInformation" name="default" ready=true
2019-10-02 20:20:13.335 [INFO][25423] daemon.go 780: No config of this type kind="FelixConfiguration" name="node.upsilon"
2019-10-02 20:20:13.340 [INFO][25423] config_params.go 253: Merging in config from datastore (global): map[CalicoVersion:v3.9.1 ClusterGUID:76ffcce211b64b55b8650bdf51ce73ae ClusterType:k8s,bgp,kdd IpInIpEnabled:true LogSeverityScreen:Info ReportingIntervalSecs:0]
2019-10-02 20:20:13.340 [INFO][25423] config_params.go 334: Parsing value for LogSeverityScreen: info (from environment variable)
2019-10-02 20:20:13.340 [INFO][25423] config_params.go 370: Parsed value for LogSeverityScreen: INFO (from environment variable)
2019-10-02 20:20:13.340 [INFO][25423] config_params.go 334: Parsing value for DatastoreType: kubernetes (from environment variable)
2019-10-02 20:20:13.340 [INFO][25423] config_params.go 370: Parsed value for DatastoreType: kubernetes (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 334: Parsing value for HealthEnabled: true (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 370: Parsed value for HealthEnabled: true (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 334: Parsing value for Ipv6Support: false (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 370: Parsed value for Ipv6Support: false (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 334: Parsing value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 370: Parsed value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 334: Parsing value for IpInIpMtu: 1440 (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 370: Parsed value for IpInIpMtu: 1440 (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 334: Parsing value for FelixHostname: upsilon (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 370: Parsed value for FelixHostname: upsilon (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 334: Parsing value for LogSeverityFile: None (from config file)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 370: Parsed value for LogSeverityFile:  (from config file)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 334: Parsing value for LogSeveritySys: None (from config file)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 370: Parsed value for LogSeveritySys:  (from config file)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 334: Parsing value for MetadataAddr: None (from config file)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 370: Parsed value for MetadataAddr:  (from config file)
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 334: Parsing value for LogFilePath: None (from config file)
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 370: Parsed value for LogFilePath:  (from config file)
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 334: Parsing value for ClusterType: k8s,bgp,kdd (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 370: Parsed value for ClusterType: k8s,bgp,kdd (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 334: Parsing value for CalicoVersion: v3.9.1 (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 370: Parsed value for CalicoVersion: v3.9.1 (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 334: Parsing value for LogSeverityScreen: Info (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 370: Parsed value for LogSeverityScreen: INFO (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 373: Skipping config value for LogSeverityScreen from datastore (global); already have a value from environment variable
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 334: Parsing value for IpInIpEnabled: true (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 370: Parsed value for IpInIpEnabled: true (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 334: Parsing value for ReportingIntervalSecs: 0 (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 370: Parsed value for ReportingIntervalSecs: 0s (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 334: Parsing value for ClusterGUID: 76ffcce211b64b55b8650bdf51ce73ae (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 370: Parsed value for ClusterGUID: 76ffcce211b64b55b8650bdf51ce73ae (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 253: Merging in config from datastore (per-host): map[IpInIpTunnelAddr:192.168.33.128]
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 334: Parsing value for HealthEnabled: true (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 370: Parsed value for HealthEnabled: true (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 334: Parsing value for Ipv6Support: false (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 370: Parsed value for Ipv6Support: false (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 334: Parsing value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 370: Parsed value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 334: Parsing value for IpInIpMtu: 1440 (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 370: Parsed value for IpInIpMtu: 1440 (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 334: Parsing value for FelixHostname: upsilon (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 370: Parsed value for FelixHostname: upsilon (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 334: Parsing value for LogSeverityScreen: info (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 370: Parsed value for LogSeverityScreen: INFO (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 334: Parsing value for DatastoreType: kubernetes (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 370: Parsed value for DatastoreType: kubernetes (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 334: Parsing value for MetadataAddr: None (from config file)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 370: Parsed value for MetadataAddr:  (from config file)
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 334: Parsing value for LogFilePath: None (from config file)
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 370: Parsed value for LogFilePath:  (from config file)
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 334: Parsing value for LogSeverityFile: None (from config file)
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 370: Parsed value for LogSeverityFile:  (from config file)
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 334: Parsing value for LogSeveritySys: None (from config file)
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 370: Parsed value for LogSeveritySys:  (from config file)
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 334: Parsing value for IpInIpTunnelAddr: 192.168.33.128 (from datastore (per-host))
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 370: Parsed value for IpInIpTunnelAddr: 192.168.33.128 (from datastore (per-host))
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 334: Parsing value for ClusterGUID: 76ffcce211b64b55b8650bdf51ce73ae (from datastore (global))
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 370: Parsed value for ClusterGUID: 76ffcce211b64b55b8650bdf51ce73ae (from datastore (global))
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 334: Parsing value for ClusterType: k8s,bgp,kdd (from datastore (global))
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 370: Parsed value for ClusterType: k8s,bgp,kdd (from datastore (global))
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 334: Parsing value for CalicoVersion: v3.9.1 (from datastore (global))
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 370: Parsed value for CalicoVersion: v3.9.1 (from datastore (global))
2019-10-02 20:20:13.345 [INFO][25423] config_params.go 334: Parsing value for LogSeverityScreen: Info (from datastore (global))
2019-10-02 20:20:13.345 [INFO][25423] config_params.go 370: Parsed value for LogSeverityScreen: INFO (from datastore (global))
2019-10-02 20:20:13.345 [INFO][25423] config_params.go 373: Skipping config value for LogSeverityScreen from datastore (global); already have a value from environment variable
2019-10-02 20:20:13.345 [INFO][25423] config_params.go 334: Parsing value for IpInIpEnabled: true (from datastore (global))
2019-10-02 20:20:13.346 [INFO][25423] config_params.go 370: Parsed value for IpInIpEnabled: true (from datastore (global))
2019-10-02 20:20:13.346 [INFO][25423] config_params.go 334: Parsing value for ReportingIntervalSecs: 0 (from datastore (global))
2019-10-02 20:20:13.346 [INFO][25423] config_params.go 370: Parsed value for ReportingIntervalSecs: 0s (from datastore (global))
2019-10-02 20:20:13.348 [INFO][25423] k8s.go 228: Using Calico IPAM
2019-10-02 20:20:13.348 [INFO][25423] health.go 133: Health of component changed lastReport=health.HealthReport{Live:true, Ready:false} name="felix-startup" newReport=&health.HealthReport{Live:true, Ready:true}
2019-10-02 20:20:13.348 [INFO][25423] health.go 230: Health enabled.  Server is already running. host="localhost" port=9099
2019-10-02 20:20:13.348 [INFO][25423] daemon.go 294: Successfully loaded configuration. GOMAXPROCS=12 builddate="7a042b762bc3a20b28a712e22fcc72160e4b5ddf" config=&config.Config{UseInternalDataplaneDriver:true, DataplaneDriver:"calico-iptables-plugin", DatastoreType:"kubernetes", FelixHostname:"upsilon", EtcdAddr:"127.0.0.1:2379", EtcdScheme:"http", EtcdKeyFile:"", EtcdCertFile:"", EtcdCaFile:"", EtcdEndpoints:[]string(nil), TyphaAddr:"", TyphaK8sServiceName:"", TyphaK8sNamespace:"kube-system", TyphaReadTimeout:30000000000, TyphaWriteTimeout:10000000000, TyphaKeyFile:"", TyphaCertFile:"", TyphaCAFile:"", TyphaCN:"", TyphaURISAN:"", Ipv6Support:false, IgnoreLooseRPF:false, IptablesBackend:"legacy", RouteRefreshInterval:90000000000, DeviceRouteSourceAddress:net.IP(nil), DeviceRouteProtocol:3, RemoveExternalRoutes:true, IptablesRefreshInterval:90000000000, IptablesPostWriteCheckIntervalSecs:1000000000, IptablesLockFilePath:"/run/xtables.lock", IptablesLockTimeoutSecs:0, IptablesLockProbeIntervalMillis:50000000, IpsetsRefreshInterval:10000000000, MaxIpsetSize:1048576, XDPRefreshInterval:90000000000, PolicySyncPathPrefix:"", NetlinkTimeoutSecs:10000000000, MetadataAddr:"", MetadataPort:8775, OpenstackRegion:"", InterfacePrefix:"cali", InterfaceExclude:[]*regexp.Regexp{(*regexp.Regexp)(0xc0000d9560)}, ChainInsertMode:"insert", DefaultEndpointToHostAction:"ACCEPT", IptablesFilterAllowAction:"ACCEPT", IptablesMangleAllowAction:"ACCEPT", LogPrefix:"calico-packet", LogFilePath:"", LogSeverityFile:"", LogSeverityScreen:"INFO", LogSeveritySys:"", VXLANEnabled:false, VXLANPort:4789, VXLANVNI:4096, VXLANMTU:1410, IPv4VXLANTunnelAddr:net.IP(nil), VXLANTunnelMACAddr:"", IpInIpEnabled:true, IpInIpMtu:1440, IpInIpTunnelAddr:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xc0, 0xa8, 0x21, 0x80}, ReportingIntervalSecs:0, ReportingTTLSecs:90000000000, EndpointReportingEnabled:false, EndpointReportingDelaySecs:1000000000, IptablesMarkMask:0xffff0000, DisableConntrackInvalidCheck:false, HealthEnabled:true, HealthPort:9099, HealthHost:"localhost", PrometheusMetricsEnabled:false, PrometheusMetricsHost:"", PrometheusMetricsPort:9091, PrometheusGoMetricsEnabled:true, PrometheusProcessMetricsEnabled:true, FailsafeInboundHostPorts:[]config.ProtoPort{config.ProtoPort{Protocol:"tcp", Port:0x16}, config.ProtoPort{Protocol:"udp", Port:0x44}, config.ProtoPort{Protocol:"tcp", Port:0xb3}, config.ProtoPort{Protocol:"tcp", Port:0x94b}, config.ProtoPort{Protocol:"tcp", Port:0x94c}, config.ProtoPort{Protocol:"tcp", Port:0x1a0a}, config.ProtoPort{Protocol:"tcp", Port:0x1a0b}}, FailsafeOutboundHostPorts:[]config.ProtoPort{config.ProtoPort{Protocol:"udp", Port:0x35}, config.ProtoPort{Protocol:"udp", Port:0x43}, config.ProtoPort{Protocol:"tcp", Port:0xb3}, config.ProtoPort{Protocol:"tcp", Port:0x94b}, config.ProtoPort{Protocol:"tcp", Port:0x94c}, config.ProtoPort{Protocol:"tcp", Port:0x1a0a}, config.ProtoPort{Protocol:"tcp", Port:0x1a0b}}, KubeNodePortRanges:[]numorstring.Port{numorstring.Port{MinPort:0x7530, MaxPort:0x7fff, PortName:""}}, NATPortRange:numorstring.Port{MinPort:0x0, MaxPort:0x0, PortName:""}, NATOutgoingAddress:net.IP(nil), UsageReportingEnabled:true, UsageReportingInitialDelaySecs:300000000000, UsageReportingIntervalSecs:86400000000000, ClusterGUID:"76ffcce211b64b55b8650bdf51ce73ae", ClusterType:"k8s,bgp,kdd", CalicoVersion:"v3.9.1", ExternalNodesCIDRList:[]string(nil), DebugMemoryProfilePath:"", DebugCPUProfilePath:"/tmp/felix-cpu-<timestamp>.pprof", DebugDisableLogDropping:false, DebugSimulateCalcGraphHangAfter:0, DebugSimulateDataplaneHangAfter:0, sourceToRawConfig:map[config.Source]map[string]string{0x1:map[string]string{"CalicoVersion":"v3.9.1", "ClusterGUID":"76ffcce211b64b55b8650bdf51ce73ae", "ClusterType":"k8s,bgp,kdd", "IpInIpEnabled":"true", "LogSeverityScreen":"Info", "ReportingIntervalSecs":"0"}, 0x2:map[string]string{"IpInIpTunnelAddr":"192.168.33.128"}, 0x3:map[string]string{"LogFilePath":"None", "LogSeverityFile":"None", "LogSeveritySys":"None", "MetadataAddr":"None"}, 0x4:map[string]string{"datastoretype":"kubernetes", "defaultendpointtohostaction":"ACCEPT", "felixhostname":"upsilon", "healthenabled":"true", "ipinipmtu":"1440", "ipv6support":"false", "logseverityscreen":"info"}}, rawValues:map[string]string{"CalicoVersion":"v3.9.1", "ClusterGUID":"76ffcce211b64b55b8650bdf51ce73ae", "ClusterType":"k8s,bgp,kdd", "DatastoreType":"kubernetes", "DefaultEndpointToHostAction":"ACCEPT", "FelixHostname":"upsilon", "HealthEnabled":"true", "IpInIpEnabled":"true", "IpInIpMtu":"1440", "IpInIpTunnelAddr":"192.168.33.128", "Ipv6Support":"false", "LogFilePath":"None", "LogSeverityFile":"None", "LogSeverityScreen":"info", "LogSeveritySys":"None", "MetadataAddr":"None", "ReportingIntervalSecs":"0"}, Err:error(nil), IptablesNATOutgoingInterfaceFilter:"", SidecarAccelerationEnabled:false, XDPEnabled:true, GenericXDPEnabled:false, loadClientConfigFromEnvironment:(func() (*apiconfig.CalicoAPIConfig, error))(0x10477e0), useNodeResourceUpdates:false} gitcommit="2019-09-26T17:59:26+0000" version="v3.9.1"
2019-10-02 20:20:13.350 [INFO][25423] driver.go 43: Using internal (linux) dataplane driver.
2019-10-02 20:20:13.350 [INFO][25423] driver.go 85: Calculated iptables mark bits acceptMark=0x10000 endpointMark=0xfff00000 endpointMarkNonCali=0x100000 passMark=0x20000 scratch0Mark=0x40000 scratch1Mark=0x80000
2019-10-02 20:20:13.350 [INFO][25423] int_dataplane.go 240: Creating internal dataplane driver. config=intdataplane.Config{Hostname:"upsilon", IPv6Enabled:false, RuleRendererOverride:rules.RuleRenderer(nil), IPIPMTU:1440, VXLANMTU:1410, IgnoreLooseRPF:false, MaxIPSetSize:1048576, IptablesBackend:"legacy", IPSetsRefreshInterval:10000000000, RouteRefreshInterval:90000000000, DeviceRouteSourceAddress:net.IP(nil), DeviceRouteProtocol:3, RemoveExternalRoutes:true, IptablesRefreshInterval:90000000000, IptablesPostWriteCheckInterval:1000000000, IptablesInsertMode:"insert", IptablesLockFilePath:"/run/xtables.lock", IptablesLockTimeout:0, IptablesLockProbeInterval:50000000, XDPRefreshInterval:90000000000, NetlinkTimeout:10000000000, RulesConfig:rules.Config{IPSetConfigV4:(*ipsets.IPVersionConfig)(0xc0000be7d0), IPSetConfigV6:(*ipsets.IPVersionConfig)(0xc0000be910), WorkloadIfacePrefixes:[]string{"cali"}, IptablesMarkAccept:0x10000, IptablesMarkPass:0x20000, IptablesMarkScratch0:0x40000, IptablesMarkScratch1:0x80000, IptablesMarkEndpoint:0xfff00000, IptablesMarkNonCaliEndpoint:0x100000, KubeNodePortRanges:[]numorstring.Port{numorstring.Port{MinPort:0x7530, MaxPort:0x7fff, PortName:""}}, KubeIPVSSupportEnabled:false, OpenStackMetadataIP:net.IP(nil), OpenStackMetadataPort:0x2247, OpenStackSpecialCasesEnabled:false, VXLANEnabled:false, VXLANPort:4789, VXLANVNI:4096, IPIPEnabled:true, IPIPTunnelAddress:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xc0, 0xa8, 0x21, 0x80}, VXLANTunnelAddress:net.IP(nil), IptablesLogPrefix:"calico-packet", EndpointToHostAction:"ACCEPT", IptablesFilterAllowAction:"ACCEPT", IptablesMangleAllowAction:"ACCEPT", FailsafeInboundHostPorts:[]config.ProtoPort{config.ProtoPort{Protocol:"tcp", Port:0x16}, config.ProtoPort{Protocol:"udp", Port:0x44}, config.ProtoPort{Protocol:"tcp", Port:0xb3}, config.ProtoPort{Protocol:"tcp", Port:0x94b}, config.ProtoPort{Protocol:"tcp", Port:0x94c}, config.ProtoPort{Protocol:"tcp", Port:0x1a0a}, config.ProtoPort{Protocol:"tcp", Port:0x1a0b}}, FailsafeOutboundHostPorts:[]config.ProtoPort{config.ProtoPort{Protocol:"udp", Port:0x35}, config.ProtoPort{Protocol:"udp", Port:0x43}, config.ProtoPort{Protocol:"tcp", Port:0xb3}, config.ProtoPort{Protocol:"tcp", Port:0x94b}, config.ProtoPort{Protocol:"tcp", Port:0x94c}, config.ProtoPort{Protocol:"tcp", Port:0x1a0a}, config.ProtoPort{Protocol:"tcp", Port:0x1a0b}}, DisableConntrackInvalid:false, NATPortRange:numorstring.Port{MinPort:0x0, MaxPort:0x0, PortName:""}, IptablesNATOutgoingInterfaceFilter:"", NATOutgoingAddress:net.IP(nil)}, IfaceMonitorConfig:ifacemonitor.Config{InterfaceExcludes:[]*regexp.Regexp{(*regexp.Regexp)(0xc0000d9560)}}, StatusReportingInterval:0, ConfigChangedRestartCallback:(func())(0x15f81e0), PostInSyncCallback:(func())(0x15dced0), HealthAggregator:(*health.HealthAggregator)(0xc000540000), DebugSimulateDataplaneHangAfter:0, ExternalNodesCidrs:[]string(nil), XDPEnabled:true, XDPAllowGeneric:false, SidecarAccelerationEnabled:false, LookPathOverride:(func(string) (string, error))(nil)}
2019-10-02 20:20:13.350 [INFO][25423] rule_defs.go 294: Creating rule renderer. config=rules.Config{IPSetConfigV4:(*ipsets.IPVersionConfig)(0xc0000be7d0), IPSetConfigV6:(*ipsets.IPVersionConfig)(0xc0000be910), WorkloadIfacePrefixes:[]string{"cali"}, IptablesMarkAccept:0x10000, IptablesMarkPass:0x20000, IptablesMarkScratch0:0x40000, IptablesMarkScratch1:0x80000, IptablesMarkEndpoint:0xfff00000, IptablesMarkNonCaliEndpoint:0x100000, KubeNodePortRanges:[]numorstring.Port{numorstring.Port{MinPort:0x7530, MaxPort:0x7fff, PortName:""}}, KubeIPVSSupportEnabled:false, OpenStackMetadataIP:net.IP(nil), OpenStackMetadataPort:0x2247, OpenStackSpecialCasesEnabled:false, VXLANEnabled:false, VXLANPort:4789, VXLANVNI:4096, IPIPEnabled:true, IPIPTunnelAddress:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xc0, 0xa8, 0x21, 0x80}, VXLANTunnelAddress:net.IP(nil), IptablesLogPrefix:"calico-packet", EndpointToHostAction:"ACCEPT", IptablesFilterAllowAction:"ACCEPT", IptablesMangleAllowAction:"ACCEPT", FailsafeInboundHostPorts:[]config.ProtoPort{config.ProtoPort{Protocol:"tcp", Port:0x16}, config.ProtoPort{Protocol:"udp", Port:0x44}, config.ProtoPort{Protocol:"tcp", Port:0xb3}, config.ProtoPort{Protocol:"tcp", Port:0x94b}, config.ProtoPort{Protocol:"tcp", Port:0x94c}, config.ProtoPort{Protocol:"tcp", Port:0x1a0a}, config.ProtoPort{Protocol:"tcp", Port:0x1a0b}}, FailsafeOutboundHostPorts:[]config.ProtoPort{config.ProtoPort{Protocol:"udp", Port:0x35}, config.ProtoPort{Protocol:"udp", Port:0x43}, config.ProtoPort{Protocol:"tcp", Port:0xb3}, config.ProtoPort{Protocol:"tcp", Port:0x94b}, config.ProtoPort{Protocol:"tcp", Port:0x94c}, config.ProtoPort{Protocol:"tcp", Port:0x1a0a}, config.ProtoPort{Protocol:"tcp", Port:0x1a0b}}, DisableConntrackInvalid:false, NATPortRange:numorstring.Port{MinPort:0x0, MaxPort:0x0, PortName:""}, IptablesNATOutgoingInterfaceFilter:"", NATOutgoingAddress:net.IP(nil)}
2019-10-02 20:20:13.350 [INFO][25423] rule_defs.go 304: Workload to host packets will be accepted.
2019-10-02 20:20:13.350 [INFO][25423] rule_defs.go 318: filter table allowed packets will be accepted immediately.
2019-10-02 20:20:13.351 [INFO][25423] rule_defs.go 326: mangle table allowed packets will be accepted immediately.
2019-10-02 20:20:13.353 [INFO][25423] feature_detect.go 110: Updating detected iptables features features=iptables.Features{SNATFullyRandom:true, MASQFullyRandom:true, RestoreSupportsLock:true} iptablesVersion=1.8.2 kernelVersion=5.0.0
2019-10-02 20:20:13.354 [INFO][25423] table.go 444: Looked up iptables command backendMode="legacy" candidates=[]string{"iptables-legacy-restore", "iptables-restore"} command="iptables-legacy-restore" ipVersion=0x4 saveOrRestore="restore"
2019-10-02 20:20:13.354 [INFO][25423] table.go 444: Looked up iptables command backendMode="legacy" candidates=[]string{"iptables-legacy-save", "iptables-save"} command="iptables-legacy-save" ipVersion=0x4 saveOrRestore="save"
2019-10-02 20:20:13.354 [INFO][25423] table.go 444: Looked up iptables command backendMode="legacy" candidates=[]string{"iptables-legacy-restore", "iptables-restore"} command="iptables-legacy-restore" ipVersion=0x4 saveOrRestore="restore"
2019-10-02 20:20:13.354 [INFO][25423] table.go 444: Looked up iptables command backendMode="legacy" candidates=[]string{"iptables-legacy-save", "iptables-save"} command="iptables-legacy-save" ipVersion=0x4 saveOrRestore="save"
2019-10-02 20:20:13.354 [INFO][25423] table.go 444: Looked up iptables command backendMode="legacy" candidates=[]string{"iptables-legacy-restore", "iptables-restore"} command="iptables-legacy-restore" ipVersion=0x4 saveOrRestore="restore"
2019-10-02 20:20:13.354 [INFO][25423] table.go 444: Looked up iptables command backendMode="legacy" candidates=[]string{"iptables-legacy-save", "iptables-save"} command="iptables-legacy-save" ipVersion=0x4 saveOrRestore="save"
2019-10-02 20:20:13.355 [INFO][25423] table.go 444: Looked up iptables command backendMode="legacy" candidates=[]string{"iptables-legacy-restore", "iptables-restore"} command="iptables-legacy-restore" ipVersion=0x4 saveOrRestore="restore"
2019-10-02 20:20:13.355 [INFO][25423] table.go 444: Looked up iptables command backendMode="legacy" candidates=[]string{"iptables-legacy-save", "iptables-save"} command="iptables-legacy-save" ipVersion=0x4 saveOrRestore="save"
2019-10-02 20:20:13.355 [INFO][25423] int_dataplane.go 355: Checking if we need to clean up the VXLAN device
2019-10-02 20:20:13.355 [WARNING][25423] int_dataplane.go 357: Failed to query VXLAN device error=Link not found
2019-10-02 20:20:13.357 [INFO][25423] int_dataplane.go 377: XDP acceleration enabled.
2019-10-02 20:20:13.362 [INFO][25423] route_table.go 170: Calculated interface name regexp regex="^cali.*"
2019-10-02 20:20:13.362 [INFO][25423] ipsets.go 119: Queueing IP set for creation family="inet" setID="all-ipam-pools" setType="hash:net"
2019-10-02 20:20:13.362 [INFO][25423] ipsets.go 119: Queueing IP set for creation family="inet" setID="masq-ipam-pools" setType="hash:net"
2019-10-02 20:20:13.362 [INFO][25423] int_dataplane.go 539: Registering to report health.
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-failsafe-in" ipVersion=0x4 table="raw"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-failsafe-out" ipVersion=0x4 table="raw"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-PREROUTING" ipVersion=0x4 table="raw"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-OUTPUT" ipVersion=0x4 table="raw"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-FORWARD" ipVersion=0x4 table="filter"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-INPUT" ipVersion=0x4 table="filter"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-wl-to-host" ipVersion=0x4 table="filter"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-failsafe-in" ipVersion=0x4 table="filter"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-OUTPUT" ipVersion=0x4 table="filter"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-failsafe-out" ipVersion=0x4 table="filter"
2019-10-02 20:20:13.363 [INFO][25423] int_dataplane.go 697: IPIP enabled, starting thread to keep tunnel configuration in sync.
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-PREROUTING" ipVersion=0x4 table="nat"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-POSTROUTING" ipVersion=0x4 table="nat"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-OUTPUT" ipVersion=0x4 table="nat"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-failsafe-in" ipVersion=0x4 table="mangle"
2019-10-02 20:20:13.363 [INFO][25423] ipip_mgr.go 85: IPIP thread started.
2019-10-02 20:20:13.364 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-PREROUTING" ipVersion=0x4 table="mangle"
2019-10-02 20:20:13.366 [WARNING][25423] int_dataplane.go 728: failed to set XDP failsafe ports, disabling XDP: failed to create map (calico_failsafe_ports_v1): exit status 255
Error: map create failed: Operation not permitted

2019-10-02 20:20:13.436 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-020661529): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-020661529'
Error: failed to load object file
 try=0
2019-10-02 20:20:13.504 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-216357796): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-216357796'
Error: failed to load object file
 try=1
2019-10-02 20:20:13.560 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-340555443): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-340555443'
Error: failed to load object file
 try=2
2019-10-02 20:20:13.624 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-394035830): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-394035830'
Error: failed to load object file
 try=3
2019-10-02 20:20:13.692 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-882563421): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-882563421'
Error: failed to load object file
 try=4
2019-10-02 20:20:13.760 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-573287704): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-573287704'
Error: failed to load object file
 try=5
2019-10-02 20:20:13.832 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-399081367): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-399081367'
Error: failed to load object file
 try=6
2019-10-02 20:20:13.896 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-183507978): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-183507978'
Error: failed to load object file
 try=7
2019-10-02 20:20:13.960 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-851860449): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-851860449'
Error: failed to load object file
 try=8
2019-10-02 20:20:14.016 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-524674508): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-524674508'
Error: failed to load object file
 try=9
2019-10-02 20:20:14.016 [PANIC][25423] int_dataplane.go 784: Failed to wipe the XDP state after 10 tries
panic: (*logrus.Entry) (0x19b44a0,0xc0006637c0)

goroutine 1 [running]:
github.com/sirupsen/logrus.Entry.log(0xc0000be050, 0xc00035fa10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7f6700000000, ...)
    /go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180627202928-fc9bbf2f57995271c5cd6911ede7a2ebc5ea7c6f/entry.go:112 +0x2d2
github.com/sirupsen/logrus.(*Entry).Panic(0xc000663180, 0xc0003d2290, 0x1, 0x1)
    /go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180627202928-fc9bbf2f57995271c5cd6911ede7a2ebc5ea7c6f/entry.go:182 +0x103
github.com/sirupsen/logrus.(*Entry).Panicf(0xc000663180, 0x1a336bb, 0x2b, 0xc0003d2340, 0x1, 0x1)
    /go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180627202928-fc9bbf2f57995271c5cd6911ede7a2ebc5ea7c6f/entry.go:230 +0xd4
github.com/sirupsen/logrus.(*Logger).Panicf(0xc0000be050, 0x1a336bb, 0x2b, 0xc0003d2340, 0x1, 0x1)
    /go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180627202928-fc9bbf2f57995271c5cd6911ede7a2ebc5ea7c6f/logger.go:173 +0x86
github.com/sirupsen/logrus.Panicf(...)
    /go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180627202928-fc9bbf2f57995271c5cd6911ede7a2ebc5ea7c6f/exported.go:145
github.com/projectcalico/felix/dataplane/linux.(*InternalDataplane).shutdownXDPCompletely(0xc00031f200)
    /go/pkg/mod/github.com/projectcalico/[email protected]/dataplane/linux/int_dataplane.go:784 +0x2cd
github.com/projectcalico/felix/dataplane/linux.(*InternalDataplane).doStaticDataplaneConfig(0xc00031f200)
    /go/pkg/mod/github.com/projectcalico/[email protected]/dataplane/linux/int_dataplane.go:729 +0xbaa
github.com/projectcalico/felix/dataplane/linux.(*InternalDataplane).Start(0xc00031f200)
    /go/pkg/mod/github.com/projectcalico/[email protected]/dataplane/linux/int_dataplane.go:592 +0x2f
github.com/projectcalico/felix/dataplane.StartDataplaneDriver(0xc0005ccf00, 0xc000540000, 0xc000607100, 0x1, 0xc00039d7d8, 0x0)
    /go/pkg/mod/github.com/projectcalico/[email protected]/dataplane/driver.go:186 +0xf09
github.com/projectcalico/felix/daemon.Run(0x1a05d48, 0x15, 0x1cbbc38, 0x6, 0x1d106e0, 0x28, 0x1ce87a0, 0x18)
    /go/pkg/mod/github.com/projectcalico/[email protected]/daemon/daemon.go:305 +0x1759
main.main()
    /go/src/github.com/projectcalico/node/cmd/calico-node/main.go:100 +0x405

Steps to Reproduce (for bugs)

(following https://docs.projectcalico.org/v3.9/getting-started/kubernetes/ )

1. Install cri-o from Ubuntu PPA: https://launchpad.net/~projectatomic/+archive/ubuntu/ppa
2. kubeadm init --pod-network-cidr=10.100.0.1/24
3. kubectl apply -f https://docs.projectcalico.org/v3.9/manifests/calico.yaml

Context

Trying to install Calico on a completely fresh kube install.

Your Environment

• Calico version 3.9.1
• Orchestrator version (e.g. kubernetes, mesos, rkt): Kubernetes 1.16.0, cri-o 1.13
• Operating System and version: Ubunto 19.04

[caseydavenport]

@thommay could you share the calico/node logs? That should show why Felix isn't becoming ready.

[thommay]

@caseydavenport they're in the "click for kubectl logs output" bit.

[fasaxc]

I had a go at setting up a CRIO system but went down a rabbit-hole and failed to find a working set of instructions. Can you share the exact steps you used?

[thommay]

@fasaxc sure:

sudo add-apt-repository ppa:projectatomic/ppa
sudo apt install cri-o-1.13
sudoedit /etc/crio/crio.conf 
 # go to line 258 and uncomment the registries stanza
sudo systemctl start crio 

follow https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm

At https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#configure-cgroup-driver-used-by-kubelet-on-control-plane-node
the cgroup driver is systemd.

kubeadm should now automatically use cri-o.

[caseydavenport]

2019-10-02 20:20:13.436 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-020661529): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-020661529'
Error: failed to load object file
 try=0

Looks like we're seeing the same thing as in this issue: https://github.com/projectcalico/calico/issues/2904

[tmjd]

The fix for this should be in v3.11. https://github.com/projectcalico/felix/pull/2165

[54853315]

这个问题其实在安装完kubeadm init时就已经提示过：

[init] Using Kubernetes version: v1.16.3
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster

大家容易忽略。

我把解决办法写出来了，创建一个文件，kubele会去读的：
vim /etc/default/kubele
写入：
Environment=KUBELET_EXTRA_ARGS=--cgroup-driver=systemd

然后删除calico-node-[name]那个pod容器，等它几秒重新创建完，ok！问题解决。

Guys, just follow the code zone ,It will finish your problem.

Have a nice day .