Calico: [ERROR]: Observed a panic: "assignment to entry in nil map"

[bonikforever]

After some steps migration job failed with error:

2019-11-11 14:44:24.286 [INFO][1] ipam_migrator.go 192: 5 nodes completed IPAM migration process.
E1111 14:44:24.293258       1 runtime.go:78] Observed a panic: "assignment to entry in nil map" (assignment to entry in nil map)
goroutine 16 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1668680, 0x1ab7000)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x82
panic(0x1668680, 0x1ab7000)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/projectcalico/kube-controllers/pkg/controllers/flannelmigration.daemonset.AddNodeSelector(0x16209ae, 0x15, 0xc000410a00, 0x184ee8a, 0xb, 0xc000140c90, 0x0, 0x0)
        /go/src/pkg/controllers/flannelmigration/k8s_resources.go:173 +0x224
github.com/projectcalico/kube-controllers/pkg/controllers/flannelmigration.(*flannelMigrationController).Run(0xc000690aa0, 0x0, 0x157cd96, 0x2, 0xc000674780)
        /go/src/pkg/controllers/flannelmigration/migration_controller.go:176 +0x4ed
created by main.(*controllerControl).RunControllers
        /go/src/cmd/kube-controllers/main.go:365 +0x187
panic: assignment to entry in nil map [recovered]
        panic: assignment to entry in nil map

goroutine 16 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:55 +0x105
panic(0x1668680, 0x1ab7000)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/projectcalico/kube-controllers/pkg/controllers/flannelmigration.daemonset.AddNodeSelector(0x16209ae, 0x15, 0xc000410a00, 0x184ee8a, 0xb, 0xc000140c90, 0x0, 0x0)
        /go/src/pkg/controllers/flannelmigration/k8s_resources.go:173 +0x224
github.com/projectcalico/kube-controllers/pkg/controllers/flannelmigration.(*flannelMigrationController).Run(0xc000690aa0, 0x0, 0x157cd96, 0x2, 0xc000674780)
        /go/src/pkg/controllers/flannelmigration/migration_controller.go:176 +0x4ed
created by main.(*controllerControl).RunControllers
        /go/src/cmd/kube-controllers/main.go:365 +0x187

Expected Behavior

Job compeled succesfully.

Current Behavior

Job failed.

Steps to Reproduce (for bugs)

Run migration job.

Context

Trying to migrate cluster from flannel to calico.

Your Environment

Calico version: 3.10, 3.9
Kubernetes: 1.15.2, 1.16.2
RHEL 7

[tmjd]

@song-jiang any thoughts on this?

[song-jiang]

@tmjd I can reproduce it with 1.16. Not sure what is happening but will look into this issue.
@bonikforever Have you seen this issue with 1.15.2 as well?

[bonikforever]

@song-jiang yes, I've tried to migrate on 1.15.2 and got this error. Then I upgrade to 1.16.2 and got it again.

[song-jiang]

@bonikforever Thanks for the information. I believe we did test flannel migration on 1.15. Let me check and come back to you this week.

[song-jiang]

@bonikforever I've raised PR to fix this issue. If you want to work around it while waiting for next Calico release. Here is the steps that I tested successfully on my rig.

• Remove previous migration job by kubectl delete -f migration-job.yaml.
• Create a flannel-ds-patch.yaml file with contents below.

spec:
  template:
    spec:
      nodeSelector:
        beta.kubernetes.io/os: linux

• Run

kubectl patch daemonset kube-flannel-ds-amd64 -n kube-system --type merge --patch "$(cat flannel-ds-patch.yaml)"

In my case, flannel daemonset has the name kube-flannel-ds-amd64. Once flannel daemonset is patched, kube-flannel pod starts to restart itself one by one on each node. Wait till all kube-flannel pods are running.

• Apply migration-job.yaml manifest again.

[bonikforever]

@song-jiang well done on 1.16.2. Thanks!

[song-jiang]

@bonikforever Does the workaround work for you on 1.16.2?

[bonikforever]

@song-jiang yes, succesfully migrated after patch.