Linkerd2: Helm install causing nil pointer dereference

[shahriak]

Bug Report

What is the issue?

I installed linkerd using Helm command. After installation, Linkerd check command is showing this error "panic: runtime error: invalid memory address or nil pointer dereference.."

How can it be reproduced?

Create certs using step cli:

trust anchor

step certificate create identity.linkerd.cluster.local ca.crt ca.key \
--profile root-ca --no-password --insecure --san identity.linkerd.cluster.local --not-after=87600h

issuer cert

step certificate create identity.linkerd.cluster.local issuer.crt issuer.key --ca ca.crt --ca-key ca.key --profile intermediate-ca --not-after 8760h --no-password --insecure --san identity.linkerd.cluster.local

Create a values.yaml file overrding global.identityTrustAnchorsPEM, identity.issuer.tls.crtPEM, identity.issuer.tls.keyPEM
remove "|" for the above fields from the main values file with the certs generated.

Example:

global:
  # Trust root certificate
  identityTrustAnchorsPEM: "-----BEGIN CERTIFICATE-----\nMIIBlTCCATugAwIBAgIQO84X4kihga8u/wsLrhzCGDAKBggqhkjOPQQDAjApMScw\nJQYDVQQDEx5pZGVudGl0eS5saW5rZXJkLmNsdXN0ZXIubG9jYWwwHhcNMjAxMDIz\nMjA0ODMyWhcNMzAxMDIxMjA0ODMyWjApMScwJQYDVQQDEx5pZGVudGl0eS5saW5r\nZXJkLmNsdXN0ZXIubG9jYWwwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAATuXJRk\nRQTASGfewS5BlnavSbYqaf6nVX06k7xb85XBsLsfC8L5zRog20riJ10wou9FskxL\n7gFybvANu/nCluWVo0UwQzAOBgNVHQ8BAf8EBAMCAQYwEgYDVR0TAQH/BAgwBgEB\n/wIBATAdBgNVHQ4EFgQUgfPPpruejEVJwtg+VErrchhvEpgwCgYIKoZIzj0EAwID\nSAAwRQIgDkYTtGnxc5hwLCk1pnlqKPdxRWPNMLcbSJaujX6tTFoCIQCvnKizQkQt\n+okJrbYsXs0uAdVVqkOCfUbMHliGsUh9mQ==\n-----END CERTIFICATE-----"

# identity configuration
identity:
  issuer:
    # issuer cert and key
    tls:
      # PEM-encoded certificate
      crtPEM: "-----BEGIN CERTIFICATE-----\nMIIBljCCATygAwIBAgIRAIl/b2HkHci2oYVXY75AkOgwCgYIKoZIzj0EAwIwKTEn\nMCUGA1UEAxMeaWRlbnRpdHkubGlua2VyZC5jbHVzdGVyLmxvY2FsMB4XDTIwMTAy\nMzIwNDg0NFoXDTIxMTAyMzIwNDg0NFowKTEnMCUGA1UEAxMeaWRlbnRpdHkubGlu\na2VyZC5jbHVzdGVyLmxvY2FsMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAECt7F\nIJ6NM4r0ASVyBU6ntBZxyXqbD/2lH8kkylzByXAIoDuh9C0ZJuVlsgGBDuLe6oA2\ntAAgU4GqRZ+254s7DaNFMEMwDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQIMAYB\nAf8CAQAwHQYDVR0OBBYEFLJEv+fzByBBEGRSBxcGbZzUfzhgMAoGCCqGSM49BAMC\nA0gAMEUCIEI7YLDYKBUbXx7XzFqJIBd7bW1u/iY+M54EGb83GHpIAiEA2iQeGRY9\nrami+AuZnGtR3BMQ5IToDbGdv8EhOIxYh4s=\n-----END CERTIFICATE-----"

      # PEM-encoded ECDSA private key
      keyPEM: "-----BEGIN EC PRIVATE KEY-----\nMHcCAQEEIK7fZgVKIi5pCMsNCL+NZLH3dZYC7HibEy7ZMT9eb8JdoAoGCCqGSM49\nAwEHoUQDQgAECt7FIJ6NM4r0ASVyBU6ntBZxyXqbD/2lH8kkylzByXAIoDuh9C0Z\nJuVlsgGBDuLe6oA2tAAgU4GqRZ+254s7DQ==\n-----END EC PRIVATE KEY-----"

change identity.yaml file expirtyAnnotation to this: {{.Values.identity.issuer.crtExpiryAnnotation}}: {{ now | date_modify "+8760h" | date "2006-01-02T15:04:05Z" }}

helm install -f values.yaml linkerd2 ./linkerd2

Logs, error output, etc

Pods are running as expected.

linkerd check output

```text

kubernetes-api

√ can initialize the client
√ can query the Kubernetes API

kubernetes-version

√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence

√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-config

√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x151c8d4]

goroutine 1 [running]:
github.com/linkerd/linkerd2/pkg/healthcheck.(HealthChecker).checkCertificatesConfig(0xc000851680, 0xc000725908, 0x431c52, 0x700000000, 0xc000095540, 0x1c2eec0, 0xc000725920)
/linkerd-build/pkg/healthcheck/healthcheck.go:1443 +0x84
github.com/linkerd/linkerd2/pkg/healthcheck.(HealthChecker).allCategories.func48(0x1c2eec0, 0xc00041b8c0, 0x6fc23ac00, 0x1c2eec0)
/linkerd-build/pkg/healthcheck/healthcheck.go:919 +0x33
github.com/linkerd/linkerd2/pkg/healthcheck.(HealthChecker).runCheck(0xc000851680, 0x1990a6c, 0x10, 0xc000725ac0, 0xc000725bf8, 0xc00016ad00)
/linkerd-build/pkg/healthcheck/healthcheck.go:1329 +0x14c
github.com/linkerd/linkerd2/pkg/healthcheck.(HealthChecker).RunChecks(0xc000851680, 0xc000725bf8, 0x4)
/linkerd-build/pkg/healthcheck/healthcheck.go:1298 +0x298
github.com/linkerd/linkerd2/cli/cmd.runChecksTable(0x1bed2a0, 0xc00000e018, 0xc000851680, 0x0)
/linkerd-build/cli/cmd/check.go:274 +0x12b
github.com/linkerd/linkerd2/cli/cmd.runChecks(0x1bed2a0, 0xc00000e018, 0x1bed2a0, 0xc00000e020, 0xc000851680, 0x1982af6, 0x5, 0x3)
/linkerd-build/cli/cmd/check.go:228 +0x5b
github.com/linkerd/linkerd2/cli/cmd.configureAndRunChecks(0x1bed2a0, 0xc00000e018, 0x1bed2a0, 0xc00000e020, 0x0, 0x0, 0xc0003a21e0, 0x0, 0x0)
/linkerd-build/cli/cmd/check.go:215 +0x5b1
github.com/linkerd/linkerd2/cli/cmd.newCmdCheck.func1(0xc00039e780, 0x294f400, 0x0, 0x0, 0x0, 0x0)
/linkerd-build/cli/cmd/check.go:135 +0x62
github.com/spf13/cobra.(Command).execute(0xc00039e780, 0x294f400, 0x0, 0x0, 0xc00039e780, 0x294f400)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:826 +0x453
github.com/spf13/cobra.(Command).ExecuteC(0x2902ba0, 0x0, 0x16cea80, 0xc000096058)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
main.main()
/linkerd-build/cli/main.go:10 +0x2d```

Environment

• Kubernetes Version:
  Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:12:48Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"964014daa6eaebac41a54a8e52906c3b7794c4af", GitTreeState:"clean", BuildDate:"2020-09-17T04:13:20Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
• Cluster Environment: (GKE, AKS, kops, ...)
  AKS
• Host OS:
  Ubuntu
• Linkerd version:
  Client version: stable-2.8.1
  Server version: edge-20.10.3

Possible solution

Additional context

[cpretzer]

Hi @shahriak it looks like the client version is different than the server version:

Client version: stable-2.8.1
Server version: edge-20.10.3

Can you try the linkerd check command with CLI version edge-20.10.3?

[cpretzer]

@shahriak it looks like this commit introduces an incompatibility between the CLI and the control plane versions that you're using.

This error won't happen with CLI edge-20.10.3 or later.

More broadly, it's recommended to use the same CLI and control plane versions for edge releases.

[shahriak]

How do we specify what version of cli to install? Seems like curl -sL https://run.linkerd.io/install-edge | sh command installs the latest version which 20.10.5

[cpretzer]

The binaries for edge-20.10.3 are here.

[shahriak]

I am able to verify using the latest control plane version 20.10.5. Closing the issue

[cpretzer]

great! thanks @shahriak I've made a note to add this to the docs