Issue Type:
What happened: When I upgrade (or install from scratch) using the CLI to linkerd 2.8.1 I start getting the following error:
2020-07-30T22:47:11.4898911Z ##[error]Bash wrote one or more lines to the standard error stream.
2020-07-30T22:47:11.4909905Z ##[error]time="2020-07-30T22:44:51Z" level=error msg="could not set namespace from kubectl context: ensure a valid KUBECONFIG path has been set"
2020-07-30T22:47:11.4911647Z ##[error]time="2020-07-30T22:44:53Z" level=error msg="could not set namespace from kubectl context: ensure a valid KUBECONFIG path has been set"
2020-07-30T22:47:11.4912834Z ##[error]time="2020-07-30T22:47:03Z" level=error msg="could not set namespace from kubectl context: ensure a valid KUBECONFIG path has been set"
What you expected to happen: no error.
How to reproduce it (as minimally and precisely as possible): Use 2.8.1 CLI to install/upgrade linkerd on AKS 1.18.02
# install command
$linkerd2_target_bin install --kubeconfig="$KUBECONFIG_FILE_NAME" --ha --controller-replicas=2 | $KUBECTL_BIN --kubeconfig="$KUBECONFIG_FILE_NAME" apply -f -
# upgrade command
$linkerd2_target_bin upgrade --kubeconfig="$KUBECONFIG_FILE_NAME" --ha --controller-replicas=2 2>/dev/null | $KUBECTL_BIN --kubeconfig="$KUBECONFIG_FILE_NAME" apply -f -
Anything else we need to know?:
2020-07-30T22:44:51.9647484Z time="2020-07-30T22:44:51Z" level=error msg="could not set namespace from kubectl context: ensure a valid KUBECONFIG path has been set"
2020-07-30T22:44:53.2489277Z time="2020-07-30T22:44:53Z" level=error msg="could not set namespace from kubectl context: ensure a valid KUBECONFIG path has been set"
2020-07-30T22:44:54.0942312Z Running linkerd version: Server version: stable-2.7.1
2020-07-30T22:44:54.0943203Z Linkerd attempting to upgrade Server version: stable-2.7.1 to stable-2.8.1
2020-07-30T22:44:54.1399004Z kubernetes-api
2020-07-30T22:44:54.1400279Z --------------
2020-07-30T22:44:54.1402288Z β can initialize the client
2020-07-30T22:44:54.3266788Z β can query the Kubernetes API
2020-07-30T22:44:54.3267559Z
2020-07-30T22:44:54.3268673Z kubernetes-version
2020-07-30T22:44:54.3269146Z ------------------
2020-07-30T22:44:54.3269618Z β is running the minimum Kubernetes API version
2020-07-30T22:44:54.7476438Z β is running the minimum kubectl version
2020-07-30T22:44:54.8042092Z
2020-07-30T22:44:54.8043068Z linkerd-existence
2020-07-30T22:44:54.8043368Z -----------------
2020-07-30T22:44:54.8043702Z β 'linkerd-config' config map exists
2020-07-30T22:44:54.8695470Z β heartbeat ServiceAccount exist
2020-07-30T22:44:55.0423587Z β control plane replica sets are ready
2020-07-30T22:44:55.1187159Z β no unschedulable pods
2020-07-30T22:44:55.1879203Z β controller pod is running
2020-07-30T22:44:55.5639435Z β can initialize the client
2020-07-30T22:44:55.8275030Z β can query the control plane API
2020-07-30T22:44:55.8827983Z
2020-07-30T22:44:55.8828933Z linkerd-config
2020-07-30T22:44:55.8829660Z --------------
2020-07-30T22:44:55.8830282Z β control plane Namespace exists
2020-07-30T22:44:55.9411313Z β control plane ClusterRoles exist
2020-07-30T22:44:55.9999288Z β control plane ClusterRoleBindings exist
2020-07-30T22:44:56.0553478Z β control plane ServiceAccounts exist
2020-07-30T22:44:56.1128850Z β control plane CustomResourceDefinitions exist
2020-07-30T22:44:56.1699541Z β control plane MutatingWebhookConfigurations exist
2020-07-30T22:44:56.2354589Z β control plane ValidatingWebhookConfigurations exist
2020-07-30T22:44:56.2929722Z β control plane PodSecurityPolicies exist
2020-07-30T22:44:56.4157382Z
2020-07-30T22:44:56.4157901Z linkerd-identity
2020-07-30T22:44:56.4158148Z ----------------
2020-07-30T22:44:56.4158419Z β certificate config is valid
2020-07-30T22:44:56.4158772Z β trust roots are using supported crypto algorithm
2020-07-30T22:44:56.4159070Z β trust roots are within their validity period
2020-07-30T22:44:56.4159380Z β trust roots are valid for at least 60 days
2020-07-30T22:44:56.4159693Z β issuer cert is using supported crypto algorithm
2020-07-30T22:44:56.4159982Z β issuer cert is within its validity period
2020-07-30T22:44:56.4160277Z β issuer cert is valid for at least 60 days
2020-07-30T22:44:56.4160552Z β issuer cert is issued by the trust root
2020-07-30T22:44:56.4899743Z
2020-07-30T22:44:56.4900975Z linkerd-api
2020-07-30T22:44:56.4901360Z -----------
2020-07-30T22:44:56.4901787Z β control plane pods are ready
2020-07-30T22:44:56.5715132Z β control plane self-check
2020-07-30T22:44:56.5715703Z β [kubernetes] control plane can talk to Kubernetes
2020-07-30T22:44:56.5716315Z β [prometheus] control plane can talk to Prometheus
2020-07-30T22:44:56.6271008Z β tap api service is running
2020-07-30T22:44:56.9519558Z
2020-07-30T22:44:56.9521381Z linkerd-version
2020-07-30T22:44:56.9522670Z ---------------
2020-07-30T22:44:56.9523225Z β can determine the latest version
2020-07-30T22:44:56.9523763Z βΌ cli is up-to-date
2020-07-30T22:44:56.9524464Z is running version 2.7.1 but the latest stable version is 2.8.1
2020-07-30T22:44:56.9525092Z see https://linkerd.io/checks/#l5d-version-cli for hints
2020-07-30T22:44:56.9525334Z
2020-07-30T22:44:56.9525689Z control-plane-version
2020-07-30T22:44:56.9526066Z ---------------------
2020-07-30T22:44:56.9526468Z βΌ control plane is up-to-date
2020-07-30T22:44:56.9526801Z is running version 2.7.1 but the latest stable version is 2.8.1
2020-07-30T22:44:56.9527400Z see https://linkerd.io/checks/#l5d-version-control for hints
2020-07-30T22:44:56.9527916Z β control plane and cli versions match
2020-07-30T22:44:57.0065064Z
2020-07-30T22:44:57.0065865Z linkerd-ha-checks
2020-07-30T22:44:57.0066328Z -----------------
2020-07-30T22:44:57.0066769Z β pod injection disabled on kube-system
2020-07-30T22:44:57.0067036Z
2020-07-30T22:44:57.0067416Z Status check results are β
2020-07-30T22:44:58.4277643Z namespace/linkerd configured
2020-07-30T22:44:58.4670143Z clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-identity unchanged
2020-07-30T22:44:58.5068907Z clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-identity unchanged
2020-07-30T22:44:58.5501330Z serviceaccount/linkerd-identity unchanged
2020-07-30T22:44:58.5892970Z clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-controller unchanged
2020-07-30T22:44:58.6298816Z clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-controller unchanged
2020-07-30T22:44:58.6706313Z serviceaccount/linkerd-controller unchanged
2020-07-30T22:44:58.7103787Z clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-destination unchanged
2020-07-30T22:44:58.7511682Z clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-destination unchanged
2020-07-30T22:44:58.7905831Z serviceaccount/linkerd-destination unchanged
2020-07-30T22:44:58.8305088Z role.rbac.authorization.k8s.io/linkerd-heartbeat unchanged
2020-07-30T22:44:58.8698751Z rolebinding.rbac.authorization.k8s.io/linkerd-heartbeat unchanged
2020-07-30T22:44:58.9109795Z serviceaccount/linkerd-heartbeat unchanged
2020-07-30T22:44:58.9509280Z role.rbac.authorization.k8s.io/linkerd-web unchanged
2020-07-30T22:44:58.9919627Z rolebinding.rbac.authorization.k8s.io/linkerd-web unchanged
2020-07-30T22:44:59.0851037Z clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-web-check configured
2020-07-30T22:44:59.1257235Z clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-web-check unchanged
2020-07-30T22:44:59.1677182Z clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-web-admin unchanged
2020-07-30T22:44:59.2089882Z serviceaccount/linkerd-web unchanged
2020-07-30T22:44:59.3024012Z customresourcedefinition.apiextensions.k8s.io/serviceprofiles.linkerd.io configured
2020-07-30T22:44:59.3914459Z customresourcedefinition.apiextensions.k8s.io/trafficsplits.split.smi-spec.io configured
2020-07-30T22:44:59.4313725Z clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-prometheus unchanged
2020-07-30T22:44:59.4715512Z clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-prometheus unchanged
2020-07-30T22:44:59.5120976Z serviceaccount/linkerd-prometheus unchanged
2020-07-30T22:44:59.5517692Z clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-proxy-injector unchanged
2020-07-30T22:44:59.6343585Z clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-proxy-injector configured
2020-07-30T22:44:59.6749793Z serviceaccount/linkerd-proxy-injector unchanged
2020-07-30T22:44:59.7620444Z secret/linkerd-proxy-injector-tls configured
2020-07-30T22:44:59.8483442Z mutatingwebhookconfiguration.admissionregistration.k8s.io/linkerd-proxy-injector-webhook-config configured
2020-07-30T22:44:59.8888729Z clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-sp-validator unchanged
2020-07-30T22:44:59.9729246Z clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-sp-validator configured
2020-07-30T22:45:00.0138172Z serviceaccount/linkerd-sp-validator unchanged
2020-07-30T22:45:00.0974370Z secret/linkerd-sp-validator-tls configured
2020-07-30T22:45:00.1981976Z validatingwebhookconfiguration.admissionregistration.k8s.io/linkerd-sp-validator-webhook-config configured
2020-07-30T22:45:00.2379394Z clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-tap unchanged
2020-07-30T22:45:00.2805335Z clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-tap-admin unchanged
2020-07-30T22:45:00.3251441Z clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-tap unchanged
2020-07-30T22:45:00.3691663Z clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-tap-auth-delegator unchanged
2020-07-30T22:45:00.4094910Z serviceaccount/linkerd-tap unchanged
2020-07-30T22:45:00.4505054Z rolebinding.rbac.authorization.k8s.io/linkerd-linkerd-tap-auth-reader unchanged
2020-07-30T22:45:00.5571038Z secret/linkerd-tap-tls configured
2020-07-30T22:45:00.6470206Z apiservice.apiregistration.k8s.io/v1alpha1.tap.linkerd.io configured
2020-07-30T22:45:00.7402886Z podsecuritypolicy.policy/linkerd-linkerd-control-plane configured
2020-07-30T22:45:00.7811473Z role.rbac.authorization.k8s.io/linkerd-psp unchanged
2020-07-30T22:45:00.8223822Z rolebinding.rbac.authorization.k8s.io/linkerd-psp unchanged
2020-07-30T22:45:00.9094791Z configmap/linkerd-config configured
2020-07-30T22:45:00.9978819Z secret/linkerd-identity-issuer configured
2020-07-30T22:45:01.1009161Z service/linkerd-identity configured
2020-07-30T22:45:01.2044794Z deployment.apps/linkerd-identity configured
2020-07-30T22:45:01.3485516Z service/linkerd-controller-api configured
2020-07-30T22:45:01.4551172Z deployment.apps/linkerd-controller configured
2020-07-30T22:45:01.5851557Z service/linkerd-dst configured
2020-07-30T22:45:01.7707391Z deployment.apps/linkerd-destination configured
2020-07-30T22:45:01.8807950Z cronjob.batch/linkerd-heartbeat configured
2020-07-30T22:45:01.9983540Z service/linkerd-web configured
2020-07-30T22:45:02.1126529Z deployment.apps/linkerd-web configured
2020-07-30T22:45:02.1969087Z configmap/linkerd-prometheus-config configured
2020-07-30T22:45:02.3410250Z service/linkerd-prometheus configured
2020-07-30T22:45:02.4391668Z deployment.apps/linkerd-prometheus configured
2020-07-30T22:45:02.5401494Z deployment.apps/linkerd-proxy-injector configured
2020-07-30T22:45:02.7280191Z service/linkerd-proxy-injector configured
2020-07-30T22:45:02.8173370Z service/linkerd-sp-validator configured
2020-07-30T22:45:02.9134524Z deployment.apps/linkerd-sp-validator configured
2020-07-30T22:45:03.0204105Z service/linkerd-tap configured
2020-07-30T22:45:03.1539240Z deployment.apps/linkerd-tap configured
2020-07-30T22:45:03.2772565Z configmap/linkerd-config-addons created
2020-07-30T22:45:03.3193358Z serviceaccount/linkerd-grafana unchanged
2020-07-30T22:45:03.4035525Z configmap/linkerd-grafana-config configured
2020-07-30T22:45:03.5033357Z service/linkerd-grafana configured
2020-07-30T22:45:03.6051518Z deployment.apps/linkerd-grafana configured
2020-07-30T22:47:03.6658653Z time="2020-07-30T22:47:03Z" level=error msg="could not set namespace from kubectl context: ensure a valid KUBECONFIG path has been set"
2020-07-30T22:47:03.6720015Z kubernetes-api
2020-07-30T22:47:03.6720709Z --------------
2020-07-30T22:47:03.6721254Z β can initialize the client
2020-07-30T22:47:03.8122238Z β can query the Kubernetes API
2020-07-30T22:47:03.8122955Z
2020-07-30T22:47:03.8123593Z kubernetes-version
2020-07-30T22:47:03.8124262Z ------------------
2020-07-30T22:47:03.8125025Z β is running the minimum Kubernetes API version
2020-07-30T22:47:03.8572540Z β is running the minimum kubectl version
2020-07-30T22:47:03.9001183Z
2020-07-30T22:47:03.9002496Z linkerd-existence
2020-07-30T22:47:03.9003018Z -----------------
2020-07-30T22:47:03.9003460Z β 'linkerd-config' config map exists
2020-07-30T22:47:03.9419593Z β heartbeat ServiceAccount exist
2020-07-30T22:47:04.0108619Z β control plane replica sets are ready
2020-07-30T22:47:04.0708164Z β no unschedulable pods
2020-07-30T22:47:04.1314701Z β controller pod is running
2020-07-30T22:47:04.4962521Z β can initialize the client
2020-07-30T22:47:04.7148308Z β can query the control plane API
2020-07-30T22:47:04.7546109Z
2020-07-30T22:47:04.7547380Z linkerd-config
2020-07-30T22:47:04.7547754Z --------------
2020-07-30T22:47:04.7548187Z β control plane Namespace exists
2020-07-30T22:47:04.8346839Z β control plane ClusterRoles exist
2020-07-30T22:47:04.8889790Z β control plane ClusterRoleBindings exist
2020-07-30T22:47:04.9357264Z β control plane ServiceAccounts exist
2020-07-30T22:47:04.9785891Z β control plane CustomResourceDefinitions exist
2020-07-30T22:47:05.0310852Z β control plane MutatingWebhookConfigurations exist
2020-07-30T22:47:05.0728671Z β control plane ValidatingWebhookConfigurations exist
2020-07-30T22:47:05.1137333Z β control plane PodSecurityPolicies exist
2020-07-30T22:47:05.2045212Z
2020-07-30T22:47:05.2045962Z linkerd-identity
2020-07-30T22:47:05.2046377Z ----------------
2020-07-30T22:47:05.2046785Z β certificate config is valid
2020-07-30T22:47:05.2047390Z β trust anchors are using supported crypto algorithm
2020-07-30T22:47:05.2047925Z β trust anchors are within their validity period
2020-07-30T22:47:05.2048431Z β trust anchors are valid for at least 60 days
2020-07-30T22:47:05.2049177Z β issuer cert is using supported crypto algorithm
2020-07-30T22:47:05.2049783Z β issuer cert is within its validity period
2020-07-30T22:47:05.2050272Z β issuer cert is valid for at least 60 days
2020-07-30T22:47:05.2050748Z β issuer cert is issued by the trust anchor
2020-07-30T22:47:05.2618333Z
2020-07-30T22:47:05.2620281Z linkerd-api
2020-07-30T22:47:05.2620585Z -----------
2020-07-30T22:47:05.2620913Z β control plane pods are ready
2020-07-30T22:47:05.5206124Z β control plane self-check
2020-07-30T22:47:05.5206704Z β [kubernetes] control plane can talk to Kubernetes
2020-07-30T22:47:05.5207229Z β [prometheus] control plane can talk to Prometheus
2020-07-30T22:47:05.5614606Z β tap api service is running
2020-07-30T22:47:05.7634754Z
2020-07-30T22:47:05.7637703Z linkerd-version
2020-07-30T22:47:05.7638124Z ---------------
2020-07-30T22:47:05.7638539Z β can determine the latest version
2020-07-30T22:47:05.7638933Z β cli is up-to-date
2020-07-30T22:47:05.7639112Z
2020-07-30T22:47:05.7639459Z control-plane-version
2020-07-30T22:47:05.7639832Z ---------------------
2020-07-30T22:47:05.7640232Z β control plane is up-to-date
2020-07-30T22:47:05.7640675Z β control plane and cli versions match
2020-07-30T22:47:05.8036992Z
2020-07-30T22:47:05.8040665Z linkerd-ha-checks
2020-07-30T22:47:05.8045221Z -----------------
2020-07-30T22:47:05.8045770Z β pod injection disabled on kube-system
2020-07-30T22:47:06.1162658Z β multiple replicas of control plane pods
2020-07-30T22:47:06.1564451Z
2020-07-30T22:47:06.1565203Z linkerd-addons
2020-07-30T22:47:06.1565590Z --------------
2020-07-30T22:47:06.1566031Z β 'linkerd-config-addons' config map exists
2020-07-30T22:47:06.2065046Z
2020-07-30T22:47:06.2066316Z linkerd-grafana
2020-07-30T22:47:06.2066818Z ---------------
2020-07-30T22:47:06.2067293Z β grafana add-on service account exists
2020-07-30T22:47:06.2450334Z β grafana add-on config map exists
2020-07-30T22:47:06.3040521Z β grafana pod is running
2020-07-30T22:47:06.3961972Z
2020-07-30T22:47:06.3963280Z Status check results are β
....
2020-07-30T22:47:11.4796135Z Release "testapp" has been upgraded. Happy Helming!
2020-07-30T22:47:11.4797148Z NAME: testapp
2020-07-30T22:47:11.4797662Z LAST DEPLOYED: Thu Jul 30 22:47:07 2020
2020-07-30T22:47:11.4798174Z NAMESPACE: default
2020-07-30T22:47:11.4798633Z STATUS: deployed
2020-07-30T22:47:11.4799058Z REVISION: 46
2020-07-30T22:47:11.4799484Z TEST SUITE: None
....
2020-07-30T22:47:11.4838938Z
2020-07-30T22:47:11.4898911Z ##[error]Bash wrote one or more lines to the standard error stream.
2020-07-30T22:47:11.4909905Z ##[error]time="2020-07-30T22:44:51Z" level=error msg="could not set namespace from kubectl context: ensure a valid KUBECONFIG path has been set"
2020-07-30T22:47:11.4911647Z ##[error]time="2020-07-30T22:44:53Z" level=error msg="could not set namespace from kubectl context: ensure a valid KUBECONFIG path has been set"
2020-07-30T22:47:11.4912834Z ##[error]time="2020-07-30T22:47:03Z" level=error msg="could not set namespace from kubectl context: ensure a valid KUBECONFIG path has been set"
2020-07-30T22:47:11.4936672Z ##[section]Finishing: Bash Script
Environment:
https://storage.googleapis.com/kubernetes-release/release/v1.18.0/bin/linux/amd64/kubectl
Hi @masters3d I'm going to move this to the Linkerd 2 issues page
@masters3d that error is printed when client-go returns an error to this code
The error
2020-07-30T22:47:11.4912834Z ##[error]time="2020-07-30T22:47:03Z" level=error msg="could not set namespace from kubectl context: ensure a valid KUBECONFIG path has been set"
indicates that the kubeconfig file is invalid, so when you run
$KUBECTL_BIN --kubeconfig="$KUBECONFIG_FILE_NAME" get po -A
does that command return with a successful response?
Yes it returns successful with all the pods.
Looking at the log again, it looks like Linkerd was successfully upgraded. What is the output from linkerd check --kubeconfig="$KUBECONFIG_FILE_NAME"?
As best as I can tell, it looks like linkerd is being installed/upgraded via a shell script that is doing some other things as well. Is it possible that the KUBECONFIG_FILE_NAME is being modified in the script?
That output is already part of the logs. After we generate the config file we donβt change it in the script. Everything working as expected with with 2.7.1.
hi @masters3d I dug into the code some more and the logic that is writing that message was added and modified between 2.7.1 and 2.8.1.
The underlying error is coming from client go.
The error message itself is benign and there are a couple of things that we can do to improve the messaging:
warnclient-godefault namespace is going to be usedAt the end of the day, it looks like Linkerd is successfully installed/upgraded. I'm very curious to understand how this particular KUBECONFIG_FILE_NAME is causing this behavior. If you're comfortable sharing it after removing all the credentials, please send it to me at [email protected]
I obfuscated the data.
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: REDACTED
server: https://SomeCluste03-961bf307.hcp.westus2.azmk8s.io:443
name: SomeContext03
contexts:
- context:
cluster: SomeContext03
user: clusterUser_SomeCluste03_SomeContext03
name: SomeContext03
current-context: SomeContext03
kind: Config
preferences: {}
users:
- name: clusterUser_SomeCluste03_SomeContext03
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
token: REDACTED
@masters3d if I'm reading this properly it looks like there isn't a namespace set in the - context key of the kubeconfig file, and I think that is causing the behavior.
I think we can address this with better logging as I described in my last message
i think this one is fixed by https://github.com/linkerd/linkerd2/pull/4824 already. we should close this ticket
I got the issue please see below
linkerd2-cli-stable-2.8.1
ERRO[1272] received error while attempting to prepare install yaml: time="2020-09-17T16:31:49+08:00" level=error msg="could not set namespace from kubectl context: ensure a valid KUBECONFIG path has been set"