Pipeline: Install Task/TaskRun Object Internal error occurred: failed calling webhook "webhook.tekton.dev"

Created on 22 Aug 2019  ·  4Comments  ·  Source: tektoncd/pipeline

Install Task/TaskRun Kind Object Internal error occurred: failed calling webhook "webhook.tekton.dev"

Expected Behavior

I expect the Task/TaskRun execution to run smoothly from the tutorial here - https://github.com/tektoncd/pipeline/blob/master/docs/tutorial.md#task.

Steps to Reproduce the Problem

  1. kubernetes version
☁  $ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T18:55:03Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
  1. install release.yaml successfully
☁  $ kubectl get crd |grep tekton
clustertasks.tekton.dev                           2019-08-22T09:30:31Z
pipelineresources.tekton.dev                      2019-08-22T09:30:33Z
pipelineruns.tekton.dev                           2019-08-22T09:30:32Z
pipelines.tekton.dev                              2019-08-22T09:30:32Z
taskruns.tekton.dev                               2019-08-22T09:30:33Z
tasks.tekton.dev                                  2019-08-22T09:30:33Z
☁  $ kubectl api-versions |grep tekton
tekton.dev/v1alpha1
☁  $ kubectl get pods -n tekton-pipelines
NAME                                           READY   STATUS    RESTARTS   AGE
tekton-pipelines-controller-76847b9469-jmtx8   1/1     Running   0          40m
tekton-pipelines-webhook-568859d4f8-vwvpw      1/1     Running   0          40m
☁  $  kubectl get svc -n tekton-pipelines
NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
tekton-pipelines-controller   ClusterIP   10.109.125.52    <none>        9090/TCP   40m
tekton-pipelines-webhook      ClusterIP   10.100.105.126   <none>        443/TCP    40m
  1. define Task Object:（task-hello.yaml）
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: echo-hello-world-task
spec:
  steps:
    - name: echo
      image: busybox
      command:
        - echo
      args:
        - "hello world"
  1. create the task object：
kubectl apply -f task-hello.yaml -v=9
I0822 10:26:57.492832   36942 loader.go:359] Config loaded from file /Users/ych/.kube/config
I0822 10:26:57.498991   36942 round_trippers.go:419] curl -k -v -XGET  -H "User-Agent: kubectl/v1.14.2 (darwin/amd64) kubernetes/66049e3" -H "Accept: application/[email protected]+protobuf" 'https://ydzs-master:6443/openapi/v2?timeout=32s'
I0822 10:26:57.521825   36942 round_trippers.go:438] GET https://ydzs-master:6443/openapi/v2?timeout=32s 200 OK in 22 milliseconds
I0822 10:26:57.522303   36942 round_trippers.go:444] Response Headers:
I0822 10:26:57.522333   36942 round_trippers.go:447]     Content-Type: application/octet-stream
......
I0822 10:26:57.803188   36942 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/v1.14.2 (darwin/amd64) kubernetes/66049e3" 'https://ydzs-master:6443/apis/tekton.dev/v1alpha1/namespaces/kube-ops/tasks/echo-hello-world-task'
I0822 10:26:58.206942   36942 round_trippers.go:438] GET https://ydzs-master:6443/apis/tekton.dev/v1alpha1/namespaces/kube-ops/tasks/echo-hello-world-task 404 Not Found in 403 milliseconds
I0822 10:26:58.206983   36942 round_trippers.go:444] Response Headers:
I0822 10:26:58.207010   36942 round_trippers.go:447]     Content-Type: application/json
I0822 10:26:58.207046   36942 round_trippers.go:447]     Content-Length: 244
I0822 10:26:58.207057   36942 round_trippers.go:447]     Date: Thu, 22 Aug 2019 10:26:57 GMT
I0822 10:26:58.207097   36942 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"tasks.tekton.dev \"echo-hello-world-task\" not found","reason":"NotFound","details":{"name":"echo-hello-world-task","group":"tekton.dev","kind":"tasks"},"code":404}
I0822 10:26:58.208485   36942 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/v1.14.2 (darwin/amd64) kubernetes/66049e3" 'https://ydzs-master:6443/api/v1/namespaces/kube-ops'
I0822 10:26:58.273437   36942 round_trippers.go:438] GET https://ydzs-master:6443/api/v1/namespaces/kube-ops 200 OK in 64 milliseconds
I0822 10:26:58.273530   36942 round_trippers.go:444] Response Headers:
I0822 10:26:58.273545   36942 round_trippers.go:447]     Content-Type: application/json
I0822 10:26:58.273555   36942 round_trippers.go:447]     Content-Length: 346
I0822 10:26:58.273564   36942 round_trippers.go:447]     Date: Thu, 22 Aug 2019 10:26:57 GMT
I0822 10:26:58.273637   36942 request.go:942] Response Body: {"kind":"Namespace","apiVersion":"v1","metadata":{"name":"kube-ops","selfLink":"/api/v1/namespaces/kube-ops","uid":"c52f7c5e-46ff-11e9-8582-525400cc3c00","resourceVersion":"51926955","creationTimestamp":"2019-03-15T08:53:15Z","finalizers":["finalizers.kubesphere.io/namespaces"]},"spec":{"finalizers":["kubernetes"]},"status":{"phase":"Active"}}
I0822 10:26:58.276106   36942 request.go:942] Request Body: {"apiVersion":"tekton.dev/v1alpha1","kind":"Task","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"tekton.dev/v1alpha1\",\"kind\":\"Task\",\"metadata\":{\"annotations\":{},\"name\":\"echo-hello-world-task\",\"namespace\":\"kube-ops\"},\"spec\":{\"steps\":[{\"args\":[\"hello world\"],\"command\":[\"echo\"],\"image\":\"busybox\",\"name\":\"echo\"}]}}\n"},"name":"echo-hello-world-task","namespace":"kube-ops"},"spec":{"steps":[{"args":["hello world"],"command":["echo"],"image":"busybox","name":"echo"}]}}
I0822 10:26:58.276515   36942 round_trippers.go:419] curl -k -v -XPOST  -H "User-Agent: kubectl/v1.14.2 (darwin/amd64) kubernetes/66049e3" -H "Accept: application/json" -H "Content-Type: application/json" 'https://ydzs-master:6443/apis/tekton.dev/v1alpha1/namespaces/kube-ops/tasks'
I0822 10:27:28.405904   36942 round_trippers.go:438] POST https://ydzs-master:6443/apis/tekton.dev/v1alpha1/namespaces/kube-ops/tasks 500 Internal Server Error in 30128 milliseconds
I0822 10:27:28.405944   36942 round_trippers.go:444] Response Headers:
I0822 10:27:28.405957   36942 round_trippers.go:447]     Content-Type: application/json
I0822 10:27:28.405968   36942 round_trippers.go:447]     Content-Length: 637
I0822 10:27:28.405977   36942 round_trippers.go:447]     Date: Thu, 22 Aug 2019 10:27:27 GMT
I0822 10:27:28.406021   36942 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"webhook.tekton.dev\": Post https://tekton-pipelines-webhook.tekton-pipelines.svc:443/?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"webhook.tekton.dev\": Post https://tekton-pipelines-webhook.tekton-pipelines.svc:443/?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"}]},"code":500}
I0822 10:27:28.407999   36942 helpers.go:196] server response object: [{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "error when creating \"task-hello.yaml\": Internal error occurred: failed calling webhook \"webhook.tekton.dev\": Post https://tekton-pipelines-webhook.tekton-pipelines.svc:443/?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)",
  "reason": "InternalError",
  "details": {
    "causes": [
      {
        "message": "failed calling webhook \"webhook.tekton.dev\": Post https://tekton-pipelines-webhook.tekton-pipelines.svc:443/?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
      }
    ]
  },
  "code": 500
}]
F0822 10:27:28.408906   36942 helpers.go:114] Error from server (InternalError): error when creating "task-hello.yaml": Internal error occurred: failed calling webhook "webhook.tekton.dev": Post https://tekton-pipelines-webhook.tekton-pipelines.svc:443/?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  1. webhook pod logs:
☁  $ kubectl logs -f tekton-pipelines-webhook-568859d4f8-vwvpw -n tekton-pipelines
{"level":"info","caller":"logging/config.go:100","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n  \"level\": \"info\",\n  \"development\": false,\n  \"sampling\": {\n    \"initial\": 100,\n    \"thereafter\": 100\n  },\n  \"outputPaths\": [\"stdout\"],\n  \"errorOutputPaths\": [\"stderr\"],\n  \"encoding\": \"json\",\n  \"encoderConfig\": {\n    \"timeKey\": \"\",\n    \"levelKey\": \"level\",\n    \"nameKey\": \"logger\",\n    \"callerKey\": \"caller\",\n    \"messageKey\": \"msg\",\n    \"stacktraceKey\": \"stacktrace\",\n    \"lineEnding\": \"\",\n    \"levelEncoder\": \"\",\n    \"timeEncoder\": \"\",\n    \"durationEncoder\": \"\",\n    \"callerEncoder\": \"\"\n  }\n}\n"}
{"level":"info","caller":"logging/config.go:101","msg":"Logging level set to info"}
{"level":"warn","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: open /var/run/ko/HEAD: no such file or directory"}
{"level":"info","logger":"webhook","caller":"webhook/main.go:56","msg":"Starting the Configuration Webhook","knative.dev/controller":"webhook"}
{"level":"info","logger":"webhook.config-store","caller":"configmap/store.go:166","msg":"defaults config \"config-defaults\" config was added or updated: &{60}","knative.dev/controller":"webhook"}
{"level":"info","logger":"webhook","caller":"webhook/webhook.go:180","msg":"Did not find existing secret, creating one","knative.dev/controller":"webhook"}
{"level":"info","logger":"webhook","caller":"webhook/webhook.go:284","msg":"Found certificates for webhook...","knative.dev/controller":"webhook"}
{"level":"info","logger":"webhook","caller":"webhook/webhook.go:403","msg":"Created a webhook","knative.dev/controller":"webhook"}
{"level":"info","logger":"webhook","caller":"webhook/webhook.go:296","msg":"Successfully registered webhook","knative.dev/controller":"webhook"}
{"level":"info","logger":"webhook.config-store","caller":"configmap/store.go:166","msg":"defaults config \"config-defaults\" config was added or updated: &{60}","knative.dev/controller":"webhook"}
{"level":"info","logger":"webhook.config-store","caller":"configmap/store.go:166","msg":"defaults config \"config-defaults\" config was added or updated: &{60}","knative.dev/controller":"webhook"}
kinbug
Source
picture cnych
👍3

Most helpful comment

@cnych how to fixed it?I have same issues now,looking forward to your reply,thanks!

picture zzguang520 on 12 May 2020
👍2

All 4 comments

@cnych what is the status of this one ?
Looking at the report it seems you are experiencing network problem on your cluster, making k8s not able to forward the creation payload to the webhook — there is little we can do here.

vdemeester picture vdemeester on 7 Nov 2019

@vdemeester Sorry, the last troubleshooting is the problem of my admission's webhook. Now it's OK to remove it.

cnych picture cnych on 8 Nov 2019

@cnych how to fixed it?I have same issues now,looking forward to your reply,thanks!

zzguang520 picture zzguang520 on 12 May 2020
👍2

@zzguang520 I was on GKE and I had to create a new firewall rule, details here.

tjwallace picture tjwallace on 20 May 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

pritidesai picture pritidesai  ·  4Comments
sbwsg picture sbwsg  ·  4Comments
objectiveous picture objectiveous  ·  3Comments
hrishin picture hrishin  ·  3Comments
csantanapr picture csantanapr  ·  3Comments