Pipeline: Waiting for connection, timeout with task and taskrun

[bpowell]

Expected Behavior

kubectl apply -f taskrun.yml to be successful

Actual Behavior

  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "error when creating \"taskrun.yml\": Internal error occurred: failed calling admission webhook \"webhook.tekton.dev\": Post https://tekton-pipelines-webhook.tekton-pipelines.svc:443/?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)",
  "reason": "InternalError",
  "details": {
    "causes": [
      {
        "message": "failed calling admission webhook \"webhook.tekton.dev\": Post https://tekton-pipelines-webhook.tekton-pipelines.svc:443/?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
      }
    ]
  },
  "code": 500
}

Steps to Reproduce the Problem

1. Install tekton in EKS kubectl apply --filename https://storage.googleapis.com/tekton-releases/previous/0.2.0/release.yaml
2. create taskrun.yml with

apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: echo-hello-world
spec:
  steps:
    - name: echo
      image: ubuntu
      command:
        - echo
      args:
        - "hello world"
---
apiVersion: tekton.dev/v1alpha1
kind: TaskRun
metadata:
  name: echo-hello-world-task-run
spec:
  taskRef:
    name: echo-hello-world
  trigger:
    type: manual

pulled from the tutorial

1. run kubetcl apply -n tekton-test taskrun.yml

Additional Info

Using EKS K8s version 1.10 and kubectl version 1.10

[ffahri]

Could you check your ec2 security groups it may be causing for that

[bpowell]

@ffahri what should I be looking for in my ec2 SG? I can see in my cluster that tekton-pipelines-webhook.tekton-pipelines.svc routes to the correct pod with nslookup.

[bpowell]

Thanks @ffahri I was able to figure it out. The worker nodes need to be able to talk to the control plane over port 443.

[cmadam]

@bpowell: could you please elaborate more on how you solved this problem? I am having exactly the same issue after setting up a Kubernetes cluster with one master and one worker, using kubeadm. I am using version 0.5.0 of tekton pipelines, and Kubernetes version 1.15. I have followed the instructions here https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/ to ensure that coredns works properly, yet I still get this timeout issue.

[bobcatfish]

@cmadam are you seeing exactly the same issue, or could it be a slightly different one? Just a heads up that there was a bug in v0.5.0 (#1068) that would cause PipelineRuns to hang (and presumably eventually timeout) which is fixed in https://github.com/tektoncd/pipeline/releases/tag/v0.5.1

(If it's the same issue described above where the webhook is having a problem, then it's something else!)

[cmadam]

@bobcatfish thank you so much! Upgrading to release 0.5.1 has solved my problem:

$ kubectl apply -f hello_task.yaml
Error from server (InternalError): error when creating "hello_task.yaml": 
Internal error occurred: failed calling webhook "webhook.tekton.dev": 
Post https://tekton-pipelines-webhook.tekton-pipelines.svc:443/?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

$ kubectl delete -f https://storage.googleapis.com/tekton-releases/previous/v0.5.0/release.yaml
namespace "tekton-pipelines" deleted
podsecuritypolicy.policy "tekton-pipelines" deleted
. . .
deployment.apps "tekton-pipelines-controller" deleted
deployment.apps "tekton-pipelines-webhook" deleted

$ kubectl create -f https://storage.googleapis.com/tekton-releases/previous/v0.5.1/release.yaml
namespace/tekton-pipelines created
podsecuritypolicy.policy/tekton-pipelines created
.  .  .
deployment.apps/tekton-pipelines-controller created
deployment.apps/tekton-pipelines-webhook created

$ kubectl apply -f hello_task.yaml
task.tekton.dev/echo-hello-world created

For the sake of completeness, this is the contents of hello_task.yaml (taken straight from tutorial):

apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: echo-hello-world
spec:
  steps:
    - name: echo
      image: ubuntu
      command:
        - echo
      args:
        - "hello world"

[bobcatfish]

Phew, glad it's working for you now @cmadam ! I don't quite know why you would see that exact error 😅 but at least you're not seeing it anymore! :D

[cmadam]

@bobcatfish : While I can create the task, I am still having a problem when trying to run this task:

$ kubectl apply -f hello_task_run.yaml
Error from server (InternalError): error when creating "hello_task_run.yaml": 
Internal error occurred: failed calling webhook "webhook.tekton.dev": 
Post https://tekton-pipelines-webhook.tekton-pipelines.svc:443/?timeout=30s: 
net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

This is the content of the task run object:

apiVersion: tekton.dev/v1alpha1
kind: TaskRun
metadata:
  name: echo-hello-world-task-run
spec:
  taskRef:
    name: echo-hello-world

```

[cmadam]

Actually, I have tried removing the hello-world task and re-creating it, and that failed, as well. First, I have got a slightly different error message:

$ kubectl apply -f hello_task.yaml
task.tekton.dev/echo-hello-world unchanged

$ kubectl delete -f hello_task.yaml
task.tekton.dev "echo-hello-world" deleted

$ kubectl apply -f hello_task.yaml
Error from server (InternalError): error when creating "hello_task.yaml": 
Internal error occurred: failed calling webhook "webhook.tekton.dev": 
Post https://tekton-pipelines-webhook.tekton-pipelines.svc:443/?timeout=30s: context deadline exceeded

And then, on subsequent invocation, I started receiving the old error message again:

$ kubectl apply -f hello_task.yaml
Error from server (InternalError): error when creating "hello_task.yaml": 
Internal error occurred: failed calling webhook "webhook.tekton.dev": 
Post https://tekton-pipelines-webhook.tekton-pipelines.svc:443/?timeout=30s: 
net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

[EricXiao]

Hi there,

I have encountered a similar issue. The tekton containers are running without any issue found in the logs. I tried v0.5.1 and the latest v0.6.0 and got the same rsult. This is only the first step to try Tekton from offical tutorial.

kubectl apply -f task.yaml
Error from server (InternalError): error when creating "task.yaml": Internal error occurred: failed calling webhook "webhook.tekton.dev": Post https://tekton-pipelines-webhook.tekton-pipelines.svc:443/?timeout=30s: Forbidden

[cmadam]

@EricXiao I am not sure how to solve this issue in a Kubernetes cluster. The way I got around it was by installing an OpenShift cluster (instead of Kubernetes) in my testbed. Tekton ran without any problems in OpenShift. I have used primarily this to install OpenShift: https://github.com/gshipley/installcentos

[cnych]

I have encountered a similar issue. I tried v0.5.2 and v0.6.0 got the same result.

$ kubectl apply -f task-hello.yaml -v=9
......
I0822 14:55:28.664968   59526 helpers.go:196] server response object: [{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "error when creating \"task-hello.yaml\": Internal error occurred: failed calling webhook \"webhook.tekton.dev\": Post https://tekton-pipelines-webhook.tekton-pipelines.svc:443/?timeout=30s: context deadline exceeded",
  "reason": "InternalError",
  "details": {
    "causes": [
      {
        "message": "failed calling webhook \"webhook.tekton.dev\": Post https://tekton-pipelines-webhook.tekton-pipelines.svc:443/?timeout=30s: context deadline exceeded"
      }
    ]
  },
  "code": 500
}]
F0822 14:55:28.665098   59526 helpers.go:114] Error from server (InternalError): error when creating "task-hello.yaml": Internal error occurred: failed calling webhook "webhook.tekton.dev": Post https://tekton-pipelines-webhook.tekton-pipelines.svc:443/?timeout=30s: context deadline exceeded
$ kubectl get mutatingwebhookconfigurations
NAME                     CREATED AT
webhook.tekton.dev       2019-08-22T11:06:58Z
$ kubectl get pods -n tekton-pipelines
NAME                                           READY   STATUS    RESTARTS   AGE
tekton-pipelines-controller-69fc665ffd-vk96g   1/1     Running   0          179m
tekton-pipelines-webhook-7999cb9544-2rzsx      1/1     Running   0          179m

maybe the admisson controller settings problem?

[strideynet]

@cnych did you resolve this issue?