Jx: webhook on Github always fails We couldn’t deliver this payload: Failure when receiving data from the peer

[dove-young]

Summary

webhook on Github always fails. Error msg said We couldn’t deliver this payload: Failure when receiving data from the peer

Steps to reproduce the behavior

Expected behavior

webhook should works well

Actual behavior

webhook on Github always fails. Error msg said We couldn’t deliver this payload: Failure when receiving data from the peer

Jx version

The output of jx version is:

NAME               VERSION
jx                 2.0.758
jenkins x platform 2.0.1247
Kubernetes cluster v1.13.5+icp-ee
kubectl            v1.13.5+icp-ee
helm client        Client: v2.12.3+geecf22f
git                2.23.0
Operating System   Red Hat Enterprise Linux Server release 7.6 (Maipo)


verifying packages

Jenkins type

• [x] Serverless Jenkins X Pipelines (Tekton + Prow)
• [ ] Classic Jenkins

Kubernetes cluster

IBM Cloud Private 3.2.0

Operating system / Environment

RHEL 7.6

kubectl get svc,ep
NAME                                  TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/deck                          ClusterIP   10.0.201.214   <none>        80/TCP     44m
service/heapster                      ClusterIP   10.0.5.104     <none>        8082/TCP   40m
service/hook                          ClusterIP   10.0.223.151   <none>        80/TCP     44m
service/jenkins-x-chartmuseum         ClusterIP   10.0.220.196   <none>        8080/TCP   40m
service/nexus                         ClusterIP   10.0.128.133   <none>        80/TCP     40m
service/pipelinerunner                ClusterIP   10.0.145.181   <none>        80/TCP     44m
service/tekton-pipelines-controller   ClusterIP   10.0.212.128   <none>        9090/TCP   44m
service/tekton-pipelines-webhook      ClusterIP   10.0.134.175   <none>        443/TCP    44m
service/tide                          ClusterIP   10.0.1.131     <none>        80/TCP     44m

NAME                                    ENDPOINTS                            AGE
endpoints/deck                          10.1.127.199:8080,10.1.212.72:8080   44m
endpoints/heapster                      10.1.94.195:8082                     40m
endpoints/hook                          10.1.212.73:8888,10.1.229.8:8888     44m
endpoints/jenkins-x-chartmuseum         10.1.77.139:8080                     40m
endpoints/nexus                         10.1.48.139:8081                     40m
endpoints/pipelinerunner                10.1.94.196:8080                     44m
endpoints/tekton-pipelines-controller   10.1.77.137:9090                     44m
endpoints/tekton-pipelines-webhook      10.1.229.9:8443                      44m
endpoints/tide                          10.1.127.202:8888                    44m

kubectl get ing
NAME          HOSTS                               ADDRESS       PORTS   AGE
chartmuseum   chartmuseum.jx.<my-ip-addr>.nip.io   <my-ip-addr>   80      40m
deck          deck.jx.<my-ip-addr>.nip.io          <my-ip-addr>   80      40m
hook          hook.jx.<my-ip-addr>.nip.io          <my-ip-addr>   80      53s
nexus         nexus.jx.<my-ip-addr>.nip.io         <my-ip-addr>   80      40m
tide          tide.jx.<my-ip-addr>.nip.io          <my-ip-addr>   80      109s

And also there is no place to debugging.

print logs from pod hook-5b79bdd4cb-658t8 and tekton-pipelines-webhook-76c6bd5b6-29cpl do not help debugging.

[arghyadeep-k]

I'm facing the same issue too.

[vamsilella]

i'm facing the same issue with freestyle project.

[vamsilella]

i have changed the IP of jenkins to public ip in webhook url and it worked.

[arghyadeep-k]

i have changed the IP of jenkins to public ip in webhook url and it worked.

That works.
Use the public IP of your server in the url instead of IP provided under Git in Configure Settings section of Jenkins.

[pmuir]

Looks like we need a FAQ for this one.

[dsanders23]

to public ip in webhook url and it worked.

Looks a nice solution, thanks! Could you share a little bit more about how can we do the same? Thanks!

[arghyadeep-k]

to public ip in webhook url and it worked.

Looks a nice solution, thanks! Could you share a little bit more about how can we do the same? Thanks!

Use the public IP of your server in the url instead of IP provided under Git in Configure Settings section of Jenkins.

[srafiqa]

Hi ,
I would really appreciate if anyone could share me the syntax of the payload with public IP address. I am also getting the error message "We couldn’t deliver this payload: Failure when receiving data from the peer" , even after provided public ip of jenkins server.

[arghyadeep-k]

Hi ,
I would really appreciate if anyone could share me the syntax of the payload with public IP address. I am also getting the error message "We couldn’t deliver this payload: Failure when receiving data from the peer" , even after provided public ip of jenkins server.

Use the address in the following format:
http://104.211.218.105:8080/github-webhook/

Replace the ip with the public ip of your jenkins server and the port on which your jenkins is running which is by default 8080.

[srafiqa]

Thank you for your response , i updated in the same way , however still getting the same issue.
Not sure it is because the github enabled with SSO .

[binbrayer]

Try to use application/x-www-form-urlencoded not application/json content type.

[preconceptstech84]

I'm seeing the same issue for our GHE. We too have SSO enabled but we are using a token so that shouldn't be an issue. Changing to the ip of the loadbalancer in front of our jenkins doesn't help

[adeoke]

I'm facing the same issue by the looks of it. Has anyone managed to find the solution?

[narayanan-n]

any updates on this?

[mmreddy1582]

Having same issue ,Please kindly post/share how to fix this issue .

[himanshush13]

Any luck about this issue. Even though i have used the public ip in the webhook url, but no changes

[himanshush13]

@vamsilella , if you can check

[vamsilella]

@himanshush13 can you provide more details like webhook url / payload
is SSO enabled? is your jenkins hosted on any container? or is it on ec2?
This would help debugging the issue.

[mmreddy1582]

Please follow the below steps to fix the "Jenkins+Github: We couldn’t deliver this payload: Couldn't connect to server"
Github won't be able to communicate with Jenkins which is running local /private IP address .
You need to specify an IP address GitHub can contact over the internet not the actual IP address .
Steps:
1.Download the https://ngrok.com/download server where it is running with Jenkins .
2../ngrok http 8080 ,it will give one http URL ..wait for some time ..some time taking 2 to 3 mins .
3.http://<......>ngrok.io eg http://3b2db437.ngrok.io
4.http://<...>/github-webhook/ eg http://3b2db437.ngrok.io/github-webhook/ ..
====>Don't give port 8080 as we already passed /ngrok http 8080
5.Final test the github webhook status .

[mmreddy1582]

step 4 to update the github webhook URL .

[himanshush13]

Jenkins is not on any container, it is working on a linux vm independently.
I tried giving the github credentials to jenkins configuration.

this is the url, I am using
http://frparvm-blkchn:8082/github-webhook/
I also used this url
http://193.57.249.2:8082/github-webhook/

I followed this blog--
https://oncomputingwell.princeton.edu/2018/01/triggering-a-jenkins-build-every-time-changes-are-pushed-to-a-git-branch-on-github/

is it an issue because, we are in our company network ?

[vamsilella]

you can use the above ip http://193.57.249.2:8082/ to access jenkins dashboard within your company using VPN. But github needs an IP ( public ) which can be accessible through internet
so your payload should be

http://publicip:8082/github-webhook/

and using http://frparvm-blkchn:8082 as it's not a public domain its just the hostname so github doesn't know what it is... as github is not hosted in your company network ;) just kidding.

[shyamshinde008]

@vamsilella I am also facing same issue and i am also in company network. But i am not sure how can i get public IP for a machine in private network. I tried ngrok but that is not working. i found smee and smee-client topic but i am not sure how to configure smee-client on my windows machine. will you please help me in triggering jenkins using webhook.
Thanks in advance!!!

[vamsilella]

@shyamshinde008 its the IT job to assign a static IP to the server where your Jenkins master is running and hostname should be set in that server (get a sub domain from IT) and they have to whitelist either bitbucket/github for ingress. thats it.

[JohnYu500]

I have the same issue. In my case, my Jenkins is running on an EC2 in AWS and it does have a public IP address. I have opened up port 8080 for HTTP traffic from github.com. When I pasted the payload URL directly to a browser I got this java error: java.lang.Exception: Method POST required. This at least indicates the URL hit the Jenkins server.

So I really have no idea why the payload could not be delivered.

Any help is appreciated.

[Bheemalingareddyh]

Install GitHub Integration plugin and check.

[Pratishvg]

Please follow the below steps to fix the "Jenkins+Github: We couldn’t deliver this payload: Couldn't connect to server"
Github won't be able to communicate with Jenkins which is running local /private IP address .
You need to specify an IP address GitHub can contact over the internet not the actual IP address .
Steps:
1.Download the https://ngrok.com/download server where it is running with Jenkins .
2../ngrok http 8080 ,it will give one http URL ..wait for some time ..some time taking 2 to 3 mins .
3.http://<......>ngrok.io eg http://3b2db437.ngrok.io
4.http://<...>/github-webhook/ eg http://3b2db437.ngrok.io/github-webhook/ ..
====>Don't give port 8080 as we already passed /ngrok http 8080
5.Final test the github webhook status .

This actually worked for me. I am working locally on my machine. Don't know how big companies work this issue out. Thanks a lot.
Just that the I downloaded the ngrok and unzipped the file. Opened the command window from the folder the ngrok.exe was stored and executed command 'ngrok http 8080' as stated earlier and had to wait. I got 2 addresses(names) - one with 'http' and the other with 'https'. I used the 'https://......ngrok.io/github-webhook/' address for webhook in github and it worked.

[JohnYu500]

Actually I was able to make it work. The issue I had before was because I was using the wrong IP address. Later I found out that github webhook has its own IP addresses. I was using the IP of the github.com. Once I added the three IP addresses of Github webook to my AWS security group, the payload works like a charm. Thanks.

[arghyadeep-k]

I don't think this is an issue anymore.
Can we close this?

[shadoath]

Actually I was able to make it work. The issue I had before was because I was using the wrong IP address. Later I found out that github webhook has its own IP addresses. I was using the IP of the github.com. Once I added the three IP addresses of Github webook to my AWS security group, the payload works like a charm. Thanks.

Those can be found https://api.github.com/meta

"hooks": [
"192.30.252.0/22",
"185.199.108.0/22",
"140.82.112.0/20"
]

[JohnYu500]

Yes. That is right.

[sittuanand007]

Ngrok will be temporary solution right because after the session ends, we will further have to get it configured from the beginning. I am working on my company's VM which is in private network. My doubt is that I have configured Jenkins on localhost in the vm that is in private network. As we know that localhost generally runs on http. So if I want my Jenkins to have https, so what will be the approach for that ??
When I will get a static IP assigned to the VM and when the IT dept will give me a sub-domain, then will it be http or https ?

[shyamshinde008]

@sittuanand007 Ngrok is just for POC not permanent solution. you have to make your Jenkins server public if you are in private network (company network) so that git is able to connect and deliver webhook payload to your system. Check with your network team they might help you in this.

[darenjacobstellic]

i have changed the IP of jenkins to public ip in webhook url and it worked.

That works.
Use the public IP of your server in the url instead of IP provided under Git in Configure Settings section of Jenkins.

Could you please state this more clearly? Is this operation happening in Jenkins, is it happening in Github? Please specify the exact steps

[dineshdevopsshare]

Try to use application/x-www-form-urlencoded not application/json content type.

Thanks mate! this saved my lot of time!!!

[prashant-bande]

I have updated the IP with the current public IP of the server. It works like a charm.

Cloud server often gets the new public IP and it happens that webhook has an old IP and updating it to current public IP will solve the issue,

[jenkins-x-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle stale

[mattmattox]

I ran into this issue when using CloudFlare in Proxy mode.