Jx: Create Spring Error: running skaffold steps

[wofish2]

When I run the command "jx create spring", pipeline stopped at pushing image to docker-registry:

...
Successfully built a904e0f4574a
Successfully tagged 4b1be5fc883375942973dbacf2e0d4b9:latest
Digest: 4b1be5fc883375942973dbacf2e0d4b9:latest
Successfully tagged 172.21.133.137:5000/jenkinsx/demo84:0.0.1
The push refers to a repository [172.21.133.137:5000/jenkinsx/demo84]
Error: running skaffold steps: build: build step: running push: Get https://172.21.133.137:5000/v2/: http: server gave HTTP response to HTTPS client
Usage:
  skaffold run [flags]

Flags:
  -f, --filename string       Filename or URL to the pipeline file (default "skaffold.yaml")
  -h, --help                  help for run
  -p, --profile stringArray   Activate profiles by name
  -t, --tag string            The optional custom tag to use for images which overrides the current Tagger configuration
      --toot                  Emit a terminal beep after the deploy is complete

Global Flags:
  -v, --verbosity string   Log level (debug, info, warn, error, fatal, panic (default "warning")

[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Promote to Environments)
Stage 'Promote to Environments' skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] cleanWs
[WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done
[Pipeline] input
Pipeline failed. 
We will keep the build pod around to help you diagnose any failures. 

I entered the jenkins-x-docker-registry pod, it shows on shell:
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:262: starting container process caused "exec: \"bash\": executable file not found in $PATH"

sh-3.2# jx version
NAME           VERSION
jx             1.1.55
Jenkins X      0.0.678
Kubernetes     v1.8.8-2+9d6e0610086578
Helm Client    v2.8.1+g6af75a8
Helm Server    v2.8.1+g6af75a8
Kubectl Client v1.7.4
Git            git version 2.6.4

[jstrachan]

I wonder if this is related to the insecure registry setting of your docker daemons in your cluster. What kind of cluster are you using? When we create clusters with, say, AWS we explicitly setup an insecure registry mask so we can communicate with the docker registry without having to try use HTTPS and setup certs for the internal docker registry

[CobraFlow]

@jstrachan I have now reached this same point with the same error.

I am assuming that all 'agent' nodes require insecure registry settings.
/etc/docker/daemon.json -

{ "insecure-registries": [ "docker-registry:5000" ] }

One thought on this though, the skaffold.yaml created by jx create spring... uses {{.JENKINS_X_DOCKER_REGISTRY_SERVICE_HOST}} which is an IP address. We won't know this value, and it can change! Should this be changed to the fixed registry dns name of 'docker-registry'?

[CobraFlow]

...sorry, I meant { "insecure-registries": [ "jenkins-x-docker-registry:5000" ] }

[jstrachan]

@CobraFlow we cannot use kubernetes service names to refer to docker registries unfortunately as its used by the docker daemon & kubelet which are not pods inside kubernetes. So we have to use the internal IP address of the docker registry service instead - hence the use of those environment variables.

The IP address is fixed for the lifetime of the kubernetes service though; but yes its not ideal if you want to refer to images inside your Dockerfile.

One option is to expose the docker registry over ingress and use that name if you wish to refer to an image inside a Dockerfile in your source code. Though there's currently an issue with exposing the docker registry over ingress: https://github.com/jenkins-x/jx/issues/318

Another option is something like Google Container Builder or something where the docker registry lives outside of the kubernetes cluster. Fixing the ingress controller for the docker registry sounds a reasonable work around that'll work on any kubernetes cluster

[CobraFlow]

...it's a bit chicken and egg. The service IP is not known till it's up, but docker needs it at startup time.
The only way to do this is, as you say, use and external name whether that be an external service or back in via an Ingress.
I am going for the Ingress option. I would also be using Nexus for a Docker repo as well as Maven...

I'll keep you updated on how I get on.

[spennec]

I've had the same error with the node-http project.

Has anybody found a solution that corrects the issue?

[jstrachan]

@spennec sorry to hear that. What’s the output of:

jx version

[spennec]

jx version never returns. It shows a helm error

WARNING: Failed to find helm installs: Command failed 'helm list': Error: could not find a ready tiller pod exit status 1

Basically, I've installed jx through brew, as well as kubernetes-helm and kubectl. All installs were done 3 days ago so I expect to run on the latest versions. I run on Mac OS X, and my vms are on a free tier Azure subscription.

jx --version returns 1.2.46.

EDIT: Something was wrong with my Azure installations. I've deleted everything and re-installed. Here's the output of jx version

NAME               VERSION
jx                 1.2.46
jenkins x platform 0.0.922
kubernetes cluster v1.9.1
kubectl            v1.10.1
helm client        v2.9.0+gf6025bb
helm server        v2.9.0+gf6025bb
git                git version 2.10.1

Still have the same error.

[ysaakpr]

facing exactly same issue: though i am using the latest version,

Error: running skaffold steps: build: build step: running push: Get https://100.64.80.169:5000/v1/_ping: http: server gave HTTP response to HTTPS client

jx 1.2.55
jenkins x platform 0.0.976
kubernetes cluster v1.9.3
kubectl v1.9.2
helm client v2.9.0+gf6025bb
helm server v2.9.0+gf6025bb
git git version 2.7.4

[CobraFlow]

@ysaakpr Your issue is the same as mine. You have to set the insecure-registries on ALL the NODES.
I created a file /etc/docker/daemon.json with the contents:
{ "insecure-registries": [ "xxx.xxx.xxx.xxx:5000" ] }
where xxx.xxx.xxx.xxx is the ip address shown for the docker-registry SERVICE. This ip will not change unless you redeploy the service!
You need to reboot the nodes. I tried just restarting docker but that failed miserably!

[ysaakpr]

I am using kops to bring up my systems. How do i can make this changes in all nodes using kops or the kubernetes way ?...

[jstrachan]

You could try jx create cluster aws which does that for you via kops. Or you need to do it yourself via kops

[jstrachan]

Here’s how it works - you have to edit the kops json and update the cluster
https://github.com/jenkins-x/jx/blob/master/pkg/jx/cmd/create_cluster_aws.go#L251

[bremnes]

As @spennec, I'm experiencing this problem on Azure Kubernetes Services (AKS). I'm not to keen on doing manual changes on the worker nodes as it would make it more problematic when physically scaling the cluster.

@jstrachan, you mentioned a workaround by using an external docker registry. Can you give some details on how to do that? Which places needs to be changed; where do I add the authentication credentials for the registry; and other details you think might be of importance.

[rohit42412]

@bremnes I'm also getting the same error on AKS. Did you manage to resolve it?
Error: running skaffold steps: build: build step: running push: Get https://10.0.82.214:5000/v1/_ping: http: server gave HTTP response to HTTPS client

[rajdavies]

Can confirm AKS is currently broken -

[liuxiaopiao]

met the same error

[airwolfnh]

Still broken, but how can that be an AKS thing?

[jstrachan]

on AWS we use the IP range 100.64.0.0/10 to try allow insecure registries on any service IP addresses.

We are working on supporting external docker registries (e.g. ECR on AWS or GCR on GKE) which should be available in a week or two...

[liuxiaopiao]

In order to resolve the issue without restarting docker daemon, I am trying updating project's JenkinsFile and skaffold.yaml.
For Jenkinsfile, I have added one line such like
sh 'docker login xxxx.io -u xxxx -p $DOCKER_SECERET'
but the pipeline failed with below info:

`[xiaopiao_python-http_master-QAMIF6QMOWPXGN3STWZEJ3N4YNTDR4YKRG5ZGBBLRCRTY2U4H74A] Running shell script

• docker login XXXX.io -u XXXXX -p 'XXXXX'
  WARNING! Using --password via the CLI is insecure. Use --password-stdin.
  Error saving credentials: open /home/jenkins/.docker/config.json: read-only file system
  script returned exit code 1`

Could you please give some advice for this? Thanks a lot

[jstrachan]

if you want to update your config.json we are using a Secret to manage that - here's how to modify it: https://jenkins-x.io/architecture/docker-registry/#using-a-different-docker-registry
e.g.

kubectl delete secret jenkins-docker-cfg
kubectl create secret generic jenkins-docker-cfg --from-file=./config.json

[liuxiaopiao]

Thanks @jstrachan a lot . Will have a try.

[liuxiaopiao]

It works:
liuxiaopiao/python-http/master #6 2m53s Running Version: 0.0.18 Checkout Source 2m33s 5s Succeeded CI Build and push snapshot 2m28s NotExecuted Build Release 2m28s 1m56s Succeeded
so user can try:

1. Manage Jenkins -> Configure System and change the DOCKER_REGISTRY
2. Prepare the config.json(maybe just use docker login and get the ~/.docker/config.json file).
3. kubectl delete secret jenkins-docker-cfg
4. kubectl create secret generic jenkins-docker-cfg --from-file=./config.json

Thanks @jstrachan a lot for the help!

[leopoldodonnell]

I hit this too. The problem has a working (for me) solution for KOPS on the Jenkins X Install on Kubernetes page: https://jenkins-x.io/getting-started/install-on-cluster/

I didn't have much joy with the rolling update though - I would recommend tearing down your Jenkins X installation (helm list then do a helm delete --purge of all of the Jenkins packages`.

If you can, it may be less of headache to recreate your cluster then re-install.

[jstrachan]

I've added this to the FAQ along with workarounds as folks hit this fairly frequently: https://jenkins-x.io/faq/issues/#http-server-gave-http-response-to-https-client

[jstrachan]

BTW on AWS and EKS we now default to ECR to avoid this issue

[yuwzho]

Hi @jstrachan , do you have any guide how to configure the insecure registry? I meet the same question and the docker info shows

Insecure Registries:
 127.0.0.0/8

[jstrachan]

@yuwzho one simple workaround is to switch to using a public docker registry service like ECR/GCR/ACR (we do that by default on AWS + EKS).

The next approach is setting up the insecure registry port ranges on your docker daemons to a port range that matches kubernetes service IP ranges. We use 100.64.0.0/10 on AWS like this... https://github.com/jenkins-x/jx/blob/388c7f94d6822e8ac316ce9f20c280b86def8493/pkg/jx/cmd/create_cluster_aws.go#L101

[yuwzho]

@jstrachan Actually ACR is a private docker registry, and Azure has no public registry, insecure registry service now. So currently for the Azure scenario, something is blocked:

• The docker config is configured like

  https://github.com/jenkins-x/cloud-environments/blob/ce3116562e6fa7d8f2da0ee4e30091fadb7387b5/env-aks/myvalues.yaml#L6-L9

  And the insecure registry configured in the AKS is configured as 127.0.0.1/8

• If we switch to ACR, we need to modify the template in code repo to use imagePullSecrets and apply the ACR credential to every namespaces. Otherwise, the deployment to AKS will fail because of no credential to pull image from ACR. My question is whether there is a standard way to use private registry like ACR?

[jmlambert78]

I confirm the same issue with ACR. I may push by creating the "jx create docker auth " with the user & secret, and the helm deploy is successful but the images are nevel allowed to be pulled from the jx-staging or other environment.
What is strange is that was working with previous version of jx (I still have it running on another cluster)

[kumarinfo90]

@yuwzho
even i am trying to create cluster in azure and facing the same issue.
whether i can use docker hub as a registry?
if it is so guide me what to change in "Jenkins Mater's Manage Jenkins -> Configure System and change the DOCKER_REGISTRY environment variable"?
and /root/.docker/config.json?

[yuwzho]

@kumarinfo90 I am not using Docker Registry but Azure Container Registry, steps are here: https://github.com/jenkins-x/jx/issues/1805#issuecomment-437261518

[kumarinfo90]

@yuwzho Thanks yuwzho its working for me now.

[ccojocar]

Closing this issue since create spring seems to work now with ACR.