Jx: Jenkins X Skaffold issues

[igdianov]

Jx uses old patched skaffold binary that runs inside Jenkins X build container pipeline. The sh skaffold run -f skaffold.yaml step always deploys a ghost pod named skaffold into the cluster after building Docker images. The ghost deployment is caused by the empty kubectl manifests section in skaffold.yaml, which causes kubectl deployment manifests auto-generation with other side-effects as noted here: #907

apiVersion: skaffold/v1alpha2
kind: Config
build:
  tagPolicy:
    envTemplate:
      template: "{{.JENKINS_X_DOCKER_REGISTRY_SERVICE_HOST}}:{{.JENKINS_X_DOCKER_REGISTRY_SERVICE_PORT}}/{{.ORG}}/{{.APP_NAME}}:{{.VERSION}}"
  artifacts:
  - imageName: changeme
    workspace: .
    docker: {}
  local: {}
deploy:
  kubectl:
    manifests:

The latest official skaffold release 0.7.0 has already removed auto-generated manifests support in: GoogleContainerTools/skaffold#640. It also provides build command with skaffold build -f skaffold.yaml to build Docker images and halt after that.

The Jx skaffold version uses the unsupported setValueTemplates spec in skaffold.yaml dev profile which limits running skaffold only inside Jenkins X CI pipelines.

profiles:
- name: dev
  build:
    tagPolicy:
      envTemplate:
        template: "{{.JENKINS_X_DOCKER_REGISTRY_SERVICE_HOST}}:{{.JENKINS_X_DOCKER_REGISTRY_SERVICE_PORT}}/{{.ORG}}/{{.APP_NAME}}:{{.DIGEST_HEX}}"
    artifacts:
    - docker: {}
    local: {}
  deploy:
    helm:
      releases:
      - name: demo
        chartPath: target/charts/demo
        setValueTemplates:
          image.repository: "{{.JENKINS_X_DOCKER_REGISTRY_SERVICE_HOST}}:{{.JENKINS_X_DOCKER_REGISTRY_SERVICE_PORT}}/{{.ORG}}/{{.APP_NAME}}"
          image.tag: "{{.TAG}}"

It would be great to use latest skaffold official releases to allow flexible CI/CD docker builds inside Jenkins X pipelines with full support of multiple image artifacts inside skaffold.yaml and custom Dockerfiles as well as reusing the same skaffold.yaml for local dev workflows.

What is the roadmap to fix skaffold issues in Jenkins X? The workaround at the moment is to patch JX build container using skaffold 0.7.0 official release at runtime and remove setValueTemplates from skaffold.yaml unitl the GoogleContainerTools/skaffold#487 gets merged into official skaffold to support dev CI/CD worflows.

$ jx version

NAME               VERSION
jx                 1.2.100
jenkins x platform 0.0.1193
kubernetes cluster v1.8.10-gke.0
kubectl            v1.10.3
helm client        v2.9.1+g20adb27
helm server        v2.9.1+g20adb27
git                git version 2.9.0.windows.1

[jstrachan]

I'm hoping we can move back to the upstream skaffold version ASAP (and keep tracking it as it releases new versions). We had a number of issues we raised PRs for on skaffold; just need to check everything's working with the upstream skaffold. I think this is the remaining PR that we need to fix up/get merged https://github.com/GoogleContainerTools/skaffold/pull/487

[igdianov]

@jstrachan That would be awesome! Thank you very much.

[jstrachan]

I've resubmitted the remaining PR on Skaffold now: https://github.com/GoogleContainerTools/skaffold/pull/707

though I hit this regression: https://github.com/GoogleContainerTools/skaffold/issues/709

[r2d4]

Those two issues should be fixed and published in the v0.8.0 release. Let me know if that works!

[jstrachan]

@r2d4 its looking really good; though unfortunately we've found one more minor regression: https://github.com/GoogleContainerTools/skaffold/issues/741

[bhack]

Is it fixed right?

[jstrachan]

@bhack its fixed now yeah - hooray! - just waiting for a new release and hopefully we're good to go

[bhack]

We have a new release

[jstrachan]

Latest platform now used 0.9.0 of upstream skaffold

[sebosp]

I'm on jx 1.3.72, testing angular-io-quickstart on dev pods, using skaffold v0.9.0.
Running skaffold dev -p dev, was getting an InvalidImageName as {{.TAG}} wasn't apparently being resolved and the image name resulted in 10.11.250.208:5000/sebosp/sebosp-main: in the deployment.
I did a jx import just in case as the jx version seems to always suggest an upgrade. Then on skaffold.yaml I had to change the setValueTemplates helm deploy reference from:
image.tag: "{{.TAG}}"
to:
image.tag: "{{.DIGEST_HEX}}"
And was able to get skaffold dev -p dev working.
That did the trick for me, not sure this is the intended behavior tho.