Jx: Promotion build status is still running after promotion succeeded

[romainverduci]

Summary

When you run jx promote to promote an app in production, it triggers a Promote stage in the app release build you want to promote. Once the promotion is done and all the steps are Succeeded, the build status is not updated and stuck in Running

Steps to reproduce the behavior

• jx promote -a your-app -v 0.0.1 -e production
• Monitor the application master build
• When the promotion is done, check the status of the build

Expected behavior

Build should be marked as 'Succeeded'

Actual behavior

Build is stuck in Running status

Here is the pipeline activity:

apiVersion: jenkins.io/v1
kind: PipelineActivity
metadata:
  creationTimestamp: "2019-10-08T19:17:45Z"
  generation: 23
  labels:
    branch: master
    build: "4"
    owner: owner
    provider: github
    repository: demo-app
    sourcerepository: owner-demo-app
  name: owner-demo-app-master-4
  namespace: jx
  resourceVersion: "4277396"
  selfLink: /apis/jenkins.io/v1/namespaces/jx/pipelineactivities/owner-demo-app-master-4
  uid: 4ef1c46f-ea00-11e9-b91f-42010a8400d2
spec:
  author: romainverduci
  batchPipelineActivity: {}
  build: "4"
  buildLogsUrl: gs://cluster-name-logs-8b3b8dea-df95-471b-be7a-9e42dce39a6c/jenkins-x/logs/owner/demo-app/master/4.log
  completedTimestamp: "2019-10-08T19:20:03Z"
  gitBranch: master
  gitOwner: owner
  gitRepository: demo-app
  gitUrl: https://github.com/owner/demo-app.git
  lastCommitMessage: Draft create
  lastCommitSHA: 08982cdc94fc574a2ffd4f1c3e3f1d84b1e2365c
  pipeline: owner/demo-app/master
  releaseNotesURL: https://github.com/owner/demo-app/releases/tag/v0.0.4
  startedTimestamp: "2019-10-08T19:17:47Z"
  status: Running
  steps:
  - kind: Stage
    stage:
      completedTimestamp: "2019-10-08T19:20:03Z"
      name: from build pack
      startedTimestamp: "2019-10-08T19:17:47Z"
      status: Succeeded
      steps:
      - completedTimestamp: "2019-10-08T19:17:47Z"
        name: Credential Initializer Txqft
        startedTimestamp: "2019-10-08T19:17:47Z"
        status: Succeeded
      - completedTimestamp: "2019-10-08T19:17:49Z"
        name: Working Dir Initializer 5j2b8
        startedTimestamp: "2019-10-08T19:17:47Z"
        status: Succeeded
      - completedTimestamp: "2019-10-08T19:17:50Z"
        name: Place Tools
        startedTimestamp: "2019-10-08T19:17:49Z"
        status: Succeeded
      - completedTimestamp: "2019-10-08T19:17:58Z"
        description: https://github.com/owner/demo-app.git
        name: Git Source Core Platform Test Org Demo App 4pklv
        startedTimestamp: "2019-10-08T19:17:50Z"
        status: Succeeded
      - completedTimestamp: "2019-10-08T19:17:58Z"
        name: Git Merge
        startedTimestamp: "2019-10-08T19:17:58Z"
        status: Succeeded
      - completedTimestamp: "2019-10-08T19:17:59Z"
        name: Setup Jx Git Credentials
        startedTimestamp: "2019-10-08T19:17:58Z"
        status: Succeeded
      - completedTimestamp: "2019-10-08T19:18:06Z"
        name: Build Make Build
        startedTimestamp: "2019-10-08T19:17:59Z"
        status: Succeeded
      - completedTimestamp: "2019-10-08T19:18:13Z"
        name: Build Container Build
        startedTimestamp: "2019-10-08T19:18:06Z"
        status: Succeeded
      - completedTimestamp: "2019-10-08T19:18:14Z"
        name: Build Post Build
        startedTimestamp: "2019-10-08T19:18:13Z"
        status: Succeeded
      - completedTimestamp: "2019-10-08T19:18:20Z"
        name: Promote Changelog
        startedTimestamp: "2019-10-08T19:18:14Z"
        status: Succeeded
      - completedTimestamp: "2019-10-08T19:18:26Z"
        name: Promote Helm Release
        startedTimestamp: "2019-10-08T19:18:20Z"
        status: Succeeded
      - completedTimestamp: "2019-10-08T19:20:03Z"
        name: Promote Jx Promote
        startedTimestamp: "2019-10-08T19:18:26Z"
        status: Succeeded
  - kind: Promote
    promote:
      applicationURL: http://demo-app.jx-staging.X.X.X.X.nip.io
      completedTimestamp: "2019-10-08T19:20:03Z"
      environment: staging
      pullRequest:
        completedTimestamp: "2019-10-08T19:20:02Z"
        mergeCommitSHA: dff4ba2d0cb1b1b725ff5ffe602d2d615bdeb4ed
        pullRequestURL: https://github.com/owner/environment-cluster-name-staging/pull/67
        startedTimestamp: "2019-10-08T19:18:35Z"
        status: Succeeded
      startedTimestamp: "2019-10-08T19:18:35Z"
      status: Succeeded
      update:
        completedTimestamp: "2019-10-08T19:20:03Z"
        startedTimestamp: "2019-10-08T19:20:02Z"
        status: Succeeded
  - kind: Promote
    promote:
      completedTimestamp: "2019-10-09T09:20:07Z"
      environment: production
      pullRequest:
        completedTimestamp: "2019-10-09T09:20:06Z"
        mergeCommitSHA: 6e7d24b256b65991cf21df16a0e06cb15f44535f
        pullRequestURL: https://github.com/owner/environment-cluster-name-production/pull/42
        startedTimestamp: "2019-10-09T09:18:37Z"
        status: Succeeded
      startedTimestamp: "2019-10-09T09:18:37Z"
      status: Succeeded
      update:
        completedTimestamp: "2019-10-09T09:20:07Z"
        startedTimestamp: "2019-10-09T09:20:06Z"
        status: Succeeded
  version: 0.0.4
  workflowStatus: Running
status: {}

Jx version

jx version
NAME               VERSION
jx                 2.0.721
Kubernetes cluster v1.13.7-gke.8
kubectl            v1.16.1
helm client        Client: v2.14.2+ga8b13cc
git                2.21.0 (Apple Git-122)
Operating System   Mac OS X 10.14.6 build 18G95

Jenkins type

• [x] Serverless Jenkins X Pipelines (Tekton + Prow)

Kubernetes cluster

GKE

Operating system / Environment

Mac OS

[abayer]

I’ve seen this with builds other than ones with promotions too. It doesn’t seem to be consistent. I’ll look into this.

[abayer]

Ok, the cases I'm seeing on prod are where there are multiple stages but the first one failed, so the second one never gets out of Pending. Probably unrelated to this, but I'll make sure before I create a new issue.

[abayer]

Yup, that's definitely a different problem - issue #5779 created for it, PR #5780 fixes it. Back to this problem... =)

[abayer]

@romainverduci says that he's only sure he's seen it for when jx promote is run outside of a pipeline, promoting to the production environment, which I know is something that tends to be a bit weird. Working on reproducing that now.

[abayer]

Ok, so in BDD tests, when it runs the manual promotion, I see:

jenkins-x/jx/PR-5780 #3                                                                          Running Version: 0.0.1
  Release                                                                          3m2s     1m0s Succeeded
  Promote: production                                                              2m2s    1m28s Succeeded
    PullRequest                                                                    2m2s    1m28s Succeeded  PullRequest: https://github.com/jenkins-x-bot-test/environment-pr-5780-3-tekton-gkebdd-production/pull/1 Merge SHA: 6f8caea41db4826fff6d10d5e0422d310c14dddc
    Update                                                                          34s       0s Succeeded

That is - it looks like the manual promotion creates an activity (or updates an existing activity) for whatever your current repo/branch is, and that activity never finishes. I'm going to ignore the weirdness of what it decides re: what activity to create/update and instead focus on making it finish.

[abayer]

Ah-ha - looks like controller_workflow is what would see that the update finished and mark the activity as completed, but we don't use controller_workflow with Tekton, just controller_build. And controller_build just listens for pod updates. Need to think about this more thoroughly to figure out how we can detect that completion when there's no pod associated.

[abayer]

So the manual promote update step is always instantaneous now - it feels like it probably...shouldn't be? We should probably be rethinking how promote activity steps get handled/recorded/etc in general as part of #5781, but for now I'm leaning towards just hacking kube.OnPromoteUpdate to automatically set the overall activity status to success/failure (as appropriate) when it claims that the update step is finished. Probably need comparable behavior in the other kube.OnPromote... functions to check for failures and update the PipelineActivity accordingly as well.