Dashboard: Look at cherry-picking the improvement for displaying step status into 0.6.x

[JustinKuli]

Kabanero might want/need this in a release sooner rather than later, so it would be good to get something ready for their testing (suggested by @a-roberts).

The commit is 09d5bce, from #1394

[JustinKuli]

My branch with b991cfc is based off of the v0.6.1.4 tag

[anneqm]

This would be a partial fix / improvement for https://github.com/tektoncd/dashboard/issues/1367.

[a-roberts]

My branch with b991cfc is based off of the v0.6.1.4 tag

Excellent, let's get this fully tested (so, on OpenShift and not), against the Tekton version we support on this release stream (current 0.11.x and 0.12.x) then I'll be happy to do a release.

The release process is that I'll basically check out your branch and then push it as a tag here for the Dashboard, kick off the release pipeline pointing to that tag and do the usual staging work. If that testing's already been done I can kick-start that process now, otherwise let's get folks assigned to this and go from there

[JustinKuli]

/assign

Beyond installing it, clicking around and running a pipeline or two, what sort of tests would make this "fully tested"?

[a-roberts]

/assign

Beyond installing it, clicking around and running a pipeline or two, what sort of tests would make this "fully tested"?

Yeah the coverage I'd be looking for, for this specific commit, would include...

• Creating PipelineRun from Import Resources
• Creating PIpelineRuns through the panelling
• Creating TaskRuns through the panelling
• Creating PipelineRuns through the webhooks extension

You'll want to check nothing odd occurs for both the good path, the error path, and the pending path - so for example when the PipelineRun passes, when it fails, and when it's still running. You've probably done this already - just check there's nothing odd popping up and we'll go from there.

[anneqm]

Prefer release 0.6.1.5 for less risk rather than 0.6.2.

[JustinKuli]

Tested on my docker desktop (kubernetes v1.16, tekton-pipeline v0.12.1):

• [x] Import Resources from github
• [x] Creating PipelineRuns through the panelling
• [ ] ~Creating TaskRuns through the panelling~ (TaskRun panelling not included in this branch)
• [ ] ~Creating PipelineRuns through the webhooks extension~ (I had my own problems with the webhooks today - probably not because of the change, but just because I hadn't tried to use them before on docker desktop: github wasn't able to deliver the request to tekton)

Anyway, things looked good while the pipeline was starting, running and afterwards.

I had the same results on my docker desktop cluster, using tekton-pipeline v0.11.3, but Rodolfo ran the same test on his openshift cluster and found that the image-digest step was appearing 2nd instead of 3rd. That meant that the 'bbb' step which should be failing was marked as "skipped", but it is an improvement over what was happening before because step 'aaa' was still appearing first and correctly marked as "completed".

When Rodolfo tested on pipeline 0.12, he had the same result as I did. We'll be getting more information from his tests, but here's the basic summary so far:

• We haven't found anything new that breaks
• On openshift with pipeline 0.11.x, the behavior might only be a small improvement: we seem to have examples where the last injected step is being placed before the failing step. I expected that would happen sometimes (when startedAt and finishedAt timestamps are equal), but not in the controlled Task that I created to test the fix.
• Using pipeline 0.12.x seems to be better from our tests so far, but there will still be times when the injected steps are placed incorrectly in the list.

[a-roberts]

Thanks @JustinKuli and @rnapoles-tktn - that's great info to know. I'll create a 0.6.1.5 release now based off your branch Justin and once that's done we can close this out.

[rnapoles-tktn]

After additional testing, mainly focusing on 0.11.3 pipelines and patching up from 0.6.1.4 dashboard I got the following:

Note that step bbb is displayed before step aaa.
Justin noticed that step bbb was started before step aaa:

steps:
    - container: step-aaa
        startedAt: '2020-05-27T12:37:49Z'
        finishedAt: '2020-05-27T12:37:55Z'
        reason: Error
    - container: step-bbb
        startedAt: '2020-05-27T12:37:38Z'
        finishedAt: '2020-05-27T12:37:55Z'
        reason: Error

@a-roberts could this be a pipelines issue?

[a-roberts]

AIAU the steps are supposed to be run in the order they are in your Pipeline (assuming you're not using things like runAfter), but there are no guarantees, @AlanGreene does this align with your findings too?

A simple https://github.com/tektoncd/pipeline/search?q=ordering&unscoped_q=ordering search like this shows there are attempts made at things being deterministic.

Let's see the example Pipeline spec you're using please and we can go from there (so, determining if the steps are really out of order in the Pipeline, at execution time when they shouldn't be, or if it's Dashboard behaviour misbehaving).

[AlanGreene]

It's most likely a display issue in the dashboard. We already know the timestamps aren't precise enough to fully rely on them for accurate ordering, this was noted in the recent change for the image-digest-exporter step.

The steps should be executed in order though, if that's not the case then it's a pipeline bug but I think this is unlikely.

[rnapoles-tktn]

It was a manual taskRun, I wanted to cycle through specific scenarios and this makes it easy to cycle through them, in this case this is the output of the taskRun:

apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  generateName: fail-bbb-skip-image-
  annotations:
    pipeline.tekton.dev/release: devel
  selfLink: >-
    /apis/tekton.dev/v1beta1/namespaces/openshift-pipelines/taskruns/fail-bbb-skip-image-7jnqs
  resourceVersion: '374715'
  name: fail-bbb-skip-image-7jnqs
  uid: 45c0d7f4-c755-4424-bdad-74aaed19fba0
  creationTimestamp: '2020-05-27T12:37:23Z'
  generation: 1
  namespace: openshift-pipelines
  labels:
    app.kubernetes.io/managed-by: tekton-pipelines
spec:
  resources:
    outputs:
      - name: image
        resourceSpec:
          params:
            - name: url
              value: fake-registry.io/fake-image
          type: image
  serviceAccountName: ''
  taskSpec:
    resources:
      outputs:
        - name: image
          type: image
    steps:
      - image: ubuntu
        name: aaa
        resources: {}
        script: |
          #!/usr/bin/env bash
          sleep 3
          echo "Step aaa will fail..."
          sleep 3
          exit 1
      - image: ubuntu
        name: bbb
        resources: {}
        script: |
          #!/usr/bin/env bash
          sleep 3
          echo "Step bbb will skip..."
          sleep 3
      - image: ubuntu
        name: ccc
        resources: {}
        script: |
          #!/usr/bin/env bash
          sleep 3
          echo "Step ccc will skip..."
          sleep 3
      - image: ubuntu
        name: ddd
        resources: {}
        script: |
          #!/usr/bin/env bash
          sleep 3
          echo "Step ddd will skip..."
          sleep 3
      - image: ubuntu
        name: eee
        resources: {}
        script: |
          #!/usr/bin/env bash
          sleep 3
          echo "Step eee will skip..."
          sleep 3
      - image: ubuntu
        name: fff
        resources: {}
        script: |
          #!/usr/bin/env bash
          sleep 3
          echo "Step fff will skip..."
          sleep 3
  timeout: 1h0m0s
status:
  completionTime: '2020-05-27T12:37:57Z'
  conditions:
    - lastTransitionTime: '2020-05-27T12:37:57Z'
      message: >-
        "step-aaa" exited with code 1 (image:
        "docker.io/library/ubuntu@sha256:5747316366b8cc9e3021cd7286f42b2d6d81e3d743e2ab571f55bcd5df788cc8");
        for logs run: kubectl -n openshift-pipelines logs
        fail-bbb-skip-image-7jnqs-pod-kr7w5 -c step-aaa
      reason: Failed
      status: 'False'
      type: Succeeded
  podName: fail-bbb-skip-image-7jnqs-pod-kr7w5
  startTime: '2020-05-27T12:37:23Z'
  steps:
    - container: step-aaa
      imageID: >-
        docker.io/library/ubuntu@sha256:5747316366b8cc9e3021cd7286f42b2d6d81e3d743e2ab571f55bcd5df788cc8
      name: aaa
      terminated:
        containerID: >-
          cri-o://c4099cf8b363c4c6ef39d4fb2cb45639f3f5e0d8ff4eda34dda50ab0fab3d6d9
        exitCode: 1
        finishedAt: '2020-05-27T12:37:55Z'
        reason: Error
        startedAt: '2020-05-27T12:37:49Z'
    - container: step-bbb
      imageID: >-
        docker.io/library/ubuntu@sha256:5747316366b8cc9e3021cd7286f42b2d6d81e3d743e2ab571f55bcd5df788cc8
      name: bbb
      terminated:
        containerID: >-
          cri-o://ede83bfe05c27bc3aa6a80a638ae9efc8a2a9ff45a37f7d931f4de8e9e67f783
        exitCode: 1
        finishedAt: '2020-05-27T12:37:55Z'
        reason: Error
        startedAt: '2020-05-27T12:37:38Z'
    - container: step-ccc
      imageID: >-
        docker.io/library/ubuntu@sha256:5747316366b8cc9e3021cd7286f42b2d6d81e3d743e2ab571f55bcd5df788cc8
      name: ccc
      terminated:
        containerID: >-
          cri-o://4fe888ca7d3f63ec1b85c98465f1aadd7e42ac7214e52369c51d7a180226f728
        exitCode: 1
        finishedAt: '2020-05-27T12:37:56Z'
        reason: Error
        startedAt: '2020-05-27T12:37:41Z'
    - container: step-ddd
      imageID: >-
        docker.io/library/ubuntu@sha256:5747316366b8cc9e3021cd7286f42b2d6d81e3d743e2ab571f55bcd5df788cc8
      name: ddd
      terminated:
        containerID: >-
          cri-o://8f19ba3e80c342f0427a6e01f1e2fad74a9be659a5dac67338904a26babcbc7c
        exitCode: 1
        finishedAt: '2020-05-27T12:37:56Z'
        reason: Error
        startedAt: '2020-05-27T12:37:43Z'
    - container: step-eee
      imageID: >-
        docker.io/library/ubuntu@sha256:5747316366b8cc9e3021cd7286f42b2d6d81e3d743e2ab571f55bcd5df788cc8
      name: eee
      terminated:
        containerID: >-
          cri-o://31a0d89491e31a20566744ac018ede7f284d744f910e4ea4850a5c9ec29c2410
        exitCode: 1
        finishedAt: '2020-05-27T12:37:56Z'
        reason: Error
        startedAt: '2020-05-27T12:37:45Z'
    - container: step-fff
      imageID: >-
        docker.io/library/ubuntu@sha256:5747316366b8cc9e3021cd7286f42b2d6d81e3d743e2ab571f55bcd5df788cc8
      name: fff
      terminated:
        containerID: >-
          cri-o://0668efd47c04b031d53faa432e2d53538e31e5e2539e57f97487a1783f9ff117
        exitCode: 1
        finishedAt: '2020-05-27T12:37:56Z'
        reason: Error
        startedAt: '2020-05-27T12:37:47Z'
    - container: step-create-dir-image-hflcm
      imageID: >-
        docker.io/library/busybox@sha256:836945da1f3afe2cfff376d379852bbb82e0237cb2925d53a13f53d6e8a8c48c
      name: create-dir-image-hflcm
      terminated:
        containerID: >-
          cri-o://ca7f9bff3c1dccde1bd4448f3060f3986fd5b2c2bdeab375b0aa007032d3243b
        exitCode: 0
        finishedAt: '2020-05-27T12:37:49Z'
        reason: Completed
        startedAt: '2020-05-27T12:37:49Z'
    - container: step-image-digest-exporter-4j8pz
      imageID: >-
        gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/imagedigestexporter@sha256:c0ac90740dc6d3771f77f8bca5e3fef6b5bb97b7eef189f90488553a874aa602
      name: image-digest-exporter-4j8pz
      terminated:
        containerID: >-
          cri-o://4a812f633f87aaf234df139a203294d49a35b8f8b110e3438d1e6dda8c42c2fa
        exitCode: 1
        finishedAt: '2020-05-27T12:37:57Z'
        reason: Error
        startedAt: '2020-05-27T12:37:48Z'

[JustinKuli]

From what I see in the full TaskRun yaml, the "startedAt" timestamps from pipelines are inconsistent... The log on step bbb does show that it went after aaa, it's just not reported well by those timestamps (which is what the "improved" code in the dashboard uses)

[a-roberts]

This is done and made it into the 0.6.1.5 release, let's work on anything else in new issues with master as our codebase and Tekton Pipelines master, where we should see the most improvements.

[JustinKuli]

Just to provide some more information about why @rnapoles-tktn's test had such an odd result: before pipeline 0.12.0, the startedAt timestamp can't be relied on in the way I was expecting.

https://github.com/tektoncd/pipeline/pull/2429 (released in pipeline 0.12.0) ensures that this timestamp is correctly set even when steps are skipped.