Dashboard: v0.7.0 image tag references a possibly broken image

[dghubble]

Describe the bug

The dashboard v0.7.0 manifests refer to image gcr.io/tekton-releases/github.com/tektoncd/dashboard/cmd/dashboard@sha256:9ba9c755e868942143868dbbc7a0d9ab29241014833dd0a7ac6156d66e45fe9b. This image works ok and seems like the one the team tested and intended for release.

The gcr.io v0.7.0 image tag refers to cd3f480d472ea2dc270eeb2cb36f558ca6bd9cf92f92e493846e42dd75069ccf which runs, but seems to return a 404 for every page without any errors in logs (debug level). Seems like a publish/release problem rather than code.

I noticed due to mainly using image tags.

Expected behaviour

The release manifests can use an image tag or sha256 (preference), but the published image tag should probably align with that choice. In this case it causes an issue because the cd3f480d472e image seems to have a problem.

Environment details

• Kubernetes version: v1.18.4
• Cloud-provider/provisioner: AWS
• Versions:
  
  
  
  • Tekton Dashboard: v0.7.0
  
  
  • Tekton Pipelines: v0.13.2
  
  
  • Tekton Triggers: v0.13.2
  
  
• Install namespaces:
  
  
  
  • Tekton Dashboard: tekton
    
    
  
  
  • Tekton Pipelines: tekton
    
    
  
  
  • Tekton Triggers: tekton
    
    
  
  

Additional Info

[a-roberts]

I think I know why, I did a test release for @eddycharly for https://github.com/tektoncd/dashboard/pull/1537 and I used the same version tag (v0.7.0) instead of something arbitrary, like mycooldebugversion 😄

I can kick off the normal release pipeline again with this version tag (v0.7.0) to override that, with the one it should be.

[dghubble]

Since that image sha unusably borked, sounds reasonable to me if you point it at the intended sha. Can't see any complaints :+1:

[a-roberts]

Since that image sha unusably borked, sounds reasonable to me if you point it at the intended sha. Can't see any complaints 👍

Yeah, my bad - I hadn't appreciated the clobber (if my theory is correct, the times do add up though), so hopefully in around 15 minutes time - (I just kicked off the pipeline again), that image on GCR will be the correct one. Thanks for spotting this, will post an update once resolved (or borked)

[a-roberts]

Can you try gcr.io/tekton-releases/github.com/tektoncd/dashboard/cmd/dashboard@sha256:1d70cf5eb39f5b892d3bf0f1007aca6d3e84ecbe48dc2716ff473b52338a135d now? From https://console.cloud.google.com/gcr/images/tekton-releases/GLOBAL/github.com/tektoncd/dashboard/cmd/dashboard?gcrImageListsize=30 with the v0.7.0 tag

[dghubble]

That image works for me now. Many thanks and cheers for v0.7.0!

[dghubble]

@a-roberts I see this issue on v0.7.1, 0.7.1-rc1, and sha256:4c1d0c9d3bd805c07f57ae6974bc7179b03d67fa83870ea8a71415d19c261a38. Debug logs look normal, nothing unusual compared with v0.7.0. But every path returns a 404 page not found. Rolling back to v0.7.0 solves it. Another problem in image publishing?

Also, sha256:3cb4da36deda733d00096d4409fb9573b809c7c1ec238844b4a64452f18c7a46 works, which is a version between v0.7.0 and what's tagged as v0.7.1.

[a-roberts]

@a-roberts I see this issue on v0.7.1, 0.7.1-rc1, and sha256:4c1d0c9d3bd805c07f57ae6974bc7179b03d67fa83870ea8a71415d19c261a38. Debug logs look normal, nothing unusual compared with v0.7.0. But every path returns a 404 page not found. Rolling back to v0.7.0 solves it. Another problem in image publishing?

Also, sha256:3cb4da36deda733d00096d4409fb9573b809c7c1ec238844b4a64452f18c7a46 works, which is a version between v0.7.0 and what's tagged as v0.7.1.

@AlanGreene ^^

[AlanGreene]

@dghubble I'm not seeing an issue with those images. How are you installing the dashboard, i.e. which release manifest / command are you using?

Are you manually updating the image reference in an older release manifest perhaps? If so, we made a change in 0.7.1 to the location of the static resources for the front end. This is configured via the --web-dir arg to the dashboard container, the previous value was /var/run/ko/web but now should be /var/run/ko

[dghubble]

Using manifests with read-only and single-namespace mode (similar to tekton-dashboard-release-readonly.yaml, but with stricter RBAC and constraints for our purposes).

we made a change in 0.7.1 to the location of the static resources for the front end

Changing the --web-dir argument fixed it, thanks! I do check the release notes or for a changelog when upgrading but hadn't noticed the change in your examples. Is there a reason users supply the path to what seems to be an internal detail?

[AlanGreene]

Can you share more details about the customisations you make to the RBAC? We've recently introduced a new installer script (still experimental / work in progress) that should help with many common customisations to the dashboard install, so it would be interesting to see what other cases people are addressing themselves so we can either consider including them or even just documenting common patterns.

Is there a reason users supply the path to what seems to be an internal detail?

Good question. It's come in handy in the past when experimenting with building on different platforms / with different tooling versions where the default location varied. This may not be needed anymore. @a-roberts @eddycharly any objections to removing this arg and hardcoding the path?

[eddycharly]

This may not be needed anymore. @a-roberts @eddycharly any objections to removing this arg and hardcoding the path?

+1 from me, i don't know of any use case that require changing it.

Eventually we discovered that it can introduce issues with ko because of files access mode.

[dghubble]

Sure, if you're interested. Plain Kubernetes manifests are managed in version control (across all kinds of workloads and clusters). Helm and templating aren't used. Customizations include seccomp profiles, dropping capabilities, and using an RBAC policy that can be more strict by forbidding features that are not used in this case. Also adds a custom oauth2 sidecar and makes minor mods to labels and port names (miscellaneous internal stylistic changes).

For RBAC, with --namespace=foo the ideal would be to eliminate ClusterRoles. Currently dashboard does still need a ClusterRole that can list namespaces even in this mode, but most other things are eliminated. Stuff like the dashboard trying to read deployments and services to introspect the version of Tekton triggers and pipelines is forbidden. That UI page will have errors, but overall that functionality isn't worth the permission grant. That kinda thing.

For --web-dir, I guess my hope would be if I don't specify it, the right default is used. But I just tested and currently it does still need to be set explicitly.

[eddycharly]

@dghubble thanks for your feedback.

I thought about using Roles instead of ClusterRoles, I think it's doable and will give it a try. If it works it will be easy to do with the installer script but was a lot harder before when we used kustomize/overlays.

Now you can use installer build to generate the manifests with your custom params, it should be easier that writing manifests from scratch.

For --web-dir, default value is not correct:

webDir             = flag.String("web-dir", "", "Dashboard web resources dir")

We should definitely remove it, more harm than good.

[dghubble]

Yeah, I appreciate the efforts on installation approaches, I know a lot of folks prefer generation or templating.

For the usage style I described, across our managed workloads, its been helpful to have the source of truth be raw manifests for auditing and mass edit tooling, with occasional comparison against what individual project's generation tools would produce. So I'm maybe not the best candidate for generator tools, but glad if it helps Tekton adoption.