Super-linter: Unexpected linter update

[leviem1]

Describe the bug
Linter is running latest v3 tag (v3.4.0) despite being set to github/[email protected].

To Reproduce
Steps to reproduce the behavior:

1. Use github/[email protected] in GitHub action.
2. Push code. It is then linted by latest v3 image (v3.4.0 currently).

Expected behavior
Linting should run at the version I set it to (v3.3.2).

Screenshots
Not applicable

Additional context
Example Run
I use github/[email protected] over docker://github/super-linter:v3.3.2 for Dependabot updates.
Introduced by #428 / 70f801e

[GaboFDC]

Given the change on #428 is effectively the same to use github/[email protected] than docker://github/super-linter:v3.3.2 it will always pull the image.
Maybe we should allow the user to build the image if he wants to?

[leviem1]

Yeah, this goes back to the slow image discussion. It seems like we can either give the user a performance boost, or give them the ability to choose their version. In my opinion, the latter is more important for stability. I'd rather intentionally wait for 10 minutes to build the image on a specific version than to have forced upgrades.

This would make updates work as expected on repositories that choose to use the Action syntax, but also allows users to sacrifice automated updates (until Dependabot ships their Actions Docker support) for using the pre-compiled container if needed. The way I see it, it comes down giving the user a choice between stability and performance, but I think it's important to _let_ the user decide.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity.
It will be closed in 14 days if no further activity occurs.
Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

[alerque]

@github-actions The only thing stale around here is you. Not only is this issue not fixed and very much still a problem, but the current hack that leads to this problem is to work around a serious shortcoming in Actions itself (that's you) whereby one cannot use the default uses syntax and get cached builds — which is just crazy pants. The only way to do is is to use off-site Docker images. And even Github's own package repository that should be a great way to do that is unusable because one can't pull public images without authentication. Hence why this project of Github's very own is using both a hack to not use the usual build process they force on everybody else and outsourcing the Docker part to a 3rd party.

[peternewman]

Given the change on #428 is effectively the same to use github/[email protected] than docker://github/super-linter:v3.3.2 it will always pull the image.
Maybe we should allow the user to build the image if he wants to?

Is the workaround not to update action.yml each time a new release is made? You're already generating Docker versions of each release:
https://github.com/github/super-linter/packages/106548/versions

And a tag for each release:
https://github.com/github/super-linter/tags

So if the tag's action.yml contained an instruction which used the Docker's specific version, you'd always get what you wanted (with the possible exception of master being one version behind I think).

[alerque]

@peternewman No that's not the issue at all and hence not a solution. The issue is they are hijacking the substance of the action by punting it from the usual build process to a Docker pull and not passing the version through at all, it's hard coded to 'latest' so even if you explicitly uses something older you get fast forwarded to whatever the 3rd party container registry thinks is latest.

[peternewman]

I've not done much Docker but this suggests you can run a particular versioned image:
https://github.com/github/super-linter/packages/106548?version=v3.3.2

I don't understand what you mean about not passing the version through? I assume you're talking about action versions not being passed to Docker versions? When you uses it selects a tag or branch. I'm suggesting if each tag was hard-coded to it's named version, I believe it would all work fine. So action.yml for version 3.3.2 would be:

runs:
  using: 'docker'
  image: 'docker://github/super-linter:v3.3.2'

For 3.8.3 would be:

  image: 'docker://github/super-linter:v3.8.3'

etc. It wouldn't magically fix old releases unless they were all re-released, but I think it should solve the problem going forward.

[leviem1]

Cheeky little workaround. I think it would be a good resolution moving forward to at least get the versioning part sorted, which is what I mostly wanted to address here.

That being said @alerque is talking about the performance impact for using a Docker action using image: 'Dockerfile' instead of the pre-compiled Docker hack. Since Actions does not cache the images between runs, it has to be rebuilt upon every execution, unless using an external service to host the images. This was talked about more in #7 and it was mentioned that this may be a conversation for actions/cache#31.

I also found the comment about using Github Packages interesting. From what I can tell, that should actually be quite possible using GITHUB_TOKEN or a Personal Access Token as your authentication, though I didn't find much on how to actually use it.

[alerque]

@peternewman You don't even have to be that tricky, you can get any old release you want using a regular action step, you just cant use the regular uses syntax. Substitute:

uses: github/[email protected]

...which currently is hijacked and actually gives you latest instead, with this:

uses: docker://github/super-linter:v3.3.2

That will get you the real thing using Docker directly. The trouble is this isn't how the Actions marketplace advertises nor is it documented that their recommended usage is actually hijacking everybody and giving them latest instead. That's what this issue is about. The reason they are hijacking these requests has to do with caching and limitations of their build system not having a cache feature yet (which is for a different issue, this is just an interesting case study on how much of a problem it is).

[peternewman]

Cheeky little workaround. I think it would be a good resolution moving forward to at least get the versioning part sorted, which is what I mostly wanted to address here.

Thanks.

That being said @alerque is talking about the performance impact for using a Docker action using image: 'Dockerfile' instead of the pre-compiled Docker hack. Since Actions does not cache the images between runs, it has to be rebuilt upon every execution, unless using an external service to host the images. This was talked about more in #7 and it was mentioned that this may be a conversation for actions/cache#31.

Yeah understood, but if you can automate most of it, then you can use the pre-compiled ones with no real impact. I'd agree though, it seems mad to me that they aren't cached. Although is it something to do with the fact you could influence the config via the inputs, and therefore it might change between runs and so wouldn't be cacheable? It does seem crazy there's not an option you can turn on if yours is cache safe though.

I also found the comment about using Github Packages interesting. From what I can tell, that should actually be quite possible using GITHUB_TOKEN or a Personal Access Token as your authentication, though I didn't find much on how to actually use it.

So I haven't done it, but an action I based some of mine off does:
https://github.com/TrueBrain/actions-flake8/blob/master/.github/workflows/publish_stable.yml

I see there is similar in this repo, although it looks more complicated here than just using GITHUB_TOKEN.

@peternewman You don't even have to be that tricky, you can get any old release you want using a regular action step, you just cant use the regular uses syntax.

Yes, I appreciate you can run a Docker image at a certain version manually.

That will get you the real thing using Docker directly. The trouble is this isn't how the Actions marketplace advertises nor is it documented that their recommended usage is actually hijacking everybody and giving them latest instead.

I think that's only the case because of the way and the fact this repo uses a Docker image.

That's what this issue is about. The reason they are hijacking these requests has to do with caching and limitations of their build system not having a cache feature yet (which is for a different issue, this is just an interesting case study on how much of a problem it is).

I agree it should be fixed at source, but have I missed something, or would my suggested fix not give the best of both worlds (until GitHub adds caching) by enabling people to just use the action by following the Marketplace documentation, while also benefiting from the faster runtimes available by using the Docker cached version?

[alerque]

Oh look, GitHub just released an update to their container registry that directly addresses one of the road-blocks to fixing thing issue. Specifically anonymous fetching of public images. Finally. Now to see if they follow it up with automatically matching prebuilt images with repo tags and using that in lieu of rebuilds when there is an exact match.

[hoffmang9]

Just got hit with this issue again. I'm specifying github/[email protected] but I'm pretty sure I got 3.10.0 whether I liked it or not. I didn't like it because it's changed the results for unchanged files. It also seems to now have a version of black that wants to make changes on the ci that it doesn't want to make locally...

Does docker://github/super-linter:v3.9.4 still work as a workaround?

[alerque]

@hoffmang9 Yes this is still very broken, and yes that workaround should still work.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity.
It will be closed in 14 days if no further activity occurs.
Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

[alerque]

Still very much broken.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity.
It will be closed in 14 days if no further activity occurs.
Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

[alerque]

Still broken. Get your bot off our backs, activity has nothing to do with it when the upstream project is the one letting everything slide. If you were an active project that actually triaged issues this would be a non-issue.

[leviem1]

Yes, not to beat a dead horse, but this behavior is confusing and frustrating. Builds fail without reason (well, the reason being that the linter broke in an update as we've seen happen in many patches lately) and can be hard to trace since two identical builds may have two different outcomes. Basically, using superlinter in your builds will lead to nondeterministic outcomes.

Again, just a slight change to the way releases are handled may improve this significantly. @alerque Are you happy with the new GHCR? If I remember correctly, that was half of this battle, the versioning being the second half. Correct me if I'm mistaken.

[alerque]

@leviem1 I saw some comments about the GHCR situation changing and remember thinking that I needed to look into if it could be leveraged to fix my own projects now. That doesn't solve the versioning issues here though. Assuming it can (an anonymous pulls work) they should look into that being a standard Actions thing ASAP. This situation is beyond crazy.

Meanwhile I've dropped SuperLinter form all but a few non-critical projects because it's so unpredictable and release management is so bad (a combination of new releases so frequently being broken and not having a choice of not having them forced on us the moment they are released).

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity.
It will be closed in 14 days if no further activity occurs.
Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

[hoffmang9]

Not stale.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity.
It will be closed in 14 days if no further activity occurs.
Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

[alerque]

Stagnate would be the word, not stale.

[ahmadnassri]

anybody aware of other actions suffering from this? _(all of mine are since I decided to stick with Docker as the delivery mechanism)_

perhaps by linking all the issues across the different action authors we can get GitHub to pay attention

[zkoppert]

Hey super-linter users and contributors, sorry for not jumping on this 🤦🏻 . I'm taking a look at this now and see if we can't get this turned around.

[alerque]

@zkoppert This isn't just a SuperLinter issue—although this is an egregious example of why hacking around the real problem creates other problems. The real issue is that the Actions runner doesn't cache builds. Presumably this is because of security concerns of caching something triggered by some random repo. Just invert it and let the upstream action tag and prime the cache. The GH Actions runner should check if there is a Docker image matching the tag in the GH Packages repo for the action project. If it exists, use it instead of rebuilding the entire container from scratch.

[zkoppert]

still working on this...

[leviem1]

Oh sweet, thanks for picking this up @zkoppert!

It's a tricky one for sure. Looks like you tried the versioning hack. Is that method flawed or is some automation around it still not working? Just curious

Also, for @alerque's sake, can you please give us a "yes", "no", or "not allowed to say either way" on if there's anything being worked on for using cached builds when specifying image: 'Dockerfile' in the action.yml? Are there any suggestions for other ways to achieve this behavior for the time being?

(I'm also curious about the technical details of why this won't be worked on if that's the case, but that's just for my own understanding)

[zkoppert]

just working on the automation around it right now. Should be working in a bit.

As far as the cached builds, I honestly don't know the answer. I can see if I can get the right people at GitHub to follow up.

[leviem1]

Awesome, thanks so much! Unless you're on call or bored or something, consider starting your weekend! Late night pushes are usually the messy ones 😉

[zkoppert]

☝🏻 so true. Breaking lots of best practice rules (no deploying on a friday, no changes straight to master, no releases without reviews). Wild times! 🤠

[yaananth]

Also, for @alerque's sake, can you please give us a "yes", "no", or "not allowed to say either way" on if there's anything being worked on for using cached builds when specifying image: 'Dockerfile' in the action.yml? Are there any suggestions for other ways to achieve this behavior for the time being?

https://github.com/actions/runner/blob/007ac8138b0d7a6a0b236c8026906c84c78980e4/src/Runner.Worker/Handlers/ContainerActionHandler.cs#L47 is the code that builds the docker image everytime.

An issue at https://github.com/actions/runner/issues is probably relevant here.

In general for docker caching though, https://github.com/actions/cache/issues/31 might help.

[leviem1]

@alerque Also, check out the new thing in the Feature Preview named "Improved Container Support". This might be a bit closer to what you're looking for.

You can more easily share containers in your organization, set granular access permissions, and anonymously access public container images. Additionally, layers are shared between images across GitHub.

@zkoppert Sorry to treat you as the spokesperson for GH, but do you have any more info on this? Docs are a bit underwhelming as is expected for something that's in preview. Also, thanks for taking a swing issue at this recently. I know there are bigger fish to fry but just knowing it's on your radar is reassuring.

[zkoppert]

@leviem1 lol yes, being a spokesperson for GH is a big responsibility that can be really difficult especially with the number of things we are working on. To dive into the docs you quoted with my own understanding:

more easily share containers in your organization

This means that you will be able to have a repo in the organization that hosts multiple organization wide packages.

set granular access permissions

I believe that this refers to being able to set scope of the visibility/permissions as documented here

anonymously access public container images

Meaning that you don't need an authenticated connection but note that this is specifically for public containers and not private ones.