Super-linter: How to make the Super linter smaller and or more effective

[admiralAwkbar]

This issue is an open discussion on the options to make the super linter faster, smaller, and easier to use.

Current Issue

• Image is too large (4+gb)
• Takes too long to download

Solutions

Break into 3 images

• Super-linter
• Super-Linter-Windows
• Super-Linter-Linux

The base Super-Linter will stay the same, and house all linters. This is legacy and will prevent users from losing functionality.

The Super-Linter-Windows image will have the Windows-based languages and the basic underlying linters. This will still be a larger image as the Windows-based language installs are very heavy...

The Super-Linter-Linux will be the main image minus the windows based languages. This should help bring down its size.

This would require a little behind-the-scenes work to host 3 images on DockerHub and GitHub Container Registry.
It would also take a small number of logic updates to make sure testing worked seamlessly.
It would also take a little work to update the deployments.

Update the Dockerfile

Currently, we try to split all the language installs apart in the Dockerfile for readability and user understanding. This creates additional layers and adds size to the image. We could optimize the Dockerfile for size and not readability and likely pull down the size as well. Removing more layers= smaller image?

Multi-Stage build of Dockerfile

This would likely require a fair amount of work on the Dockerfile and the build process. We would need to go through the image layer by layer and pull in only the libs we truly need. This could greatly pull down the size of the image as well but will require deep Docker knowledge and extensive testing...

Conclusions

All have their Pros and Cons... But we need to have an open discussion on this to make sure we are all on board and take the proper direction(s).

[admiralAwkbar]

@nemchik @ferrarimarco @zkoppert @IAmHughes @jwiebalk @Hanse00 @GaboFDC

[ferrarimarco]

Yes, we really need to do something about this.

My gut feeling (I don't have numbers, yet), is that splitting by OS might be not enough. I'd take the chance to have a greater granularity.

I see a couple of approaches:

1. Tweaking our images.
2. Avoid building images altogether.

Tweaking our images

Here's how I would do this:

1. Assess the linters we're installing. We need to know (even qualitatively only) the runtime environment that they need. Example: how many linters need ruby? how many linters need node? how many linters need python? etc.
2. Build an "intermediate" image for each of those "runtime environment" classes/types. <- I suspect that there are official images from most of the cases we find in the previous step. We only have to build what we're missing.
3. Build and tag one "final" image per language, using the "intermediate" images as a base for those final images. This can be a multi-staged build.

One issue with this approach is that we miss the nice "fire and forget" feature that super-linter currently implement. You can point super-linter to a directory, and it can scan new file types without any intervention from the user. If you're fine with the default configuration, you don't have to do anything to scan new file types. For example, imagine that you have Python source files to scan. If you add Markdown files, you don't have to edit the default configuration to make super-linter scan them. To solve this, we might add a relatively small (optional) feature that makes super-linter warn/exit with an error/suggest which linters (and so which images) you might want to enable when it finds files that super-linter doesn't recognize.

Avoid building images altogether

This option is a bit less "conservative" than the previous one.

We might want to avoid building images altogether, and the shell (or whatever) to orchestrate the execution of linters that are already containerized. This is fine for linters that already have an "official" container image (likely the ones that we use in our FROM directives). For the non-containerized linters, we either provide a containerization, or we drop them from super-linter.

Corollary: being containerized might be one of the criteria to decide if a linter should be included in super-linter or not. I'd keep the usage of non containerized linters to a minimum, because it places a lot of management burden on us.

PS: this second option came to mind because of this question: Why don't we reuse already-built images?
For the most part, we don't add any customizations or fixes to the existing images. We just copy part of their contents in our image.

[nemchik]

More things to consider:

I agree that using linters from within docker containers makes it REALLY easy to accomplish what you want. So in my eyes, having the images available that contain the linters is the important thing. I think another potential strategy would be to leverage docker image tags, and do one for each linter. The idea here would be that super-linter (as a docker image) would become super-linter:markdownlint and super-linter:eslint and many more, for the sake of having all the linters in one place. the super-linter:latest image might be able to be some combination of all of the other tags (technically each of the other tags would be almost like a build stage) by using each of the other tags as a FROM. Some linters already have their own official images, which is great, because we can have a very simple dockerfile that just copies their existing image for the sake of getting the tool into a tagged image pushed as super-linter:tag. This would also mean a lot of the bash logic about determining which files to lint and what to do with the results could be separated out into other tags (maybe super-linter:git and super-linter:local) and those images could be smaller and could determine which of the other super-linter:tags are needed and use docker-in-docker to pull and run only the images needed based on what the user defines they want to lint, or we have an auto-detect to determine what languages even can be linted based on the files available. A cool thing about this approach is super-linter can remain an all-in-one tool while also providing the ability to just make use of individual linters in ways we aren't even considering (end users can be creative).

[ferrarimarco]

I'd also take the chance to streamline our tests. In the current implementation, we test more than we should, because we're also testing that linters are behaving correctly, which it's not our concern. We "just" have to test that linters are:

1. Available and executable.
2. The version we expect.
3. Picking up the expected configuration.

We don't care if their output matches with anything. In my opinion, that is a concern for the linter themselves. I can open a new issue to talk about this if you feel it's an entirely different conversation.

[admiralAwkbar]

@ferrarimarco I agree with you! We shoudl only validate they are installed and at the version, we are expecting. And basic functionality of the shell interpretation... i.e fail when we should fail, pass when we should pass... The tap is nice, but agree that should be handled by the linters maintainers themselves.

[admiralAwkbar]

@nemchik could we version the specific linters? super-linter:markdown:v1.0.0 not sure if that's a real capability...
That would be super helpful though...

[ferrarimarco]

You can do what most images do (even official ones): super-linter:markdown-v1.0.0

[ferrarimarco]

Because you cannot nest image tags AFAIK.

[nemchik]

Correct, we can't nest tags. We could maintain a separate image for each linter, but that feels excessive. tags feel more like they would make sense for this purpose, and using a -version would make sense. We would probably want to have super-linter:markdown-v1.0.0 and super-linter:markdown where the later is the most recent version (which would also have a named tag). Essentially when pushing a new version of a linter we push it to its named version and to its name at the same time. This would keep it simple to find linters and also to use specific versions.

[nvuillam]

Create Super-Linter flavors :)

• not by OS but by "project style", + the main image embedding everything for the cases not covered

https://nvuillam.github.io/mega-linter/flavors/

Another idea could be to use "squashed" images... not tested yet but I think having N layers in docker images is not useful in such case

By the way, i'd be more than glad to welcome you on the Mega-Linter project, to avoid to rebuild in bash what is already done there 🤟

[ferrarimarco]

Create Super-Linter flavors :)

• not by OS but by "project style", + the main image embedding everything for the cases not covered

https://nvuillam.github.io/mega-linter/flavors/

Cool visualization :)
The sizes of your images validate my thinking IMHO. Those images are still way too big, because linters for a given language/flavor/whatever might have completely different dependencies (example: dockerfilelint and hadolint).

Another idea could be to use "squashed" images... not tested yet but I think having N layers in docker images is not useful in such case

Squashed images are not ideal for this use case IMHO, because they make caching intermediate layers and sharing layers between different images impossible.

[nvuillam]

Cool visualization :)

@ferrarimarco thanks :) If you didn't push me about making YML descriptors instead of the classes I initially did, it would have been harder :D

The sizes of your images validate my thinking IMHO. Those images are still way too big, because linters for a given language/flavor/whatever might have completely different dependencies (example: dockerfilelint and hadolint).

The images still have a certain size... but it's because in all of them I put everything related to documentation and CI (meaning Java for the case there may be just a Jenkinsfile... )

All Dockerfiles and images are automatically generated (sources + images via CI), so theoretically I could have 100 combinations, but i'd be afraid to reach the max limits of GH Actions minutes, or docker images storage (if such limits exists)

I think i'll just add requested combinations, but for the moment users seem satisfied with the existing ones

Squashed images are not ideal for this use case IMHO, because they make caching intermediate layers and sharing layers between different images impossible.

I agree with all that, but... if we think only about user performances, when there is no caching (like it seems with public Github Actions CI), images would be faster to load ^^

And even with caching... there are so many layers related to so many linters, who can be upgraded anytime... so not sure there is a great benefit for layer caching :/