Skaffold: Provide mechanism to indicate language runtime type for an artifact

[briandealwis]

skaffold debug currently uses a set of heuristics to guess the language runtime used in a container image:

We guess the runtime technology for an image container by examining the referenced image container configuration, specifically looking at the environment variables and command-line. It turns out that most of the language runtime base images define a XXXX_VERSION environment variable or use a well-known command-name (java or node or nodemon).

These heuristics fail for Go and C++ programs. Although Go does have some runtime environment variables, they aren't always used. And these heuristics may result in false-positives for mixed-runtime images. We need a way to define the language runtime for a particular artifact.

Proposal

• Allow setting hints via an environment variable: SKAFFOLD_RUNTIME?
• Allow providing hints via a container image label/annotation: skaffold.dev/debug/runtime
• Allow providing hints via a k8s manifest label/annotation: skaffold.dev/debug/runtime
• Allow setting the language runtime type in skaffold.yaml:
  yaml artifacts: <ul> <li>image: gcr.io/foo/bar<br /> runtimeType: go<br />

[corneliusweig]

Sounds like a good idea. Some thoughts:

• An environment variable can only be set once, so for multi-artifact projects with different technologies per artifact, this would fail miserably. If the env variable would be extended to SKAFFOLD_RUNTIME_<IMAGENAME> this could work, but I think this is too complicated. In essence, I would not support that setting via env variable (and also via command line for the same reasons)
• The annotation is a good idea and I think Skaffold should add these labels. However, it goes against the usual Skaffold way of expecting those labels to be added in k8s manifests. Usually configuration either comes from skaffold.yaml or the command line.
• The last suggestion to put this in skaffold.yaml seems the cleanest approach to me. I'm not entirely sure about the config name though. As is, it does not make explicit that this only relevant for debugging and users may get confused. What about debugRuntime or debugType?

Maybe you should also involve the IDE people here. They may have preferences and ideas how this can best work together with CloudCode.

[briandealwis]

Environment variables are on a per-artifact basis (well, per-container really), so that's not really a concern. skaffold debug should allow any of these approaches so that the developer can do whatever is most convenient for them; the recommended practice though is the skaffold.yaml.

[etanshaul]

Environment variables are on a per-artifact basis (well, per-container really), so that's not really a concern

I think I also interpreted it like @corneliusweig, where the env var would be set on the Skaffold process. Whereas I think here you mean to set it in the k8s config.

[etanshaul]

My intuition is that adding this to skaffold.yaml makes most sense as well. This is a configuration option specifically intended for Skaffold, and as such, seems like it belongs in the skaffold configuration.

The easiest solution from an IDE perspective would be to have this as a command flag - but given that it needs to be on a per artifact basis, it's probably infeasible?

@iantalarico for comment as well

[briandealwis]

Ah sorry. I meant either Dockerfile ENV or k8s container env flags.

I think a command-line flag can be done, and debug needs to support something like that for #2184. I think specifying a path to the artifact should be sufficient:

skaffold debug --type /path/to/artifact=go --type /path/to/other/artifact=c++

[etanshaul]

ah right. that would work. Just to be explicit, even though it was probably clear - this would be the easiest option from the IDE perspective because it wouldn't involve modifying any configuration. Instead we could just pass this "runtime type" at command execution time.

Side question: if a runtime type isn't supplied, _and_ skaffold debug fails to do its inference, presumably entire skaffold session won't fail, right? Instead it will just not apply any debug transforms. Similarly it would be nice if this failure to debug was output as part of the enhanced debug event api.

[briandealwis]

I haven't looked into how to hook into the events yet, but I figured I'd just return a map of artifact → debug configuration (runtime-type + debug ports). And indeterminate artifacts would be a null mapping.

[corneliusweig]

Ah sorry. I meant either Dockerfile ENV or k8s container env flags.

That makes perfectly sense. Do you think it's worth adding a note to your original issue post?

I think the standard way to identify an artifact in Skaffold is to use the image name and not the path. That has the advantage of being stable wrt. refactorings (unless the image name is changed :/), so what do you think about:

skaffold debug --type gcr.io/skaffold/getting-started=go

[corneliusweig]

While looking at #2184 I just had another idea how the runtime type could be specified elegantly on the cmdline: the switch to enable debugging could also be used to specify the runtime

skaffold debug --debug-artifact gcr.io/skaffold/getting-started=go

This would save typing/pasting of the image name and is equally expressive (the separator may be up to debate).

One drawback: it will no longer be possible to specify the runtime without restricting the debugged artifacts.

[iantalarico]

Jumping in a little late but I agree that passing the information to skaffold would be the easiest from the IDE side.

[briandealwis]

You're right @corneliusweig the image tag is better since several artifacts can be mapped to the same location, as happens with Jib multi-module projects.

[etanshaul]

using the image tag should work fine from the IDE perspective.

[etanshaul]

I haven't looked into how to hook into the events yet, but I figured I'd just return a map of artifact → debug configuration (runtime-type + debug ports). And indeterminate artifacts would be a null mapping.

That works. Having it return a null mapping, but still have the artifact information, will be helpful for the client.

[balopat]

Just rephrasing so that I understand: We need to enrich the heuristics in runtimes/artifacts so that skaffold can detect what language runtime an artifact is using, so that skaffold debug can do the right "magic" of rewriting the manifests.

Another way I would phrase this: With this direction we prescribe a set of standard mechanisms to make a runtime image Skaffold Debuggable

Allow setting hints via an environment variable: SKAFFOLD_RUNTIME?

+1 on env vars. I think this is a natural fit with the rest of the heuristics and eventually can become a standard thing across Dockerhub images as well, if we wanted to make an effort - we could raise a PR for all of them to include this ;)

Allow providing hints via a container image label/annotation: skaffold.dev/debug/runtime

I'm okay with this, but not convinced. One benefit I see, is that the Dockerfile doesn't have to change for the image. But that also means that if we'd ever inject a skaffold utility inside the container, it wouldn't know about the runtime. What is your thinking around the benefits for this mechanism?

Allow providing hints via a k8s manifest label/annotation: skaffold.dev/debug/runtime

I wouldn't expose this as part of the manifests. It is redundant, and also kind-of-in-the-middle: We are talking about images to be connected to skaffold. It is redundant because with env vars you can simulate this even in a pod spec. It is in the middle: on the image end we can use env vars to define runtimes, on the skaffold end in the artifact config directly.

Allow setting the language runtime type in skaffold.yaml:

Big plus on this one. As I mentioned, this feature connects images to skaffold. Having the ability to define it/override it in skaffold.yaml makes sense to me.

I'm luke warm on the ability to pass it in as a flag - it would be useful for IDEs as then you could override things without touching the yaml - however the IDE would need to store it somewhere...so why not just create a debug profile that has the override?

[balopat]

A crazy idea: we could have a set of known image names -> runtimes mapping themselves listed explicitly somewhere as well as another source of heuristics.

[briandealwis]

This could be a good way to allow specifying an Alpine-based Delve image (GoogleContainerTools/container-debug-support#30).

[jkleckner]

It also seems to me that skaffold.yaml would be the right place to annotate the runtime explicitly.

[briandealwis]

I'm going to add an runtimes attribute to artifacts. Multiple runtimes allows expressing needs like Go-on-alpine (runtimes: ["go","alpine"]). It turns out I need this for Buildpacks support.

[eikemeier]

I'm going to add an runtimes attribute to artifacts. Multiple runtimes allows expressing needs like Go-on-alpine (runtimes: ["go","alpine"]). It turns out I need this for Buildpacks support.

v2beta2 is published, but is identical to v2beta1 :grin:. Will this be in v2beta3?

[nkubala]

@eikemeier 🤦

yes hopefully so. the PR is still in draft, but we don't have any other config changes in the works to my knowledge, so it should go out with the next config bump. however, that isn't tied to our next release necessarily. in any case, stay tuned and we'll get this shipped ASAP.

[tejal29]

@briandealwis going through quarterly scrub and just wanted to make sure if this is still a priority.