Pipeline: Feature Request: Using ConfigMap Values For Params

[tomfrenken]

Expected Behavior

When defining parameters for a Task in my Pipeline, I would like to use values for the parameters which I defined in a ConfigMap, instead of hardcoding them in the Pipeline.

A cool solution would look like the use of ConfigMaps when defining environment variables in a Pod, there I am able to do this:

        env:
        - name: SOME_ENV_VARIABLE
          valueFrom:
            configMapKeyRef:
              name: my-config
              key: my-key

if params could work the same way it would be like this:

          params:
          - name: some-parameter
            valueFrom:
              configMapKeyRef:
                name: pipeline-config
                key: some-key

Actual Behavior

If I want to use a ConfigMap I can only use it on a Step level, and I can't directly access its values, this becomes an issue in the following Task:

apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: pmd-test
spec:
  inputs:
    resources:
      - name: git-source-code
        type: git
    params:
      - name: image
        type: string
        description: The image that will be used in the test
        default: rawdee/pmd
      - name: command
        type: string
        description: The command that will be used in the test
        default: pmd
      - name: args
        type: string
        description: The args that will be used in the test
        default: -language java -dir /workspace/git-source-code -rulesets ./my-rulesets.xml
  steps:
    - name: test-pmd
      image: $(input.params.image)
      command: ["$(input.params.command)"]
      args: ['$(input.params.args)']

in this example, I want to have the possibility configure the entire Step based on the ConfigMap, but currently it would look like this:

apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: pmd-test
spec:
  volumes:
    - name: config-volume
      configMap:
        name: pipeline-config
  inputs:
    resources:
      - name: git-source-code
        type: git
  steps:
    - name: test-pmd
      volumeMounts:
        - name: config-volume
          mountPath: /my-path
      image: $(cat /my-path/my-image)
      command: ["$(cat /my-path/my-command)"]
      args: ['$(cat /my-path/my-args)]

Note that this wouldn't work for the image since I'm only able to use variable substitution there.

[vdemeester]

/kind feature

[ImJasonH]

+1, we should make this change. This makes configuring the controller more standard, and could let users update the support images to use (git-init, creds-init, etc.) without requiring a restart of the controller pod.

[ImJasonH]

Thinking through this a bit more:

Luckily at least some of the groundwork is already done: support images are parsed from flags and set as fields on the pipeline.Images type.

These fields should become methods that return the current value, and some background process could monitor the ConfigMap and update the values returned by that method.

As a first step we can update one image to use the ConfigMap style, then prove it works, then update the rest. It probably wouldn't be a very hard change at all.

[ImJasonH]

Knative configures their queue proxy image using a ConfigMap: https://github.com/knative/serving/blob/b3079512daae9708f11c7b8ea41ef54023c91725/config/config-deployment.yaml#L26

Their pkg/deployment/config.go returns a *Config with a QueueSidecarImage field that gets updated when the ConfigMap changes.

Their "deployment config" object is effectively the same as our "pipeline.Images" object.

[skaegi]

sigh -- and after I just closed #1197 . The case can be made for this feature however I can say that I really thought I wanted this feature but in practice found I never needed it.

[ImJasonH]

I'm not sure I understand how these are related?

That issue seems to be about letting users declare TaskRun/PipelineRun values as coming from a ConfigMap (which I admit does sound useful?).

This issue is about letting operators specify/update controller args using ConfigMaps, which is more of an operator concern and not really exposed to end users.

Am I misunderstanding?

[skaegi]

Hmm... maybe I'm the one misunderstanding. @tomfrenken Is this issue for something like an install time kustomization of a Task and Pipeline. e.g. not something at runtime? Can you maybe describe a bit more what you're looking for.

[tomfrenken]

@skaegi I think @ImJasonH sums it up very well, that's precisely what I was meaning
But to answer your question a little better, yes in my case I thought about customization at install time.

[ankitm123]

if no one's working on it, I can take a look @ImJasonH

[mgreau]

I’m really interested in that feature. I don’t know if it’s out of scope here but I think it could be great to be able to get the default value from the ConfigMap. It would mean having a way to deploy Tasks and Pipelines which could be invoked by operators via ConfigMaps and also by user via the CLI.

[sbwsg]

Hey folks, I'm coming late to this issue but I was wondering if someone could clearly articulate use-cases and scope of this change. The description sounds to me exactly like @skaegi 's valueFrom issue and customization at install time isn't filling in the gaps for me. For example, from the issue description the following is stated as being desirable:

          params:
          - name: some-parameter
            valueFrom:
              configMapKeyRef:
                name: pipeline-config
                key: some-key

But then we seem to says this is not what's being asked for:

That issue seems to be about letting users declare TaskRun/PipelineRun values as coming from a ConfigMap (which I admit does sound useful?).
This issue is about letting operators specify/update controller args using ConfigMaps, which is more of an operator concern and not really exposed to end users.

And now I'm really confused. I see no mention of the controller or its arguments anywhere in the initial issue description. This issue really reads to me as precisely valueFrom as described in @skaegi's linked issue. So I'm clearly missing something, either from a gap in my own understanding or a gap in the way this issue is articulated.

So before we move ahead could somebody write out one or two clearly defined user-facing use cases and a description of what's in-scope / out-of-scope? It would be excellent to see what the intended final syntax looks like as well in the context of a Task / TaskRun / Pipeline / PipelineRun. I don't mind if it's a short design doc or just a comment here with some code blocks.

[sbwsg]

Also: the title of this issue is literally "Feature Request: Using ConfigMap Values For Params". I am so confused.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.