Argo: [BUG] resource can not be passes as parameter

[dmonakhov]

I try to pass container resources (cpu in my case) as workflow parameter( see workflow below)
But argo cli validation failed like follows:

$ argo  submit resource-param.yaml 
2018/01/26 19:00:54 resource-param.yaml failed to parse: error unmarshaling JSON: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'

AFAIU this is happens because expression was not actually substituted.

WORKFLOW

# Try to pass cpu resource as an argument                                                                                                   
#                                                                                                                                           
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: resource-param-
spec:
  arguments:
    parameters:
    - name: nrcpu
      value: 4
  entrypoint: stress-ng
  templates:
  - name: stress-ng
    inputs:
      parameters:
      - name: nrcpu
    container:
      image: lorel/docker-stress-ng
      args: [ "--cpu", "{{workflow.parameters.nrcpu}}", "--timeout", "15s", "--metrics-brief" ]
      resources:
        requests:
          memory: 1Gi
          #cpu: 4                                                                                                                           
          cpu: "{{workflow.parameters.nrcpu}}"

[jessesuen]

K8s has custom unmarshal/marshal functions for the container.resources field. Need to find a way to workaround this.

[jessesuen]

This is going to be a tricky one to fix. Since we reuse the Container data structure from the k8s API types (among others), we inherit all of logic that comes with marshaling and unmarshaling of the various types. This includes the custom unmarshalling functions deeply nested any of their sub fields (in this case resource/quantity.go Quantity). We definitely do not want to move away from reusing Container data type since we need to keep in-line with k8s. Our only option it appears is to implement custom unmarshalling at the template level. I'm also very weary of doing that since that is always tricky to get right.

[dmonakhov]

@jessesuen Thank you for explanation.
May be there is workaround for this I'm not an exert in k8s API.
My scenario:
a CI job which want to compile. So I want complete it ASAP, for that reason I use
'make -j $NRCPU', where NRCPU is $(numcpu-1) on kube-nodes. It is obvious that hardcoding NRCPU is bad for portability because it NRCPU varies from cluster to cluster

Is it possible to define resource via generic kubernetes API? Via configMap or other config?

[dmonakhov]

Yeah. It seems that 'kind: LimitRange' resource does exactly what I need. https://kubernetes.io/docs/tasks/administer-cluster/cpu-constraint-namespace .
Since workaround exists this issue can be moved to enhancement section.

[jessesuen]

Thanks for the legwork on the workaround. Yes, container.resources obtained from parameters is a use case we should support, so eventually this is something I'd like fixed. The right way to go about doing it, escapes me at the moment.

[discordianfish]

Is there a workaround for this? This is almost a show-stopper for the usecase here. We have several tasks that have vastely different resource requirements, which forces us to create many templates.

[epa095]

A bit of brainstorming on this.
Would it maybe make sense to be able to define a "patch" to a workflow step, which could override parts of the yaml definition of the step. I am thinking something similar to overlays in kustomize.
This could take in exactly the same parameters as the step itself, and be applied to the workflow step right before execution. This would allow you to still use the k8s API types for the workflow step, but people would be "on their own" in regard to what they write in the patch/overlay step.

[igor47]

also got bitten by this issue. as @discordianfish alludes to, the only known workaround so far is to have multiple copies of the same template but with different resource requests. for example:

spec:
  entrypoint: test
  templates:
    - name: test
      dag:
        tasks:
          - name: task1
             template: pod-lowmem
          - name: task2
             template: pod-highmem
      - name: pod-lowmem
         container:
           image: myimage:latest
           resources:
             requests:
               memory: 500Mi
       - name: pod-highmem
          container:
            image: myimage:latest
            resources:
               requests:
                 memory: 2Gi

some way to DRY this manifest would be greatly appreciated. i guess it's possible to use YAML anchors but the syntax there is quite confusing

[elsonrodriguez]

As a workaround for this I ended up using a resource to create a Job. Messy but workable.

Food for thought: A killer use case for this would be in combination with GKE's node auto-provisioner, especially for compute-heavy jobs. This way memory/cpu/gpu could be defined as a parameter, and eventually it will just run without needing to define/provision nodes.

[Snapple49]

@elsonrodriguez Hi, would you mind explaining your workaround please? Not sure if I follow, but I'm interested in solutions to this problem as well

[elsonrodriguez]

@Snapple49 Basically I just use this pattern:

https://github.com/argoproj/argo/blob/baf37052976458401a6c0e44d06f30dc8d819680/examples/k8s-jobs.yaml

The manifest portion is fully template compatible, so you can swap out anything for a variable, including resource allocation.

For example:

                resources:
                  limits:
                    memory: "{{inputs.parameters.memory}}"
                    cpu: "{{inputs.parameters.cpu}}"

The downside is that logs won't show up directly associated with the workflow, it's less readable, and more complex.

[Snapple49]

I see, thanks a lot! That's neat, I'll have a play with that, messy or not it's a workaround :grin:

[Oskoss]

+1 Just running into this issue for us as well.

[jessesuen]

Would it maybe make sense to be able to define a "patch" to a workflow step, which could override parts of the yaml definition of the step. I am thinking something similar to overlays in kustomize. This could take in exactly the same parameters as the step itself, and be applied to the workflow step right before execution. This would allow you to still use the k8s API types for the workflow step, but people would be "on their own" in regard to what they write in the patch/overlay step.

So far, this is the the most promising proposal I've seen to address this problem. I think a first class patch spec, which supports parameter substitution, and then applied on top of the container before submission, might be able to handle this in a generic way (not just for resources, but for other non-string fields like bools and ints).

[jgbaum]

Hi @sarabala1979.

Is this in release 2.4.2? If so, can you please provide an example on how to pass the cpu or memory resource requirements?

Thanks!

-J

[sarabala1979]

https://github.com/argoproj/argo/blob/master/examples/pod-spec-patch-wf-tmpl.yaml
https://github.com/argoproj/argo/blob/master/examples/pod-spec-patch.yaml
https://github.com/argoproj/argo/blob/master/examples/pod-spec-yaml-patch.yaml

[jgbaum]

Perfect!

-J

Sent from mobile device
On Oct 25, 2019, 2:52 PM -0700, Saravanan Balasubramanian [e-mail], wrote:

https://github.com/argoproj/argo/blob/master/examples/pod-spec-patch-wf-tmpl.yaml
https://github.com/argoproj/argo/blob/master/examples/pod-spec-patch.yaml
https://github.com/argoproj/argo/blob/master/examples/pod-spec-yaml-patch.yaml
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.