Pipelines: Cannot use resource request based on PipelineParam

[gabrielvcbessa]

We are trying to build a generic pipeline that will have some steps with configurable resources. This is needed because based on what is being run it might need more or less cpu, memory and etc.

Consider the pipeline described bellow:

@dsl.pipeline(
    name='Generic Pipe'
)
def pipeline_definition(memory_request: str = '8G', cpu_request: str = '4'):
    op = ContainerOp(...)

    op.set_memory_request(memory_request)\
      .set_cpu_request(cpu_request)

This pipeline will fail to match the expected resource Regex and would not be successfully uploaded. This is what would be generated on the yaml file:

 resources:
    requests:
        cpu: '{{inputs.parameters.cpu-request}}'
        memory: '{{inputs.parameters.memory-request}}'

And here is the error:

kfp_server_api.rest.ApiException: (500)
Reason: Internal Server Error
HTTP response headers: HTTPHeaderDict({'Server': 'nginx/1.15.6', 'Date': 'Mon, 26 Aug 2019 21:46:03 GMT', 'Content-Type': 'text/plain; charset=utf-8', 'Content-Length': '601', 'Connection': 'keep-alive', 'x-powered-by': 'Express', 'x-envoy-upstream-service-time': '55'})
HTTP response body: {"error_message":"Error creating pipeline: Create pipeline failed: Failed to get parameters from the workflow: InvalidInputError: Failed to parse the parameter.: error unmarshaling JSON: while decoding JSON: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'","error_details":"Error creating pipeline: Create pipeline failed: Failed to get parameters from the workflow: InvalidInputError: Failed to parse the parameter.: error unmarshaling JSON: while decoding JSON: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'"}

[gabrielvcbessa]

@IronPan Do you have any idea of how to solve this?

[elikatsis]

Hello,

This issue is related to argoproj/argo#703 and it is not fixed yet.

[kierenj]

It looks like https://github.com/argoproj/argo/pull/1687 might fix this. I'm very new to kubeflow - can anyone explain the workaround in https://github.com/argoproj/argo/issues/703 - is it available to kubeflow-land?

[rmgogogo]

@Ark-kun, any suggested workaround?

[Ark-kun]

This is a problem with trying to use templates in structures with strict schemas. @numerology had the same issue when trying to template protobufs.

The addition pf PatchPodSpec https://github.com/argoproj/argo/pull/1687 in Argo v2.4.2 makes it possible to fix this, although this will require some amount of work on the SDK side, upgrade to Argo v2.4.2. It will also be backwards incompatible with existing KFP backends (since older versions of Argo do not support the feature).

[jackwhelpton]

I'm getting blocked by this too. We're also trying to configure GPU dynamically, but that's running into problems because add_node_selector_constraint can't take PipelineParams: I was hoping I could use a previous parse step to build the node selector and then apply it using parse_task.outputs.

Would that be covered by this ticket, or need a new one, or is there a workaround I'm missing?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

[jackwhelpton]

/reopen

This would still be a useful feature to have.

[k8s-ci-robot]

@jackwhelpton: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

This would still be a useful feature to have.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Bobgy]

/reopen
for @jackwhelpton

[k8s-ci-robot]

@Bobgy: Reopened this issue.

In response to this:

/reopen
for @jackwhelpton

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mnoble-vody]

Argo was upgraded to 2.7.5 in https://github.com/kubeflow/pipelines/pull/3537 so now Argo's pod spec patching (https://github.com/argoproj/argo/pull/1687) is available to implement a fix in the Pipelines SDK.

[Bobgy]

/assign @chensun
FYI, this is the top open issue with the most thumb ups

[numerology]

@chensun

I don't think this will be super complicated to fix. I guess it's currently blocked by this allowlist here. We can take a look when we have some cycle.