Pipelines: Unable to list pipeline versions using the kubeflow client

[munagekar]

What steps did you take:

Unable to list pipeline versions using the kubeflow client.
Unable to reach endpoint : GET /apis/v1beta1/pipeline_versions

    client = kfp.Client(namespace=PIPELINE_NAMESPACE)
    client.pipelines.list_pipelines() # This works
    client.pipelines.list_pipeline_versions() # This does not work

What happened:

kfp_server_api.rest.ApiException: (503)
Reason: Service Unavailable
HTTP response body: Error: 'ssh: rejected: connect failed (Connection refused)'
Trying to reach: 'http://ip:8888/apis/v1beta1/pipeline_versions'

What did you expect to happen:

Endpoint was working

Environment:

How did you deploy Kubeflow Pipelines (KFP)?
Kubeflow Pipelines as part of a full Kubeflow deployment.

KFP version: Build commit: 743746b

KFP SDK version:
kfp 0.5.1
kfp-server-api 0.5.0

/kind bug

[Bobgy]

@munagekar KFP installation version was 743746b a.k.a 0.2.5, so it only guarantees features in kfp-server-api 0.2.5 or below.

[Bobgy]

/assign @jingzhang36

[munagekar]

@Bobgy Thank you for your prompt reply.This is the reference spec for 0.5.1. How can I find the reference spec for 0.2.5 ?

~I guess this endpoint is a not a part of Kubeflow 0.2.5 and this issue can be closed.~

[Bobgy]

@munagekar We didn't publish it, you may find open api definition in https://github.com/kubeflow/pipelines/blob/0.2.5/backend/api/swagger/pipeline.swagger.json.

You can also use swagger documentation site to visualize it https://petstore.swagger.io/, please paste https://raw.githubusercontent.com/kubeflow/pipelines/0.2.5/backend/api/swagger/pipeline.swagger.json into the UI's top url box.

[jingzhang36]

@munagekar when you are using 0.2.5 KFP api server with a 0.5.1 KFP sdk, there might have unexpected behavior. In other words, the behavior could be different from the expected behavior of either version.

[munagekar]

@jingzhang36 I can confirm this issue is present with KFP sdk client 0.2.5 as well.

pip list | grep kfp
kfp                           0.2.5
kfp-server-api                0.1.40

Same result.
I cannot test for the latest-api server since KFP was deployed as a part of Kubeflow.

[Gsantomaggio]

I think I figured out what's happening,

So the request.ResourceKey must be set

if you try the last kubeflow pipeline version you will get the error:

HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Tue, 11 Aug 2020 09:15:56 GMT', 'Trailer': 'Grpc-Trailer-Content-Type', 'Transfer-Encoding': 'chunked'})
HTTP response body: {"error":"Invalid input error: ResourceKey must be set in the input","message":"Invalid input error: ResourceKey must be set in the input","code":3,"details":[{"@type":"type.googleapis.com/api.Error","error_message":"ResourceKey must be set in the input","error_details":"Invalid input error: ResourceKey must be set in the input"}]}

ResourceKey must be set in the input error.

I think that there is an incongruence on the documentation since "name": "resource_key.id" seems not to be required

[jingzhang36]

@munagekar @munagekar, I don't think this is a bug on server side. Let's try a couple of things to fix your issue.

First, client.pipelines.list_pipeline_versions() needs a parameter, specifying the pipeline id. That is because list pipeline version is meant to list versions under a given pipeline. E.g. https://github.com/kubeflow/pipelines/blob/d7e589491280f1645d5e733165435d78a0bd7732/sdk/python/kfp/_client.py#L467

Second, it is very important to have matched versions for (1) kfp, (2) kfp-server-api, and (3) KFP API server. E..g., if you use the latest sdk, which supports pipeline versions, and connects to an old api backend, which does't support pipeline versions. It will definitely break. Therefore, it is import to keep the afore-mentioned three components with a matched version.

[Gsantomaggio]

@jingzhang36 you are right, the error I found was form and old pipeline version, fixed here

With the last version, I get the correct error message

kfp (1.0.0)
kfp-server-api (1.0.0)

With the correct resource key, it works:

r = client.pipelines.list_pipeline_versions(resource_key_id="c73bc6bf-6ca4-4980-915e-28d9d456ece2")
print(r)

result:

{'next_page_token': None,
 'total_size': 1,
 'versions': [{'code_source_url': None,
               'created_at': datetime.datetime(2020, 8, 6, 9, 10, 24, tzinfo=tzutc()),
               'id': 'c73bc6bf-6ca4-4980-915e-28d9d456ece2',
               'name': 'Demo',
               'package_url': None,
               'parameters': [{'name': 'project_id',
                               'value': 'kubeflow-ai-254012'}],
               'resource_references': [{'key': {'id': 'c73bc6bf-6ca4-4980-915e-28d9d456ece2',
                                                'type': 'PIPELINE'},
                                        'name': None,
                                        'relationship': 'OWNER'}]}]}

[munagekar]

@Gsantomaggio Thank you for verifying this. Leaving this open for this

I think that there is an incongruence on the documentation since "name": "resource_key.id" seems not to be required

[Gsantomaggio]

You are welcome.
yes, Additional to https://github.com/kubeflow/pipelines/issues/4063#issuecomment-671833660 we should change resource_key.id to "required": true :

   {
            "name": "resource_key.id",
            "description": "The ID of the resource that referred to.",
            "in": "query",
            "required": false,
            "type": "string"
          },

If it makes sense to @jingzhang36 I can prepare this tiny PR about the Swagger Documentation

[Bobgy]

swagger json is generated from proto where everything is optional for a reason.
So I don't think we will want to change that, if something is required, we use documentation on the field to mention that.

[Bobgy]

e.g. the description is generated from https://github.com/kubeflow/pipelines/blob/d87c3be6114af867760d07f1d70f4d95d4456c52/backend/api/resource_reference.proto#L40

[Gsantomaggio]

@Bobgy thank you for the clarification, I am still getting practise with the API code!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.