Pipelines: Unable to download directory artifact from the UX link

[sadeel]

What steps did you take:

I am using the off-the-shelf component to download a directory from GCS. It seems to run to completion, but when I attempt to download the artifact from minio, it fails.

here is the pipeline:

import_op = components.load_component_from_url(
    'https://raw.githubusercontent.com/kubeflow/pipelines/2dac60c/components/google-cloud/storage/download_dir/component.yaml'

def trainer(train_path: 'URI' = 'gs://my-fake-bucket/tf_records/*'):
  import_train_data_task = import_op(train_path)

What happened:

Error when trying to download

What did you expect to happen:

Successfully able to download files

Environment:

Hosted KFP

How did you deploy Kubeflow Pipelines (KFP)?

KFP version:

KFP SDK version:

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue.]

/kind bug

[Ark-kun]

This is likely an issue with the UX or ReadArtifact API, not the component. Let me check.
Also, try 'gs://my-fake-bucket/tf_records/' (no star in the end).

[rmgogogo]

is it still an issue now? github has outage recently.

https://www.githubstatus.com/

you may try download from another website for a quick check

[sadeel]

Hmmm, for this bug, I had no issues downloading the component. I was able successfully execute the component, but was unable to download the artifacts stored in minio.

[Ark-kun]

Can you please post a screenshot or an error log.

Also does the component succeed or fail? Can you then pass the data to a downstream component?

[Bobgy]

@sadeel Thanks for the report! Would be helpful if you provide the detailed error message/log of ml-pipeline-ui. This looks like your artifact was successfully downloaded, but UI couldn't handle visualization/showing directory artifact

[Bobgy]

Actually UI cannot really handle directory artifact as of now.

[Bobgy]

So, we could either:

1. support directory artifact (I don't find it likely, because the artifact should be tar-gzipped in artifact store, is that right?)
2. have a clear error message, showing directory artifact is not supported by KFP UI and suggest a workaround

[Ark-kun]

1. support directory artifact (I don't find it likely, because the artifact should be tar-gzipped in artifact store, is that right?)

JFYI: Currently some components output directory artifacts. Currently all artifacts are tar-gzipped in the artifact store.

In the future we might want to support"directory artifact preview" feature in the UX that can show the files inside the directory and their sizes (and maybe download links).

[Bobgy]

1. support directory artifact (I don't find it likely, because the artifact should be tar-gzipped in artifact store, is that right?)

JFYI: Currently some components output directory artifacts. Currently all artifacts are tar-gzipped in the artifact store.

In the future we might want to support"directory artifact preview" feature in the UX that can show the files inside the directory and their sizes (and maybe download links).

For that feature, we'd need to reimplement a folder browser UI. If there's some existing library we can reuse, then it's fine, but otherwise I don't think that's sustainable. That doesn't seem like a core feature of KFP. So I prefer finding a way to integrate with existing UI services that provide such browsing features.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

[Bobgy]

/reopen
/frozen

[k8s-ci-robot]

@Bobgy: Reopened this issue.

In response to this:

/reopen
/frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Ark-kun]

So, we could either:

1. support directory artifact (I don't find it likely, because the artifact should be tar-gzipped in artifact store, is that right?)
2. have a clear error message, showing directory artifact is not supported by KFP UI and suggest a workaround

Can we let the user download the actual .tar.gz archive? I think this solves an important problem and makes every artifact downloadable. I just had an issue where I could not download a Tensorflow SavedModel, because it's a directory format.

The algorithm can be as follows:
If the archive contains a single file, return that file. Otherwise return the whole archive.

[Bobgy]

Sounds like a good idea, I'll take a look if that's feasible.

[paveldournov]

Sounds like a good idea, I'll take a look if that's feasible.

We can make it even simpler - the link in the UI can always download. The user can see the single file or a directory locally, both can be handled the same way.

[Bobgy]

I'm working on this