Tfx: Add visualizations for custom pipelines in Kubeflow on GKE

[ianhellstrom]

The example pipeline in Kubeflow in GCP has artifacts (static HTML) that show the various outputs in visual form (e.g. with TFDV and TFMA). I have created my own pipeline by adapting the TFX kubeflow Python module file, but when running it in Kubeflow (on GCP) I see output paths that lead to the data, but no artifacts being generated.

I have been unable to find output viewers in the TFX repo. Since it's available in the example pipeline I'm assuming it's possible.

Is there special set-up that needs to be done in the code (e.g. an annotation or parameters) or is this something that's lacking in the Docker image that runs in Kubeflow?

Kubeflow's doc says it needs to be in a file /mlpipeline-ui-metadata.json in the root of the container that's run for the step. One of the example notebooks actually has different images for the steps, which might be needed to avoid the JSON from being overwritten.

A default Tensorboard viewer for the training would also be extremely helpful and appears to be supported by Kubeflow.

[neuromage]

Thanks for bring this up. It's actually been asked for by users several times, and it's on my todo list. I should have something to show for this some time over the next few weeks.

[rummens]

Happy to help with this because having the visualization for all TFX components, is a nice and required feature.

[neuromage]

We have an active on-going project in KFP for achieving this functionality. I've opened https://github.com/kubeflow/pipelines/issues/1472 to track this in the KFP repo. Closing this bug in favour of that one.