Pipelines: Unable to generate visualization: Unable to initialize visualization

[dagondagon666]

What happened:
Trying to use the Visualization Creator inside Pipeline > Experiment.
Type: TFDV
Source: gs://ml-pipeline-playground/tfx_taxi_simple/data/data.csv

Received:

Internal error: Post http://ml-pipeline-ml-pipeline-visualizationserver:8888: EOF
Unable to initialize visualization request.
github.com/kubeflow/pipelines/backend/src/common/util.Wrap
    backend/src/common/util/error.go:246
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*VisualizationServer).generateVisualizationFromRequest
    backend/src/apiserver/server/visualization_server.go:77
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*VisualizationServer).CreateVisualization
    backend/src/apiserver/server/visualization_server.go:26
github.com/kubeflow/pipelines/backend/api/go_client._VisualizationService_CreateVisualization_Handler.func1
    bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/visualization.pb.go:228
main.apiServerInterceptor
    backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/go_client._VisualizationService_CreateVisualization_Handler
    bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/visualization.pb.go:230
google.golang.org/grpc.(*Server).processUnaryRPC
    external/org_golang_google_grpc/server.go:966
google.golang.org/grpc.(*Server).handleStream
    external/org_golang_google_grpc/server.go:1245
google.golang.org/grpc.(*Server).serveStreams.func1.1
    external/org_golang_google_grpc/server.go:685
runtime.goexit
    GOROOT/src/runtime/asm_amd64.s:1333

What did you expect to happen:
Generate a Visualization of the given data just as in the documentation:
https://www.kubeflow.org/docs/pipelines/sdk/python-based-visualizations/

[ajchili]

Hey @keeplearnin. While developing the python-based-visualizations I ran into issues with older versions of TFDV and TFMA. Unfortunately, I did not have ample time to fully resolve the issues faced and I am unsure if they were resolved after my departure from Pipelines. From my understanding, the TFX data in the ml-pipeline-playground is not compatible with the version of TFDV and TFMA that is used to generate visualizations. If possible, can you add the logs from the ml-pipeline-visualization-service within your Kubernetes cluster?

[ajchili]

/assign @ajchili

[Felihong]

Hey @ajchili , I'm running into similar issue while trying to visualize the outputs from a component:

• Environment: Kubeflow 0.7 deployed in GKE
• Type: Table
• Source: gs://ml-pipeline-playground/tfx_taxi_simple/data/data.csv
  (here I also tried my own bucket, where the service account pipeline-runner has edit access)
• Received (exactly the same as above):

Internal error: Post http://ml-pipeline-ml-pipeline-visualizationserver:8888: EOF 
Unable to initialize visualization request.
github.com/kubeflow/pipelines/backend/src/common/util.Wrap
\tbackend/src/common/util/error.go:246\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.

I checked the logs of Pod ml-pipeline-ml-pipeline-visualizationserver which returns multiple

server.py:error:unrecognized arguments: {}

As the augment is optional, do I have to define something here?

Thank you!

[ajchili]

Hey @Felihong, thanks for the additional report! I will try to duplicate your cluster and see why the issue is occurring. While it has been a bit since I have worked on this, I do not believe that you have to provide arguments for the table visualization. I will confirm this and see where the issue is arising.

As an FYI I just started my final semester at school so I may run into delays but I will update this ticket as frequently as I can, thanks!

[edacostacambioupgrade]

@ajchili fyi, a fresh GCP kf cluster gets this error too.
(kind of fresh, at commit https://github.com/kubeflow/pipelines/commit/ee207f2b056ee8026efe988f740b3d0dc3dcaf1e)

[enriquedacostacambio]

I think the problem is in the way the arguments are sent to the visualization server.
i think https://github.com/kubeflow/pipelines/blob/ee207f2b056ee8026efe988f740b3d0dc3dcaf1e/backend/src/apiserver/server/visualization_server.go#L70 builds the body using each parameter as a form post field equivalent to this:

curl 'http://localhost:8888/' -H 'content-type: application/x-www-form-urlencoded' -H 'accept: */*' --data-binary 'arguments={}&source=gs://mybucket/data.csv&type=tfdv'

while server.py expects a single form field called "arguments" with the parameters expressed as "command line" arguments, equivalent to this:

curl 'http://localhost:8888/' -H 'content-type: application/x-www-form-urlencoded' -H 'accept: */*' --data-binary 'arguments=--arguments {} --source gs://mybucket/data.csv --type tfdv'

i created a container with he visualizer image (gcr.io/ml-pipeline/visualization-server:0.1.27) and i was able to reproduce for failure with the first curl command.

my command

root@3458739067ce:/src# curl 'http://localhost:8888/' -H 'content-type: application/x-www-form-urlencoded' -H 'accept: */*' --data-binary 'arguments={}&source=gs://mybucket/data.csv&type=tfdv'
curl: (52) Empty reply from server
````
server output:

root@3458739067ce:/src# python3 server.py

later...

usage: server.py [-h] [--type TYPE] [--source SOURCE] [--arguments ARGUMENTS]
server.py: error: unrecognized arguments: {}
root@3458739067ce:/src#
```
(the process exists due to a bad request, i think that's another issue)

[enriquedacostacambio]

@ajchili i just retested those commands with gcr.io/ml-pipeline/visualization-server:0.1.31 and gcr.io/ml-pipeline/visualization-server:0.1.40 and both work when using the standard "forms fields" format of the command instead of the "command line" format that worked for 0.1.27.

At least in my case the error was a result of using pipelines 0.1.31 with visualization-server 0.1.27.

@keeplearnin do you have a similar setup?

this is the pr that changed how the parameters are sent: https://github.com/kubeflow/pipelines/pull/1951

[enriquedacostacambio]

nailed down to the manifests:
https://github.com/kubeflow/manifests/blob/v0.7-branch/pipeline/pipeline-visualization-service/base/kustomization.yaml#L11 points to visualization-server 0.1.27
while
https://github.com/kubeflow/manifests/blob/v0.7-branch/pipeline/api-service/base/kustomization.yaml points to api-server 0.1.31

[ajchili]

Hey @enriquedacostacambio, thanks for taking a deep dive into this issue and discovering the cause! I do not know if I have the bandwidth at the moment to continue the work that you have started. If possible, can you submit a PR with a fix for the manifests file? If not, I can get around to it on the weekend.

[Felihong]

Hi @edacostacambioupgrade , I changed the visualization-server version to 0.1.31 and it works fine, thanks for debugging! 😎

[enriquedacostacambio]

Sounds good. I created https://github.com/kubeflow/manifests/pull/709 targeting v0.7-branch
Thanks! Have a great semester!

[dagondagon666]

@enriquedacostacambio Changing visualization-server version 0.1.31 works for me as well! Thanks a lot!

[ajchili]

@edacostacambioupgrade thanks! I LGTM'd it and assigned @IronPan who oversaw the project during development, he should be able to help get it approved and tested.

[ajchili]

Looks like this should be resolved by https://github.com/kubeflow/manifests/pull/802. I will test in my own cluster before closing this issue.

[Bobgy]

/close
Looks like the issue is fixed

[k8s-ci-robot]

@Bobgy: Closing this issue.

In response to this:

/close
Looks like the issue is fixed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.