Website: Pipeline Metrics file_output not explained

Created on 26 Jun 2020 · 7Comments · Source: kubeflow/website

https://www.kubeflow.org/docs/pipelines/sdk/pipelines-metrics/

I just passed a whole day to understand why the metrics were not shown in the UI after my pipeline.
In fast (using ContainerOps) you need to define a file_outputs. That is not said in the documentation and the information is quite hard to find.

You should declare your step as follow:

operations['your_step'] = dsl.ContainerOp(
        name = 'step_2',
        image = '<IMAGE_NAME_HERE>:<IMAGE_TAG_HERE>,
        command = ['python'],
        arguments = [
            'your_step.py'
        ],
        file_outputs = {
            'mlpipeline-metrics': '/mlpipeline-metrics.json'
        }
    )

I really think it should be explained in the doc somewhere on the metrics page or at least have a link to the explanation on this page. And that's the same for the output viewer, the process is the same and the explanation is not here too.

arebackend aredocs arepipelines kinfeature prioritp1

Source

GraphtyLove

👍2

Most helpful comment

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/pipelines | 0.93 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

issue-label-bot[bot] on 26 Jun 2020

👍2

All 7 comments

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/pipelines | 0.93 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

issue-label-bot[bot] on 26 Jun 2020

👍2

This issue is relevant outside of the scope of ContainerOps.
In the .yaml files defining components, it looks like we need to specify dedicated outputs and fileOutputs, as shown below:
````yaml
outputs:

{ name: MLPipeline Metrics, type: Metrics }
# Other outputs
implementation:
container: ...
image: ...
command: ...
args: ...
fileOutputs:
MLPipeline Metrics: /mlpipeline-metrics.json
````
I got this from the confusion matrix example.

In this regards, the text about fileOutputs in the kubeflow component specification section is a bit misleading, at it hints that the user is unlikely to have to rely on this field when defining components.

matphilippe on 7 Aug 2020

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/docs | 0.64 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

issue-label-bot[bot] on 7 Aug 2020

👍1

In the .yaml files defining components, it looks like we need to specify dedicated outputs and fileOutputs, as shown below:
In this regards, the text about fileOutputs in the kubeflow component specification section is a bit misleading, at it hints that the user is unlikely to have to rely on this field when defining components.

There is no contradiction. The confusion matrix is a really old component and the effort to fix it started back in 2018. https://github.com/kubeflow/pipelines/pull/580/files#diff-a65b15d94e67f77a796e4940651e5c62

Here is how the component.yaml should look like:

  outputs:
  - { name: MLPipeline Metrics, type: Metrics }
  # Other outputs
  implementation:
   container: ...
    image: ...
    command: ...
    args:
    - ...
    - --metrics-path
    - {outputPath: MLPipeline Metrics}

Ark-kun on 1 Sep 2020

/cc @Bobgy @rmgogogo

I'm having some issues with metrics reporting. The api-server shows errors in the logs:

"ResourceNotFoundError: artifact runs/8550f4eb-b670-4872-8777-c958b3f85c88/nodes/produce-metrics-2-f6c8w/artifacts/mlpipeline-metrics not found."

But the artifact exists and can be downloaded from the UX

    workflows.argoproj.io/outputs: >-
      {"artifacts":[{"name":"mlpipeline-metrics","path":"/tmp/outputs/mlpipeline_metrics/data","s3":{"endpoint":"minio-service.default:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/produce-metrics-2-f6c8w/produce-metrics-2-f6c8w-1130634955/mlpipeline-metrics.tgz"},"optional":true},{"name":"main-logs","archiveLogs":true,"s3":{"endpoint":"minio-service.default:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/produce-metrics-2-f6c8w/produce-metrics-2-f6c8w-1130634955/main.log"}}]}

2020-08-31 23:48:30.057 PDT
"/api.RunService/ReadArtifact handler starting"
Info
2020-08-31 23:48:30.061 PDT
"ResourceNotFoundError: artifact runs/8550f4eb-b670-4872-8777-c958b3f85c88/nodes/produce-metrics-2-f6c8w/artifacts/mlpipeline-metrics not found."
Error
2020-08-31 23:48:30.061 PDT
github.com/kubeflow/pipelines/backend/src/common/util.NewResourceNotFoundError
Error
2020-08-31 23:48:30.061 PDT
backend/src/common/util/error.go:150
Error
2020-08-31 23:48:30.061 PDT
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).ReadArtifact
Error
2020-08-31 23:48:30.061 PDT
backend/src/apiserver/resource/resource_manager.go:953
Error
2020-08-31 23:48:30.061 PDT
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).ReadArtifact
Error
2020-08-31 23:48:30.061 PDT
backend/src/apiserver/server/run_server.go:167
Error
2020-08-31 23:48:30.061 PDT
github.com/kubeflow/pipelines/backend/api/go_client._RunService_ReadArtifact_Handler.func1
Error
2020-08-31 23:48:30.061 PDT
bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/run.pb.go:1403
Error
2020-08-31 23:48:30.061 PDT
main.apiServerInterceptor
Error
2020-08-31 23:48:30.061 PDT
backend/src/apiserver/interceptor.go:30
Error
2020-08-31 23:48:30.061 PDT
github.com/kubeflow/pipelines/backend/api/go_client._RunService_ReadArtifact_Handler
Error
2020-08-31 23:48:30.061 PDT
bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/run.pb.go:1405
Error
2020-08-31 23:48:30.061 PDT
google.golang.org/grpc.(*Server).processUnaryRPC
Error
2020-08-31 23:48:30.061 PDT
external/org_golang_google_grpc/server.go:966
Error
2020-08-31 23:48:30.061 PDT
google.golang.org/grpc.(*Server).handleStream
Error
2020-08-31 23:48:30.061 PDT
external/org_golang_google_grpc/server.go:1245
Error
2020-08-31 23:48:30.061 PDT
google.golang.org/grpc.(*Server).serveStreams.func1.1
Error
2020-08-31 23:48:30.061 PDT
external/org_golang_google_grpc/server.go:685
Error
2020-08-31 23:48:30.061 PDT
runtime.goexit
Error
2020-08-31 23:48:30.061 PDT
GOROOT/src/runtime/asm_amd64.s:1333
Error
2020-08-31 23:48:30.061 PDT
failed to read artifact 'run_id:"8550f4eb-b670-4872-8777-c958b3f85c88" node_id:"produce-metrics-2-f6c8w" artifact_name:"mlpipeline-metrics" '.
Error
2020-08-31 23:48:30.061 PDT
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrapf
Error
2020-08-31 23:48:30.061 PDT
backend/src/common/util/error.go:206
Error
2020-08-31 23:48:30.061 PDT
github.com/kubeflow/pipelines/backend/src/common/util.Wrapf
Error
2020-08-31 23:48:30.061 PDT
backend/src/common/util/error.go:231
Error
2020-08-31 23:48:30.061 PDT
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).ReadArtifact
Error
2020-08-31 23:48:30.061 PDT
backend/src/apiserver/server/run_server.go:170
Error
2020-08-31 23:48:30.061 PDT
github.com/kubeflow/pipelines/backend/api/go_client._RunService_ReadArtifact_Handler.func1
Error
2020-08-31 23:48:30.061 PDT
bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/run.pb.go:1403
Error
2020-08-31 23:48:30.061 PDT
main.apiServerInterceptor
Error
2020-08-31 23:48:30.061 PDT
backend/src/apiserver/interceptor.go:30
Error
2020-08-31 23:48:30.061 PDT
github.com/kubeflow/pipelines/backend/api/go_client._RunService_ReadArtifact_Handler
Error
2020-08-31 23:48:30.061 PDT
bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/run.pb.go:1405
Error
2020-08-31 23:48:30.061 PDT
google.golang.org/grpc.(*Server).processUnaryRPC
Error
2020-08-31 23:48:30.061 PDT
external/org_golang_google_grpc/server.go:966
Error
2020-08-31 23:48:30.061 PDT
google.golang.org/grpc.(*Server).handleStream
Error
2020-08-31 23:48:30.061 PDT
external/org_golang_google_grpc/server.go:1245
Error
2020-08-31 23:48:30.061 PDT
google.golang.org/grpc.(*Server).serveStreams.func1.1
Error
2020-08-31 23:48:30.061 PDT
external/org_golang_google_grpc/server.go:685
Error
2020-08-31 23:48:30.061 PDT
runtime.goexit
Error
2020-08-31 23:48:30.061 PDT
GOROOT/src/runtime/asm_amd64.s:1333
Error
2020-08-31 23:48:30.061 PDT
/api.RunService/ReadArtifact call failed
Error
2020-08-31 23:48:30.061 PDT
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrapf
Error
2020-08-31 23:48:30.061 PDT
backend/src/common/util/error.go:206
Error
2020-08-31 23:48:30.061 PDT
github.com/kubeflow/pipelines/backend/src/common/util.Wrapf
Error
2020-08-31 23:48:30.061 PDT
backend/src/common/util/error.go:231
Error
2020-08-31 23:48:30.061 PDT
main.apiServerInterceptor
Error
2020-08-31 23:48:30.061 PDT
backend/src/apiserver/interceptor.go:32
Error
2020-08-31 23:48:30.061 PDT
github.com/kubeflow/pipelines/backend/api/go_client._RunService_ReadArtifact_Handler
Error
2020-08-31 23:48:30.061 PDT
bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/run.pb.go:1405
Error
2020-08-31 23:48:30.061 PDT
google.golang.org/grpc.(*Server).processUnaryRPC
Error
2020-08-31 23:48:30.061 PDT
external/org_golang_google_grpc/server.go:966
Error
2020-08-31 23:48:30.061 PDT
google.golang.org/grpc.(*Server).handleStream
Error
2020-08-31 23:48:30.061 PDT
external/org_golang_google_grpc/server.go:1245
Error
2020-08-31 23:48:30.061 PDT
google.golang.org/grpc.(*Server).serveStreams.func1.1

Ark-kun on 1 Sep 2020

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/backend | 0.62 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

issue-label-bot[bot] on 1 Sep 2020

@Ark-kun sorry I didn't see your comment. Can you give me an example to verify?