Katib: Metrics-logger-and-collector breaks GKE logs, makes them unsearchable in stackdriver

[kylepad]

/kind bug

This might both a bug and a feature requests.

TL;DR:

• normally in GKE each node runs a FluentD sidecar which reads the STDOUT and STDERR of each container and emits these to Stackdriver
• it appears the metrics-logger-and-collector is somehow intercepting these logs and consuming them before FluentD can read them
• FluentD then reads the logs being emitted from the metrics-logger-and-collector which have been formatted to be useless to us (details below)

We would like a way to avoid this behaviour so we can get the original Trial container logs into StackDriver

What steps did you take and what happened:
In order to make our logs searchable/filterable in Stackdriver (GCP's log viewer) we emit them as JSON from within the training script. Example log: {"filename": "run.py", "lineno": 119, "message": "parsing hyperparameters", "name": "train.run", "severity": "DEBUG", "trial_id": "hyperparam-tuning-899dw-train-c6zvn2s7", "job_id": "20200831T2206_2762ec"}

The metric collector also emits JSON, but our searchable JSON payload has been converted to a single string in the "message" field:
{ message: "{"filename": "run.py", "lineno": 119, "message": "parsing hyperparameters", "name": "train.run", "severity": "DEBUG", "trial_id": "hyperparam-tuning-899dw-train-c6zvn2s7", "job_id": "20200831T2206_2762ec"}" pid: "84", source: "main.go:81", severity: "INFO" }
Additionally the severity level has been effectively ignored and set to INFO.

Some example problems this creates:

• An exception occurs in a training script and we want to log it as an ERROR, and attach the stack trace and a bunch of metadata to make it filterable in stackdriver => the metrics collector removes our ability to filter by ERROR or by any of the metadata we've attached
• An exception occurs due to a missing dependency before any code has executed (IE: before our logging setup function runs). GKE marks any log printed to STDERR without a severity level as an ERROR, and any log printed to STDOUT as INFO. Again this error is effectively hidden by as the metrics collector will set the level to INFO and print it to STDERR.

What can be done to resolve this?

Ideally we would like the metrics collector to read the Trial container logs rather than consume and regurgitate them, allowing FluentD to pick up the logs and send them to stack driver as normal.

Alternatively a flag that has the metrics collector pass through logs without modifying them would also be great (and also print them to STDOUT or STDERR depending which stream they're coming in on).

Or should we roll our own metrics collector?

Environment:

• Katib: v1beta1
• Kubernetes: Google Kubernetes Engine
• Other: Stackdriver logging

Thanks!

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/katib | 0.82 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[issue-label-bot[bot]]

Issue Label Bot is not confident enough to auto-label this issue.
See dashboard for more details.

[andreyvelich]

Thanks for submitting this @kylepad.

Have you tried to use file-metrics-collector ?
This MC does not parse metrics from your StdOut and it does not redirect training container output to a specific file.

Can you read logs from the specific container in StackDriver? If yes, can you ignore metrics collector container logs and just take training container logs ?

[kylepad]

Have you tried to use file-metrics-collector ?
This MC does not parse metrics from your StdOut and it does not redirect training container output to a specific file.

I have now, and it solved my problem! Thanks @andreyvelich !