Flux: All logging goes to Stderr

[olib963]

It seems that all logs from flux go to Stderr, which makes it difficult in stack driver to determine what logs are actually errors. I think this line is the cause. Happy to look into opening a PR for this, but wanted a bit of guidance on preferred solution/implementation. Thanks.

[stefanprodan]

All Kubernetes services/operators are logging to stderr. You need to write a Fluentd filter for Flux and determine if a log contains an error or not based on the err='' field.

GCP Ingress Controller: https://github.com/kubernetes/ingress-gce/blob/master/cmd/glbc/main.go#L61

[hiddeco]

I am unsure if a Fluentd filter is a solution for GKE but you should be able to do the same filtering with Advanced Filters.

[stefanprodan]

@hiddeco I prefer to run my own Fluentd on GKE to filter the garbage before it reaches Stackdriver (where you pay a lot for storage). This is how you can customize Fluentd on GKE https://cloud.google.com/solutions/customizing-stackdriver-logs-fluentd

[olib963]

Thank you both for the feedback, I was not aware this was a convention. I will try implementing the Fluentd solution.

[squaremo]

@olib963 Do you mind closing this issue, if you are happy with the advice here? You can still comment here, e.g., to report results for posterity.

[hpurmann]

Has someone written and published a fluentd filter/parser? @stefanprodan ?

[krackjack29]

Anybody still looking at the parser/filter? If yes, could you please share the filter. Stackdriver is completely filled with error logs even while there is no error

[krackjack29]

Thank you both for the feedback, I was not aware this was a convention. I will try implementing the Fluentd solution.

@olib963 do you have the solution