Cloudsql-proxy: Feature Request: Add flag for structured logging

[vankofalcon]

Hey guys. Is there a way to output the logs from the proxy in a JSON format?

[kurtisvg]

Hey @vankofalcon - The proxy currently doesn't support this. If you can explain why this would be useful or how it could be implemented, we could consider a feature request for it.

[vankofalcon]

Hi @kurtisvg Our logging system is indexing only JSON formatted entries. All the logs go into elastic search where the fields are properly indexed. For example ES has a definition schema https://www.elastic.co/guide/en/ecs/current/ecs-reference.html that you can use. But if we ignore ES for a moment, I just need is a common json format (configurable if possible) which contains a log level (INFO, ERROR, DEBUG, TRACE ...), message, stack_trace and a timestamp of the log entry.

[wyaeld]

Probably a meta question, but does google have a standard format for json logging that is targeted at GCP and Stackdriver, since that would be very useful to have the proxy emit.

[thejsj]

Want to just second that this would be a useful feature. Log ingestions for services like Stackdriver and Datadog just work better if they take in JSON.

[Carrotman42]

If you are willing to compile the proxy yourself, it should be pretty straightforward to overwrite the definitions of the logging functions and write out JSON instead of newline-separated messages: https://github.com/GoogleCloudPlatform/cloudsql-proxy/blob/master/logging/logging.go

[Carrotman42]

(that is: supporting a new flag, log destination, configuration, etc is not a decision that the Proxy project can make lightly; in the meantime, if you need this feature, the source code is there for you to use as you wish.)

[thejsj]

Sounds like a very nice way of saying "No", but appreciate the response. Just wanted to leave a +1 for this since @kurtisvg was asking why this would be useful.

[Carrotman42]

To be clear: I am no longer on the Cloud SQL team, I just hang out on the GitHub :) None of my comments are official (or unofficial) statements of what is going to go on, apologies if I implied otherwise.

My previous comment was based on my experience maintaining open-source software.

[Camoflame]

I tried using
gcr.io/cloudsql-docker/gce-proxy:1.16 image
and _-log_debug_stdout=true_ flag

But still there is those _Client closed local connection on..._ at Google Cloud logs

So how could I suppress the sql proxy logs?

[Carrotman42]

As the name implies -log_debug_stdout prints debug-level messages to
stdout, it doesn't suppress them entirely. I am guessing whatever setup you
have still pipes those logs to the "Google Cloud logs".

If you want to get rid of them entirely, try setting -verbose=false. Pass
--help to see all flags, in case you're curious about other options, or
look in the main file
https://github.com/GoogleCloudPlatform/cloudsql-proxy/blob/master/cmd/cloud_sql_proxy/cloud_sql_proxy.go
for
a more organized list.

On Tue, Mar 10, 2020 at 7:54 AM Mika Tuhkanen [e-mail]
wrote:

I tried using
gcr.io/cloudsql-docker/gce-proxy:1.16 image
and -log_debug_stdout=true flag

But still there is those Client closed local connection on... at Google
Cloud logs

So how could I suppress the sql proxy logs?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/cloudsql-proxy/issues/302?email_source=notifications&email_token=AAELF35MKAJIXYZCBFVF67TRGZBB3A5CNFSM4IPOSK2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOLP6MA#issuecomment-597098288,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAELF37WVUOGNEVKPMUWSHTRGZBB3ANCNFSM4IPOSK2A
.

[Camoflame]

Thanks @Carrotman42 the verbose=false works
So now the Cloud SQL proxy part in the Kubernetes deployment looks like this

- command:
  - /cloud_sql_proxy
  - -instances=my-project:europe-west1:my-database=tcp:3306
  - -credential_file=/path/to/my-secrets.json
  - -verbose=false
  image: gcr.io/cloudsql-docker/gce-proxy:1.16

[adamf]

+1 to this request. Any sort of structure to these logs would be useful - configurable date format, log levels, request durations, etc. It'd be a huge help in debugging.