Google-cloud-go: stackdriver: json logging in cloud functions

[Sheshagiri]

not sure if the issue belongs here or in https://github.com/GoogleCloudPlatform/fluent-plugin-google-cloud.

I'm dumping json logs from a google cloud function written in Go and the log is being parsed as textPayload. I'm using logrus here and when I dump the logs in similar fashion on app engine standard environment the logs are correctly parsed as jsonPayload. The issue only appears with cloud functions. Stackdriver documentation suggests me to use https://cloud.google.com/logging/docs/reference/libraries but since logrus seem to be working for appengine I was thinking if there is a way for to get the logs parsed as json in cloud functions as well.

quick google search pointed me to adding a stackdriver hook to logrus using https://github.com/knq/sdhook but looks like this hook is also sending logs as TextPayload https://github.com/knq/sdhook/blob/0b7fa827d09a4499ed19ce724144a0c84dfcef00/sdhook.go#L271. This seem to be a separate issue though, just posting my research here.

log snapshot from cloud functions

log snapshot from app engine standard environment

Describe Your Environment

Google Cloud Functions

Expected Behavior

json logs should be parsed as jsonPayload

Actual Behavior

json logs are being parsed as textPayload

[codyoss]

Hey I am not super familiar with logrus, but if you would like the logs to show up json style by using the client libraries you cloud do something like this.

    client, _ := logging.NewClient(ctx, "my-workspace")
    lg := client.Logger("mylog")
    lg.Log(logging.Entry{Payload: json.RawMessage([]byte("{\"foo\":\"bar\"}"))})

They key here is wrapping the payload in json.RawMessage([]byte). This will tell the back end to put the payload in the jsonPayload field instead of textPayload. Maybe the lib you are using could hook into this somehow? If you wanted direct support for this I might try opening up an issue GCP issue tracker for Cloud Functions. Hope this helps.

[Sheshagiri]

@codyoss thanks for looking at this.

yup what you mentioned is the standard way and as I mentioned in my original comment its documented in stackdriver documentation. Since it was working on app engine standard env I was hoping that it would work in cloud function env as well but there could be some changes.

I'll try opening an issue with GCP.

[sharkyze]

Hey @Sheshagiri, I am having the same issue where the same logger's output is being parsed as jsonPayload on GAE and as textPayload on Cloud functions.

Did you ever fix this issue or open another issue with GCP ?

[Sheshagiri]

@sharkyze
I’m still using logrus and my logs are being parsed as text.
I did open a ticket against google cloud. More details are in https://issuetracker.google.com/issues/145068625

[sharkyze]

Thank you @Sheshagiri for your answer.

It doesn't look like they are working on a fix for this unfortunately. I will give this package a try in this case: https://godoc.org/cloud.google.com/go/logging

[tbpg]

Related to https://github.com/GoogleCloudPlatform/golang-samples/issues/802.

[tbpg]

You can now log JSON messages to stdout and they will be parsed as JSON. For example:

fmt.Println(`{ "message" : "msg", "severity": "error" }`)

[mitchellsuter]

This is still a problem in Python. When trying to replicate the following, I am unable to get the logger to parse to json:
https://cloud.google.com/functions/docs/monitoring/logging

[codyoss]

@mitchellsuter If you are having issues with the Python client, please bring this up in that repository: googleapis/python-logging.

[clarencedesmarais]

I think @mitchellsuter was referring to the example provided in the docs, which doesn't use this lib googleapis/python-logging.

I am facing the same issue.

Brought up here https://github.com/GoogleCloudPlatform/python-docs-samples/issues/5036