Google-cloud-go: logging: Parsing pub/sub message to logging.Entry giving Payload:<nil>

[chiragtayal]

Logging

Environment

* OS *
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial

Audit log -> pubsub sink -> push sub -> custom pub/sub parser

Go Environment

$ go version: go1.13.6
$ go env:

Code

func Parser(msg *pubsub.Message) (*logging.Entry, string,  error) {
        var entry logging.Entry
        if err := json.Unmarshal(msg.Data, &entry); err != nil {
        return nil, logType, errors.New(fmt.Sprintf("jsonpb unmarshal error: %s", err))
    }
 ....

In this I am getting payload as , do you know what's wrong with the code?

Expected behavior

Payload should be parsed into []byte for AuditLog

Actual behavior

Payload:<nil>

logging.Entry:
{Timestamp:2020-07-04 19:05:19.727 +0000 UTC Severity:Notice Payload: Labels:map[] InsertID:-c7futsdaq2s HTTPRequest: Operation: id:"operation-1593889515470-5a9a252617d6a-5b533f9e-10ee1201" producer:"compute.googleapis.com" last:true LogName:projects/pp-devcaas-service01/logs/cloudaudit.googleapis.com%2Factivity Resource:type:"gce_instance" labels:{key:"instance_id" value:"7225393337839828246"} labels:{key:"project_id" value:"pp-devcaas-service01"} labels:{key:"zone" value:"us-central1-a"} Trace: SpanID: TraceSampled:false SourceLocation:}

[codyoss]

Hey,

The logging.Entry struct does not support unmarshaling like this today. To get the behavior I think you are looking for you could do something like this:

                         // import "google.golang.org/genproto/googleapis/cloud/audit"
            var m map[string]json.RawMessage
            if err := json.Unmarshal(msg.Data, &m); err != nil {
                log.Printf("jsonpb unmarshal error: %s\n", err)
            }
            var aLog *audit.AuditLog
            json.Unmarshal(m["protoPayload"], &aLog)

or

                         // import loggingpb "google.golang.org/genproto/googleapis/logging/v2"
            var entry loggingpb.LogEntry
            if err := json.Unmarshal(msg.Data, &entry); err != nil {
                log.Printf("jsonpb unmarshal error: %s\n", err)
            }

[chiragtayal]

@codyoss

I have tried using following code but it failed saying it cannot parse severity

            var entry loggingpb.LogEntry
            if err := json.Unmarshal(msg.Data, &entry); err != nil {
                log.Printf("jsonpb unmarshal error: %s\n", err)
            }

[codyoss]

That is due to a bug that has been fixed, but not released. For now you can get that code by running go get -u cloud.google.com/go/logging@master. I will try to get a tagged release out soon.

[chiragtayal]

Do you know when new version of logging will be released ?

[chiragtayal]

@codyoss
Still same error after doing go get -u cloud.google.com/go/logging@master
json unmarshal error: json: cannot unmarshal string into Go struct field LogEntry.severity of type ltype.LogSeverity

[codyoss]

Whoops sorry about that, I thought I had tested it to make sure it worked...

This worked for me

import (
        _ "google.golang.org/genproto/googleapis/cloud/audit"
    loggingpb "google.golang.org/genproto/googleapis/logging/v2"
        "google.golang.org/protobuf/encoding/protojson"
)
...
var entry loggingpb.LogEntry
uo := protojson.UnmarshalOptions{DiscardUnknown: true}
if err := uo.Unmarshal(msg.Data, &entry); err != nil {
    log.Printf("unmarshal error: %s\n", err)
}
log.Println(entry.Severity)

The keys here were to:

• make use of the protojson package. It is the recommended library when working with protos
• adding the _ import to get that proto message structure registered with the unmarshaller

The logging library I had you import,go get cloud.google.com/go/logging@master, was not really needed, sorry about that.

Let me know if you are still having issues.

[chiragtayal]

@codyoss thanks this works