Vector: bug(sinks gcp_stackdriver_logs): Cannot set log messages severity level

[rubn-g]

Hi Vector team,

When using the gcp_stackdriver_logs sink, all messages have a severity level of DEFAULT as the severity property of the LogEntry is not being set.

I can set a severity field, but it's added to the jsonPayload property of the LogEntry and stackdriver is not gettint that as the message severity.

Proposed solution

We should have a severity configuration with this posible values:

DEFAULT  # default configuration value
DEBUG
INFO
NOTICE
WARNING
ERROR
CRITICAL
ALERT
EMERGENCY

Thank you all

[Hoverbear]

Tagging this as a bug since you're not able to accomplish what you need. :)

[rubn-g]

Forgot to mention, we should be able to set that value from a log field, with templating, or maybe not having a config at all and always get the severity field if it's set.

I don't know, however you do it, it will be fine. Thanks!

[mikehardenize]

For another data point, this is also causing me problems. Just spent quite a lot of time trying to make it work until I came across this issue.

One concern I do have is that I'm using the journald source which sticks a PRIORITY number in the jsonPayload, but the numbers do not match the numbers that Stackdriver expects for LogServerity. E.g for INFO, I'm getting a PRIORITY of "6" from the journald sink, but Stackdriver expects the code "200" for a log entry with a severity of INFO.

I've been looking to see if there is a transform that would help in conversion, but I can't see one. Perhaps it would be nice if the journald source also provided a PRIORITY_NAME of "INFO" when the PRIORITY is 6, and if the Stackdriver sink could accept "INFO" for severity?

[rubn-g]

@mikehardenize as a workaround, maybe you could use a lua type transform to make the replacements? it's quite easy to do a lua script to get what you want

[bruceg]

While a lua transform would be able to replace the priority level number, there is no way to set the severity in the outgoing request, as that would end up just another field in the jsonPayload. I am working on support for this now.

[mikehardenize]

I just took a look at the PR, and it does allow us to set the severity, but only by the stackdriver severity code. I think it's pretty unlikely that there will be any sources that produce a field for example containing "400" to indicate that the log level is WARN. I think it would be extremely useful if the field containing the severity key could take many values. E.g all of the following values would map to a stackdriver 400 severity code:

• WARN
• WARNING
• warn
• 400
• 4

Where the final "4" is the syslog code.

Perhaps even anything matching ^(?i)WARN.*$

With the PR as it is right now, it seems to me that anybody that wants to set a stackdriver severity (pretty much anyone using that sink?), will have to write a LUA based transform first to map the codes...

[bruceg]

I agree some remapping facility to be able to produce the required numbers is a good idea. Since these numbers are specific to stackdriver, we are leaning to doing it internal to this this sink. For flexibility, I am planning to use a case-insensitive prefix match as you suggest for textual values.

I am however concerned about converting syslog code numbers directly as it could lead to some subtle surprises. For one, the syslog levels range from 7 for DEBUG to 0 for EMERG while stackdriver ranges from 100 for DEBUG to 800 for EMERGENCY. More importantly, there is a conflict with 0 meaning EMERG in syslog but DEFAULT for stackdriver. You could argue that if a severity key is configured, we can ignore DEFAULT, but it seems to violate the idea of "least surprise".

[mikehardenize]

This still leaves me in the situation of needing to stick a LUA based mapping transform in the middle, as the journald source only outputs a syslog priority code. This looks pretty straight forward, but it still leaves a bitter taste in my mouth. Perhaps what is needed is for the journald source to include a "severity" field containing the syslog name, just like the syslog source does?

[bruceg]

I agree this is awkward at best for the use case you describe. Since PRIORITY appears to be a relatively standard journald field, adding remapping for it seems like a reasonable thing. I think it'll have to be opt-in, but that's easy enough.

[mloughran]

Very pleased to find #2732, and would not wish to delay it's merging :) (please say if it would be better to raise a separate PR).

Would you consider adding FATAL to the remap_severity function? My personal interest is allowing a clean mapping of Ruby's log levels (debug < info < warn < error < fatal), however the use of FATAL nomenclature instead of CRITIAL/ALERT/EMERGENCY seems fairly common, e.g.:

• Ruby https://ruby-doc.org/stdlib-2.4.0/libdoc/logger/rdoc/Logger.html
• Log4j https://logging.apache.org/log4j/2.x/manual/customloglevels.html
• Golang (nomenclature only, standard logger not leveled)

At the risk of opening a can of worms, I suspect it would also be useful to add TRACE. Google's stackdriver logs say DEBUG: (100) Debug or trace information., and trace is used by e.g.:

• Rust https://docs.rs/log/0.4.0/log/enum.Level.html
• Log4j https://logging.apache.org/log4j/2.x/manual/customloglevels.html
• .NET https://docs.microsoft.com/en-us/aspnet/core/fundamentals/logging/?view=aspnetcore-3.1#log-level

How about mapping FATAL -> 800 & TRACE -> 100 ?

[bruceg]

@mloughran This makes sense and is easy to add.

[Sharath52]

Can anyone tell what permissions/roles were given for the gcp service account to send logs to stackdriver?

[bruceg]

IIRC this requires roles/logging.logWriter for the logging, and roles/logging.viewer for the healthcheck to succeed.