Manageiq: Should we still log to `log/*.log` files on podified / systemd

[agrare]

On k8s we have a multicast logger to write logs to stdout so they can be aggregated across pods, as far as I know nothing rotates the /var/www/miq/vmdb/log/*.log files, do the pods just fill up their filesystem and die at that point?

On systemd we write logs to journald, these logs can be queried by worker type, specific worker, and by other metadata. We also have the ability of exposing an appliance's journald over an API so they could be aggregated across appliances.

In both production environments the actual log files are at best duplicates of the "main" logging method. Aside from legacy / familiarization are there benefits of having the log files still?

[agrare]

cc @jrafanie @Fryguy @bdunne

[agrare]

We would need to look at exporting the journald logs for support purposes.

[Fryguy]

On containerized, we've seem some strangeness with logs not being sync'ed properly even though sync=true is set, but otherwise, I don't know why we would write to an _actual_ log file. Same goes for journald. I'd be :+1: for not writing to the actual log files in those cases.

[jrafanie]

I'd think in containers, we should do this with mind towards some form of centralized logging. It's absolutely brutal that we lose logging from the non-stdout loggers on reboot and stdout loggers on the 2nd restart. Combined with kubernetes seemingly "quietly" OOM processes that exceed memory limits with no audit/system event, it's very easy to lose these logs forever. Don't get me started on kubernetes throttling CPU when exceeding CPU limits without cutting an event or notifying anyone.

[agrare]

Yeah (totally off topic) but I think we need to have some form of log aggregator optional as part of our deployment for our own pods, but could be replaced by an "external log aggregator" if for example you already have something on your cluster.

Just to make sure I'm following you, you're not suggesting we wait to do this until we have centralized logging right? I think the log files get lost on a pod restart anyway just want to make sure I'm not missing something.

It is incredibly irritating not being able to see all our logs in one place and losing them if a pod crashes (especially if you want to see _why_ it crashed haha)

[Fryguy]

Internal log aggregator is a REALLY interesting idea. Wonder if we could piggyback onto https://github.com/RedHatInsights/haberdasher

[jrafanie]

Just to make sure I'm following you, you're not suggesting we wait to do this until we have centralized logging right? I think the log files get lost on a pod restart anyway just want to make sure I'm not missing something.

Nope, I think we can decide to log to stdout for everything in podified/systemd instead of all these log files but that will bring up other concerns such as noise with debug logging by default and merged logs. Is there a thought to just drop logging to these loggers entirely? I can imagine automation.log, audit.log, etc. might be needed in some situations.

Either way, if anyone gets cycles to work on logging, it sounds like this would need to gel with whatever we think we want to do with centralized logging.

[Fryguy]

I still think we should not have DEBUG as a default for containers...personally, I think we should just treat it like every other log and only do INFO until someone turns it on

[agrare]

Is there a thought to just drop logging to these loggers entirely? I can imagine automation.log, audit.log, etc. might be needed in some situations.

I think it depends on what we're getting out of these. Is it that we have a file or that it is separate from the "main" log stream?

If it is the latter then we can make use of the additional "metadata" that can be added to container logs (its just json) and journald (you can add arbitrary metadata and query it later). So journald we could add like _LOG=audit and see what we used to see in the audit.log

I've been wanting to put e.g. the provision id as metadata so that we can essentially query an entire provision run across evm.log and automate.log

[chessbyte]

why does log level default to DEBUG in containers? why is there a difference between appliance and podified?

[agrare]

I believe the logic was that its all going to a log analysis tool like elasticsearch+kibana, so it is easy to filter out level=debug, and if there are no downsides then why not capture everything. I don't happen to agree since there are downsides to it but that was the logic

[chessbyte]

yeah, I would rather we stick with a single default for the product no matter how it is deployed - as long as users have an easy way to change this setting

[bdunne]

Yes, the idea was that everything would be logged to STDOUT, Fluentd would pick this up and store it in Elasticsearch then you could view and filter it in Kibana.

If we change to INFO level, does this just apply to our app? (not the operator, postgres, memcached, kafka, zookeeper, etc.)
If we don't log DEBUG by default, do we go back to the app settings for the log level or ENV var, CR value or something else?

[Fryguy]

If it is the latter then we can make use of the additional "metadata" that can be added to container logs (its just json) and journald (you can add arbitrary metadata and query it later). So journald we could add like _LOG=audit and see what we used to see in the audit.log

Big :+1: from me for metadata on the log message to specify which "log" it comes from.

[Fryguy]

I've been wanting to put e.g. the provision id as metadata so that we can essentially query an entire provision run across evm.log and automate.log

I believe we already have this in the request_id field. Not sure on the specifics, but the "tracking_label" was supposed to end up in that field.

[Fryguy]

If we change to INFO level, does this just apply to our app? (not the operator, postgres, memcached, kafka, zookeeper, etc.)

Yes, just our app. I don't think we have a way to tweak the log levels for the rest.

If we don't log DEBUG by default, do we go back to the app settings for the log level or ENV var, CR value or something else?

It would come from the app Settings like it does now for the other loggers. I don't think we should use a CR value for that (at least not yet).