Fluent-bit: Feature Request: AWS Cloudwatch logs support

[max-rocket-internet]

Like many people on AWS, we send our logs to Cloudwatch Logs. The reason is that once the logs are in Cloudwatch, they are easy to push to other AWS services. e.g. S3, AWS Elasticsearch, own ES and more.

Currently the only easy way to get logs from k8s nodes into Cloudwatch Logs is to use fluentd-kubernetes-daemonset and like probably most people here know, fluentd uses a lot of resources. Currently some of our clusters have to give it 750MB of memory on every node otherwise we hit OOM 😧

So, I would like to request an output plugin for Fluent Bit to send logs to AWS Cloudwatch Logs 🚀

[repeatedly]

Currently some of our clusters have to give it 750MB of memory on every node otherwise we hit OOM

Do you use file buffer?

[max-rocket-internet]

Do you use file buffer?

@repeatedly No we don't. Should we?

[repeatedly]

Yes. With memory buffer, if you stores 500MB logs in the buffer, fluentd consumes 500+MB. fluent-bit is also same. file buffer mitigates this problem.
Of course, fluent-bit is good choice on forwarder side :)

[max-rocket-internet]

OK cool, I'll test that configuration to reduce memory usage.

But still, it would be great to have Cloudwatch logs support in Fluent-bit 😃

[cosmo0920]

AWS SDK for C is not provided.
Instead, AWS SDK for C++ is provided.
Is C++ code acceptable for fluent-bit project?

[nigels-com]

I doubt fluent-bit wants a C++ dependency. It's probably not all that complex to adapt the HTTP code for talking AWS REST API in C, but it would be non-trivial.

[PettitWesley]

I think the best option for now is to create an external plugin using Fluent Bit Go. I'm working on writing one.

Here are the options that I'm thinking of supporting:

• region: AWS Region that contains the Log Group.
• log_group: CloudWatch Log Group to send logs to.
• log_stream_prefix: Prefix for the Log Stream name. The log tag is appended to the prefix to construct the full log stream name. This is similar to the use_tag_as_stream option in the Fluentd CloudWatch plugin, except you add a prefix which is prepended to the tag to create the Log Stream name. I think this is useful since a lot of the time the tag name might be fairly meaningless (ex a Docker Container ID).
• log_stream: Name of the Log Stream. Not compatible with option log_stream_prefix .
• auto_create_stream: Automatically create the log stream. Defaults to true.
• auto_create_group: Automatically create the log group. Defaults to false.

Example configuration:

[OUTPUT]
    Name  cloudwatch
    Match some-tag
    region us-west-2
    log_group my-log-group
    log_stream_prefix my-prefix-

When the plugin receives log messages, it would auto-create the Log Stream "my-prefix-some-tag" and then send logs there.

@max-rocket-internet @cosmo0920 @nigels-com @edsiper Any comments or suggestions?

[max-rocket-internet]

I think the best option for now is to create an external plugin using Fluent Bit Go. I'm considering writing one myself.

Sounds good! I didn't even know the plugins were separate. Great.

@max-rocket-internet @cosmo0920 @nigels-com @edsiper Any comments or suggestions?

I would say that sounds good 👍

[mbenabda]

@PettitWesley thank you for working on this ! we've restrained from migrating to fluent-bit because there was no plugin for cloudwatch, so yours will enable us to make the move !
One suggestion for your fluent-bit cloudwatch plugin though: try to use the same configuration api as https://github.com/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs to limit the migration effort from fluentd, since fluentd users probably use this plugin as an forwarder for cloudwatch

[PettitWesley]

I'm glad to see excitement for this!

I should've noted in my original comment that I work for AWS, and that this plugin will be under AWSLabs.

Let me know if there are other AWS Services that you wish had Fluent Bit support. (No promises of course on what we'll implement, but we do take your requests/feedback seriously!)

[rverma-nikiai]

It would be nice if this can be supported by app-mesh as an auto injected sidecar.
I doubt it would require and support on the plugin side implementation but you may need to include it in app mesh/ Istio deployment. Something like x-ray sidecar would make it really awesome.

Another big advantage is since eks CNI has IP limitation on instance types, sidecar will save us from the extra pod.

[max-rocket-internet]

@PettitWesley

Let me know if there are other AWS Services that you wish had Fluent Bit support.

It would be great to be able to go direct to Kinesis Data Firehose. The reason is that in AWS you are able to stream from Firehose to AWS Elasticsearch service directly. And the AWS ES service is a very popular log solution as it includes Kibana. Using this arrangement once the logs are collected by fluent-bit and sent to firehose, transport (and S3 buffering) to ES is done completely inside AWS without needing to via ES VPC or public endpoint. And that's quite an elegant solution 🙂

[PettitWesley]

@max-rocket-internet You're in luck! We're working on a Firehose plugin as well! 😃

On the topic of using Firehose & Fluentd as a log solution; I'd like to link to this.

[cosmo0920]

@PettitWesley I'm working on fluent-bit-go-cloudwatch-logs.
I've published very recently the above package, so it is not matured to use in production for now!

log_stream_prefix: Prefix for the Log Stream name. The log tag is appended to the prefix to construct the full log stream name. This is similar to the use_tag_as_stream option in the Fluentd CloudWatch plugin, except you add a prefix which is prepended to the tag to create the Log Stream name. I think this is useful since a lot of the time the tag name might be fairly meaningless (ex a Docker Container ID).

I doubt use_tag_as_stream is really useful.
fluent-bit-go c-shared plugin might be used in sidecar pattern and micro containers.
It is very complicated to achieve functionality, I think.

auto_create_stream: Automatically create the log stream. Defaults to true.
auto_create_group: Automatically create the log group. Defaults to false.

fluent-plugin-cloudwatch-logs also supports these functionality but it provides only auto_create_stream parameter.

[vasilij-icabbi]

@PettitWesley is there any news on when output to cloudwatch will see first release?
Thank you.

[mbenabda]

@cosmo0920 I think you're mistaken in assuming that it's only going to be used as a sidecar. The company i'm working for, for instance, is not planning on having an extra process per container for logs collection, but rather one per host. and use_tag_as_stream would be very useful there. being able to use the record's metadata would also be very useful. for example, we use kubernetes metadata to build our group and stream names

[PettitWesley]

The CloudWatch Logs and Kinesis Data Firehose plugins repositories are now public 🚀

Check them out here:
https://github.com/aws/amazon-kinesis-firehose-for-fluent-bit
https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit

You can pull the code, build it, and begin using it now. We understand this is not a very convenient way to consume these plugins though; I'll have an update on that in the near future.

[PettitWesley]

We understand this is not a very convenient way to consume these plugins though; I'll have an update on that in the near future.

Here's that update! We are distributing an image on Docker Hub: https://hub.docker.com/r/amazon/aws-for-fluent-bit

And also Amazon ECR; check out the READMEs of the plugins for that.

Finally, a small plug, we wrote a tutorial on using Fluent Bit with Amazon ECS and Amazon EKS: https://aws.amazon.com/blogs/opensource/centralized-container-logging-fluent-bit/

[talawahtech]

@PettitWesley where can I find the github repo/dockerfile for the aws-for-fluent-bit image?

[PettitWesley]

@talawahtech Currently, I have it on the Docker Hub page: https://hub.docker.com/r/amazon/aws-for-fluent-bit

[talawahtech]

@PettitWesley thx!

[edsiper]

FYI: just a heads up, we (Fluent Bit) are working together with AWS team intro bring this kind of plugins and interfaces into the core.

Current status: building AWS Signv4 support into Fluent Bit.

[PettitWesley]

github repo/dockerfile for the aws-for-fluent-bit image?

(Old News) We have a GitHub repo: https://github.com/aws/aws-for-fluent-bit

[PettitWesley]

A core plugin for CloudWatch was released in 1.5!

https://docs.fluentbit.io/manual/v/1.5/pipeline/outputs/cloudwatch