Fluent-bit: Support NATS Streaming Server (stan)

[digitalsanity]

I would like to use NATS Streaming Server (aka stan -- https://github.com/nats-io/nats-streaming-server ) as my logging/analytics back-end. It uses a similar protocol to the existing NATS plugin but has a few additions/changes to support additional desirable features.

Reference:
A golang NATS Streaming Server client is available @ https://github.com/nats-io/stan.go

Steps:
Clone the existing nats plugin, replace connector code, review methods, arguments, configuration options.

Purpose:
NATS is lightweight (~30MB RAM) and ideal for streaming analytics data but the data is transient.

NATS Streaming Server is a lightweight addition (~10MB RAM) that allows for disk-backed, rewindable streams tracked per subscriber with at-least-once delivery. It runs on the same port/connection as NATS but uses a different client/protocol.

I am currently using a workaround of running a separate "replicator" service that copies NATS subjects to NATS Streams. I was able to use fluent-bit tail (throttled at ~1000 records/second) to import~5.6GB of nginx json access.log data (~5 million json records) with very little CPU/memory usage. The replicator: https://github.com/nats-io/nats-replicator

NATS Streaming Server is an ideal partner for event-sourcing Fluent Bit inputs.

[edsiper]

@wallyqs would you please point me out to the Protocol spec?

[digitalsanity]

Sorry - I was thinking fluent-bit used golang plugins .. the nats.c client has a stan (streaming) example:

https://github.com/nats-io/nats.c/tree/master/examples/stan

[edsiper]

Pure, fast and optimized C code :)

[wallyqs]

@edsiper the protocol spec is here internally it is using NATS Requests/Responses with the payloads encoded with protocol buffers, but the STAN library that @digitalsanity pointed out should do all of that for you: https://github.com/nats-io/nats.c/blob/master/examples/stan/pub.c#L79-L96

[edsiper]

thanks @wallyqs , I will take a look

[jhonatanmorais]

Hi guys, is there any working in progress about it?

[rsilva-nk]

Hi @edsiper ! Thanks for taking on that it surely also help others.

@digitalsanity pls, how does it work that workaround you mentioned using NATS Replicator.. Fluent-bit -> NATS -> NATS-Replicator - NATS-Streamming? Do you mind sharing your Replicator config?

[digitalsanity]

@rsilva-nk

I cloned this repo:

https://github.com/nats-io/nats-replicator

Using a config like this:

`reconnectinterval: 5000,

logging: {
Time: true,
Debug: true,
Trace: false,
Colors: true,
PID: false,
},

monitoring: {
httpport: 9090,
}

nats: [
{
Name: "nats",
Servers: ["172.17.0.1:4222"],
ConnectTimeout: 50000,
MaxReconnects: 50000,
ReconnectWait: 15000,
}
]

stan: [
{
Name: "stan",
NATSConnection: "nats",
ClusterID: "test-cluster",
ClientID: "replicator-one"
}
]

connect: [
{
type: NATSToStan,
id: "alpha",
IncomingSubject: "in",
IncomingConnection: "nats",
OutgoingChannel: "foo",
OutgoingConnection: "stan",
},
{
type: NATSToStan,
id: "alpha",
IncomingSubject: "my_cpu",
IncomingConnection: "nats",
OutgoingChannel: "my_cpu.stream",
OutgoingConnection: "stan",
},
{
type: NATSToStan,
id: "alpha",
IncomingSubject: "abc.abc1.access-log-test",
IncomingConnection: "nats",
OutgoingChannel: "stan.abc.abc1.access-log-test",
OutgoingConnection: "stan",
},

],
`
The IncomingSubject stream gets copied to the OutgoingChannel stream. You can then use a "stan" client to subscribe to the outgoing channel and rewind/etc.

I am launching with this:

docker run -v /work/src/nats-replicator/conf/replicator.conf:/conf/replicator.conf -p 9090:9090 "nats-io/nats-replicator" -c /conf/replicator.conf

This expects your NATS to be running on 172.17.0.1:4222.

So now whenever fluentbit logs into abc.abc1.access-log-test NATS, it gets automatically forwarded to stan.abc.abc1.access-log-test STAN using the stan instance running on that same NATS. Replicator acts as a standard subscriber to NATS and publisher for STAN.

*STAN will be replaced by LiftBridge eventually .. but there were some issues last I tested.

[rsilva-nk]

@digitalsanity thank you so much for sharing.

I ended up using Fluentd instead with a NATS Stan community plugin. However, still considering to use Fluent-bit as collector and Fluentd as proxy based on Forward plugin for interoperability between Fluent-bit and Fluentd.

In any case, NATS Stan plugin for Fluentd: https://github.com/hc-chien/fluent-plugin-nats-streaming