Rsyslog: New module request: omclickhouse

[jay7x]

As far as I see ClickHouse is pretending to replace ElasticSearch for log analysis. So it'd be great to have omclickhouse module in rsyslog.

Few links:

• ClickHouse itself:
  
  
  
  • https://clickhouse.yandex/
  
  
  • https://github.com/yandex/ClickHouse
  
  
• ClickHouse interfaces: https://clickhouse.yandex/docs/en/interfaces/index.html
• Logstash module: https://github.com/mikechris/logstash-output-clickhouse
• LogHouse (Kubernetes logging solution): https://github.com/flant/loghouse
• Post in Altinity blog about Logstash + ClickHouse: https://www.altinity.com/blog/2017/12/18/logstash-with-clickhouse
• loghouse-acceptor (syslog -> CH relay): https://github.com/kak-tus/loghouse-acceptor

[jackpgao]

watching

[evmnaumov]

Hi!
What's new in this issue?

[davidelang]

On Thu, 2 Aug 2018, evmnaumov wrote:

Hi!
What's new in this issue?

It looks like nothing. I'll point out that the omelasticsearch module was
contributed from outside the core developers. Getting the core developers to
create a new module can be a slow process (there are so many demands on their
time), so either contributing code, or hiring adiscon (or some other developer)
to create the module is the way to go if you want it quickly.

David Lang

[ghost]

I have created an output module for Clickhouse.
Documentation on how to use it will soon be committed to the rsyslog-doc repository.
It would be great if someone would be willing to test the module.

[jay7x]

Wah! Great! Thank you @PascalWithopf! Hopefully I didn't spent much time on it yet :-D
I'll try to test myself in a week. I'll spread this a bit in russian community now as well.

[jay7x]

@PascalWithopf As I see from code you're using INSERT INTO table (...) VALUES (...) [, (...) ] syntax. Was there anything behind this decision? I was targeting to use TabSeparated (TSV) format for input. ClickHouse prefer single big batch (e.g. 100k records) once per second over 100k sequential INSERTs. So bulk mode will be enabled by default in most installations I guess. TSV format may save some bandwidth (200k saving on brackets only 😂) and may be a bit faster to parse on server side (just guessing here tho).

And I'd prefer to have 2 templates

• one for query beginning (INSERT INTO logs FORMAT TSV)
• another for query payload (%TIMESTAMP%\t%HOST%\t%TAG%\t%MSG%\n). This one is used when building batches.

This allow to use any format user prefers actually. Not sure how heavy is template expansion/substitution though..

P.S. Here is performance section in the end: https://clickhouse.yandex/docs/en/query_language/insert_into/

[ghost]

@jay7x I was just most familiar with this syntax, no special reason.
At the moment the module has only basic functions, supporting multiple input formats is a good idea for further development of the module.
Splitting the template into two parts is a good idea.

[rgerhards]

I have merged the module today - feedback is very much appreciated.

@jay7x I, too, would love to see TSV format. It would be great if you could open a new issue for this, maybe with a pointer to the doc for the casual onlookers (like me ;)).

Opening new issues for feature requests and bug reports is in general the way to move this plugin forward.