Fluent-bit: [Output] ElasticSearch - Ability to configure number_of_shards & number_of_replicas

[Kanshiroron]

Hello,

When configuring FluentBit with the ElasticSearch output, it's currently not possible to set the number_of_shards and the number_of_replicas settings in the ElasticSearch index. This might be very useful when seeking performances & HA on a ElasticSearch cluster.

Here are current settings used by FluentBit:

$ curl http://elasticsearch:9200/logstash-2018.08.25/_settings?pretty
{
  "logstash-2018.08.25" : {
    "settings" : {
      "index" : {
        "creation_date" : "1535155195523",
        "number_of_shards" : "5",
        "number_of_replicas" : "1",
        "uuid" : "Ru9Nmum4Q8K2HUfMZNh6kw",
        "version" : {
          "created" : "6030299"
        },
        "provided_name" : "logstash-2018.08.25"
      }
    }
  }
}

Thanks!

[MohdAhmad]

Does anyone know if there is a way to accomplish this?

[krudskoy]

We have the same problem. Does anyone know if there is a way to accomplish this?

[cosmo0920]

Elasticsearch can handle this settings changing with index template:
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html

fluent-bit does not have index template handling/installing feature for now.

[cliedeman]

Simple fix for a dev cluster

PUT _template/kubernetes_cluster_template
{ "index_patterns": ["kubernetes_cluster-*"], "settings": { "number_of_shards": 1, "number_of_replicas": 0 } }

[sidps]

@cosmo0920 is index-template support within fluent-bit planned?

[mohdahmad17]

+1

[mohdahmad17]

I feel this is a legitimate ask. @edsiper can we please reopen this?

[edsiper]

if there is a specific requirement or enhancement like: "please we need to have feature A & B that accomplish C" would be easier to put in the roadmap. If that is the case please open a new github issue with further details.

[Kanshiroron]

@mohdahmad17 Sorry I closed it because I felt like configuring this via Elastic Search index template is definitively to correct way to achieving this.