Fluent-bit: Elasticsearch output should probably not use a type (flb_type)

[dperetti]

In Elasticsearch 7.0, document types are now deprecated.
However, fluent-bit ES output sets by default the document type to flb_type which can yield tricky issues when you're not an Elasticsearch expert.
It would be wise to set the type do "_doc" by default or not use a type at all.

[pappnase99]

Since ES 7.* is available on AWS as well, one would expect more people running into this. Any news?

[nanmu42]

I just ran into this issue:

2019-11-12T11:02:55.01839169Z "stacktrace": ["java.lang.IllegalArgumentException: Rejecting mapping update to [logstash-2019.11.12] as the final mapping would have more than 1 type: [_doc, flb_type]",

Adding line

Type  _doc

to Fluent Bit Elasticsearch output config walks around this problem.

My Enviroment:

fluent/fluent-bit:1.2.1
Elasticsearch: 7.4.2

[nikinaydenov]

It does not solve the problem. Fluent bit should stop sending _type to ES 7. The message on HTTP is clear:
HTTP/1.1 200 OK Warning: 299 Elasticsearch-7.4.2-2f90bbf7b93631e52bafb59b3b049cb44ec25e96 "[types removal] Specifying types in bulk requests is deprecated."
The same will stop to work completely (with Error, not warning) in ES8, but this is in the future.
Probably we need a separate setting for the version or special value for "Type: off" or what ever you decide, but with final effect of not sending _type at all.
Now the body of _bulk looks like:
{"index":{"_index":"indexname-2019.11.24","_type":"_doc"}}
It should looks like:
{"index":{"_index":"indexname-2019.11.24"}}

[grimmxsleeper]

bump, running into this issue as well @edsiper

[yuha0]

We'd also like to see this to be implemented.
Elasticsearch 7.0.0 was released 9 months ago and a lot of users have upgraded since then.

[hornetmadness]

I'm on 7.2.0 and fluent-bit 1.3.5 and still getting this error

[2020/01/08 21:11:23] [debug] [out_es] HTTP Status=200 URI=/_bulk [2020/01/08 21:11:23] [error] [out_es] could not pack/validate JSON response {"took":2338,"errors":true,"items":[{"index":{"_index":"logs-2020.01.08","_type":"flb_type","_id":"dBv_hm8BnqxPixOGYeWJ","status":400,"error":{"type":"illegal_argument_exception","reason":"Rejecting mapping update to [logs-2020.01.08] as the final mapping would have more than 1 type: [_doc, flb_type]"}}},

[debu99]

we have the same issue

[angristan]

This can be closed? https://github.com/fluent/fluent-bit/commit/04ed3d8104ca8a2f491453777ae6e38e5377817e

[edsiper]

thanks, closing it.

[cpu100]

set

[OUTPUT]
    Name            es
    Type            _doc

then delete existing indexes to avoid more than 1 type error.

DELETE /logstash-*

[mikebou]

I don't believe this issue is actually solve. Elastic 7.8 will reject the POST to _bulk if the _type is part of the parameter set.

I have entered the supporting document in issue https://github.com/fluent/fluent-bit/issues/2448