Logstash: deprecated config setting document_type warning

[jdoles]

The following warning is logged when starting logstash:
[2019-02-21T14:40:07,771][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc.

Might be similar to #10352 ?

• Version: 7.0.0 beta 1
• Operating System: Windows 2016
• Config File (if you have sensitive info, please remove it):

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

• Sample Data:
• Steps to Reproduce:
  Start logstash
  logstash -f ../config/logstash-sample.conf

[fabiogermann]

@jdoles did you find a solution? I'm currently experiencing the same issue.
Logstash version: 7.2.0
Elasticsearch version: 7.2.0
OS: Debian Linux, Stretch

[jdoles]

@fabiogermann No I have not. In fact it's still present in 7.3.1.

[fktkrt]

This issue still persists in 7.5.0.

[danlsgiga]

Still in 7.5.1 as well

[RamiusKahn]

7.5.2 also.
OS = Ubuntu 18.04
ES = 7.5.2
LS = 7.5.2

[yaauie]

I believe the warning is not coming from your main pipeline, but from the x-pack monitoring pipeline (what have been called "internal collectors"), which is spawned separately within Logstash for shipping monitoring metrics about Logstash back to your Elasticsearch cluster.

If you can use the pipeline.separate_logs feature (introduced in Logstash 7.5.0 via https://github.com/elastic/logstash/pull/11108, https://github.com/elastic/logstash/pull/11177/files), we could positively identify that the deprecation warning is coming from the monitoring pipeline.

---

Analysis:

The X-Pack Monitoring pipeline (enabled with xpack.monitoring.enabled: true or other xpack.monitoring.* settings in a complete distribution of Logstash, not available in the OSS-only distribution) still uses the deprecated document_type declaration in its configuration of the Elasticsearch Output Plugin in its pipeline template, which is why we are seeing a deprecation warning about using the document_type option:

~ ruby
document_type => "%{[@metadata][document_type]}"
~
-- elastic/logstash:x-pack/lib/template.cfg.erb:[email protected]

The functionality continues to work because Elasticsearch 7.x's special _monitoring/bulk endpoint is separately-versioned, and includes an adapter that allows it to work with events being pushed using older API versions.

---

Starting with version 7.3, Metricbeat now ships with the logstash-xpack module for collecting relevant data from the Logstash API, which is the favored way of monitoring Logstash within the Elastic Stack moving forward. That means that we are now beginning to discourage the use of the internal collectors that exhibit the above behaviour (e.g., deprecation warnings have been merged to 7.x [targeting 7.7.0]), and we plan to remove the implementation entirely in a future major release of Logstash, likely 8.0.0 (see: https://github.com/elastic/logstash/issues/11169). Before that happens, we are making specific steps to ease the transition (see: https://github.com/elastic/logstash/issues/11403, https://github.com/elastic/logstash/issues/11328).