Kibana: Setting elasticsearch.sniff* settings in Kibana.yml causes authentication issues

[krisATelastic]

Kibana version:
7.2.0

Elasticsearch version:
7.2.0

Server OS version:
Centos 7.x

Browser version:
Chrome Version 75.0.3770.142 (Official Build) (64-bit)

Browser OS version:
Mac Mojave

Original install method (e.g. download page, yum, from source, etc.):
Yum from our repo

Describe the bug:
When setting the following keys in the kibana.yml configuration file;

elasticsearch.sniffInterval
elasticsearch.sniffOnConnectionFault
elasticsearch.sniffOnStart

... to anything other than false, the elasticsearch.username and elasticsearch.password seem to be ignored and revert to anonymous connections;

{"type":"log","@timestamp":"2019-07-30T02:55:36Z","tags":["status","plugin:[email protected]","error"],"pid":19934,"state":"red","message":"Status changed from yellow to red - [security_exception] missing authentication credentials for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } }","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

I also found that putting a typo in the elasticsearch.password to force a 403 would fail as expected however.

{"type":"log","@timestamp":"2019-07-30T03:06:05Z","tags":["status","plugin:[email protected]","error"],"pid":20139,"state":"red","message":"Status changed from yellow to red - [security_exception] failed to authenticate user [kibana], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } }","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

Steps to reproduce:

1. Build simple single node cluster with Kibana on same node
2. Enable xpack.security.enabled: true on Elasticsearch, run setup passwords
3. Configure kibana.yml to use the kibana user credentials in the elasticsearch.username and elasticsearch.password settings
4. Start Kibana, it will connect fine.
5. Stop Kibana and set elasticsearch.sniffInterval: 3000, elasticsearch.sniffOnConnectionFault: true and elasticsearch.sniffOnStart: true
6. Start Kibana, the above error should appear along with a long list of other authentication failures for missing authentication credentials

Work around
Do not use these settings, either drop them from the configuration or set all 3 to false.

[elasticmachine]

Pinging @elastic/kibana-operations

[azasypkin]

@elastic/kibana-operations I've tagged you on this issue since you implemented this feature in https://github.com/elastic/kibana/pull/21928, but let me know if the issue is related to the authentication itself that Security Team should normally handle.

[krisATelastic]

I can also confirm that i've reproduced this on version 7.3.1 also.

With elasticsearch.sniffInterval and elasticsearch.sniffOnStart settings set, this is returned in the Kibana log;

Sep 5 00:24:31 krisr-kibanatest01 kibana[24825]: {"type":"log","@timestamp":"2019-09-05T00:24:31Z","tags":["license","warning","xpack"],"pid":24825,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [security_exception] missing authentication credentials for REST request [/_xpack], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/_xpack\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication credentials for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication credentials for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}"}

[tylersmalley]

@jbudz do you have some cycles to look into this?

[jbudz]

Yeah, I'll have to take a closer look at client lib but this came up in testing. The sniff settings will pull cluster information from elasticsearch, but it isn't necessarily the same endpoints and information kibana uses to connect (proxies and so on). I didn't find it very useful in practice, but there's room for improvement with auth here.

[benatbermejo]

Reproduced on version 7.5. When I change elasticsearch.sniffOnStart to true, I get 401 error trying to get license information on startup.

[xiu]

We have the same issue in 7.4.2

[douniwan5788]

same issue in 7.6.2
Thank God! finally found the reason

[joshdover]

@elastic/kibana-platform do we know if this problem also exists with the new client?

[pgayvallet]

@delvedor ^ Maybe you will be able to answer this one?