Kibana: Kibana fails to validate config if elasticsearch.hosts is not a string

[anapsix]

Kibana version: 6.6.0 / 6.6.1

Elasticsearch version: 6.6.0

Original install method (e.g. download page, yum, from source, etc.): stable/kibana v6.6.0 helm chart

Describe the bug:
According to Kibana Documentation elasticsearch.hosts is an array of strings. When passing array, it fails validation.

## Array example 1:
kibana.yml:
  elasticsearch.hosts: '["http://es1:9200", "http://es2:9200", "http://es3:9200"]'

## Array example 2:
kibana.yml:
  elasticsearch.hosts: ["http://es1:9200", "http://es2:9200", "http://es3:9200"]

## Array example 3:
kibana.yml:
  elasticsearch.hosts:
  - http://es1:9200
  - http://es2:9200
  - http://es3:9200

Steps to reproduce:

1. Use array as value to elasticsearch.hosts
2. Launch Kibana

Expected behavior: Expecting for Kibana to accept array of ES hosts

Provide logs and/or server output (if relevant):
Following error is displayed during plugin install step:

Plugin installation was unsuccessful due to error "Command failed: /usr/share/kibana/node/bin/node /usr/share/kibana/src/cli --env.name=production --optimize.useBundleCache=false --server.autoListen=false --plugins.initialize=false

 FATAL  ValidationError: child "elasticsearch" fails because [child "hosts" fails because [single value of "hosts" fails because ["hosts" must be a string]]]

{"type":"log","@timestamp":"2019-03-01T13:53:39Z","tags":["info","plugins-service"],"pid":51,"message":"Plugin initialization disabled."}
{"type":"log","@timestamp":"2019-03-01T13:53:41Z","tags":["plugin","warning"],"pid":51,"path":"/usr/share/kibana/src/legacy/core_plugins/ems_util","message":"Skipping non-plugin directory at /usr/share/kibana/src/legacy/core_plugins/ems_util"}
{"type":"log","@timestamp":"2019-03-01T13:53:42Z","tags":["fatal","root"],"pid":51,"message":"{ ValidationError: child \"elasticsearch\" fails because [child \"hosts\" fails because [single value of \"hosts\" fails because [\"hosts\" must be a string]]]
    at Object.exports.process (/usr/share/kibana/node_modules/joi/lib/errors.js:196:19)
    at internals.Object._validateWithOptions (/usr/share/kibana/node_modules/joi/lib/types/any/index.js:675:31)
    at module.exports.internals.Any.root.validate (/usr/share/kibana/node_modules/joi/lib/index.js:146:23)
    at Config._commit (/usr/share/kibana/src/server/config/config.js:139:35)
    at Config.set (/usr/share/kibana/src/server/config/config.js:108:10)
    at Config.extendSchema (/usr/share/kibana/src/server/config/config.js:81:10)
    at extendConfigService (/usr/share/kibana/src/plugin_discovery/plugin_config/extend_config_service.js:45:10) name: 'ValidationError' }"}

[elasticmachine]

Pinging @elastic/kibana-operations

[legal90]

I confirm that issue exists in Kibana 6.6.1. Putting a plain string (for a single host) doesn't work too, in any of these forms neither:

• "http://es1:9200,"
• "http://es1:9200"

I had to switch back to elasticsearch.url. Does that mean that multi-ES setup is not available in fact?

[incredincomp]

Multi ES is available. Currently running a three node cluster on centos7 with 6.6.1. That being said, it took a whole lot of messing to get them to speak to each other and I was forced to nominate them all as hosts+defaults. I have managed to make them work together by setting the kibana.yml ES host to the local instance of ES that is installed on the same IP as kibana.

[jbudz]

Thanks for the info - just to clarify, everyone seeing this is using helm charts?

[jbudz]

I believe this is upstream, https://github.com/helm/charts/issues/12378. I tested locally and things worked as expected. Closing this out for now, let me know if there's still issues.

https://github.com/elastic/helm-charts/tree/master/kibana lgtm too, if you want to give that a try.

[legal90]

@jbudz,

I tested locally and things worked as expected

Just wondering - did you test it using ELASTICSEARCH_HOSTS env variable only (as it's done in https://github.com/elastic/helm-charts/tree/master/kibana), or with kibana.yaml setting too?

It seems that issue exists only for the case with kibana.yaml. I run into it with the "upstream" helm chart, stable/elasticsearch.

[anapsix]

sorry about the noise
just tested directly with Docker Image docker.elastic.co/kibana/kibana-oss:6.6.0, and it appears fine.

# kibana.yml
---
server.host: "0"
server.name: kibana-test
kibana.index: ".kibana-test"
elasticsearch.hosts:
  - http://es1:9200
  - http://es2:9200
  - http://es3:9200

$ docker run \
   -it --rm \
   -v ${PWD}/kibana.yml:/usr/share/kibana/config/kibana.yml \
   -p 5601:5601 \
   docker.elastic.co/kibana/kibana-oss:6.6.0

  log   [10:18:04.465] [warning][plugin] Skipping non-plugin directory at /usr/share/kibana/src/legacy/core_plugins/ems_util
  log   [10:18:05.087] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [10:18:05.116] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [10:18:05.121] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [10:18:05.201] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [10:18:05.213] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [10:18:05.339] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [10:18:06.567] [info][status][plugin:[email protected]] Status changed from yellow to green - Ready
  log   [10:18:06.861] [info][migrations] Creating index .kibana-test_1.
  log   [10:18:07.097] [info][migrations] Pointing alias .kibana-test to .kibana-test_1.
  log   [10:18:07.568] [info][migrations] Finished in 707ms.
  log   [10:18:07.569] [info][listening] Server running at http://0:5601

[andrewschouten]

How do you specify the hosts in the docker-compose.yml file i've been trying something like this, but can't seem to get it to work?

  kibana:
    image: docker.elastic.co/kibana/kibana:6.7.1
    container_name: kibana
    ports:
      - 5601:5601
    environment:
      - ELASTICSEARCH_HOSTS=['http://example01.com:9200','http://example02.com:9200']

I've tried several different patterns, quotes, no quotes, etc but always seem to get this error:

FATAL ValidationError: child "elasticsearch" fails because [child "hosts" fails because [single value of "hosts" fails because ["hosts" must be a valid uri with a scheme matching the http|https pattern]]]

[anapsix]

@andrewschouten I suggest changing container instance command / entrypoint to something other than script that start the service (or update docker-compose.yml to mount path where config file is generated), and see how config file is generated

[smbambling]

@andrewschouten Did you ever figure out the correct syntax for this ? I'm also looking to figure this out

[andrewschouten]

I believe I ended up using the file kibana.yml and mounting it, it was a while back and I ended up having to rollback anyway as I was not able to get our prod servers upgraded to the version required.

[nikita-safonov]

@andrewschouten @smbambling just figure this out:

docker run \
           --name kibana \
           --publish 0.0.0.0:5601:5601 \
           -e ELASTICSEARCH_HOSTS='["http://example1.com:9200","http://example2.com:9200"]'  \
           -d \
           kibana:7.2.0

[nathan-zhu]

@nikita-safonov
hi, can use other ports to connect es cluster for ELASTICSEARCH_HOSTS in kibana?
because i use docker-compose on local machine, try to use different port to build es cluster, use port 9201, 9202. setting like below:

docker run \
           --name kibana \
           --publish 0.0.0.0:5601:5601 \
           -e ELASTICSEARCH_HOSTS='["http://example1.com:9201","http://example2.com:9202"]'  \
           -d \
           kibana:7.5.1

but i get error info lie this,

{"type":"error","@timestamp":"2020-01-11T06:01:59Z","tags":["warning","stats-collection"],"pid":6,"level":"error","error":{"message":"No Living connections","name":"Error","stack":"Error: No Living connectionsn at sendReqWithConnection (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:266:15)n at next (/usr/share/kibana/node_modules/elasticsearch/src/lib/connection_pool.js:243:7)n at process._tickCallback (internal/process/next_tick.js:61:11)"},"message":"No Living connections"}

[fartosh]

As of Kibana 7.6.0, config similar to this one

docker run
--name kibana
--publish 0.0.0.0:5601:5601
-e ELASTICSEARCH_HOSTS='["http://example1.com:9201","http://example2.com:9202"]'
-d
kibana:7.5.1

does not work. I use docker swarm and providing the following config:

kibana:
    image: docker.elastic.co/kibana/kibana:7.6.0
    environment:
      - ELASTICSEARCH_HOSTS='["http://elasticsearch01:9200","http://elasticsearch02:9200"]'
    ports:
      - 5601:5601
    networks:
      - elastic_network
    deploy:
      mode: global

ends up with:

[config validation of [elasticsearch].hosts.0]: expected URI with scheme [http|https] but got ['["http://elasticsearch01:9200","http://elasticsearch02:9200"]'].

[sanacl]

@fartosh for 6.8.6 I had the same issue and fixed it remove the quotes like this:

  kibana:
    image: docker.elastic.co/kibana/kibana:7.6.0
    environment:
      - ELASTICSEARCH_HOSTS=["http://elasticsearch01:9200","http://elasticsearch02:9200"]
    ports:
      - 5601:5601
    networks:
      - elastic_network
    deploy:
      mode: global

[mvpbang]

@andrewschouten @smbambling just figure this out:

docker run \
           --name kibana \
           --publish 0.0.0.0:5601:5601 \
           -e ELASTICSEARCH_HOSTS='["http://example1.com:9200","http://example2.com:9200"]'  \
           -d \
           kibana:7.2.0

the right to solve my problem!!!