Kibana: Kibana does not start when `kibana_system` user's password includes `%` character

[immon]

Kibana version: 7.9.0

Elasticsearch version: 7.9.0

Server OS version: Ubuntu 18.04

Original install method (e.g. download page, yum, from source, etc.): tar

Describe the bug:

Kibana does not start if password of kibana_system user includes percent character: %.

Steps to reproduce:

1. Enable xpack security in ES
2. change kibana_system password to changeme%
3. configure elasticsearch.username: "kibana_system" and elasticsearch.password: "changeme%" in kibana.yml
4. Kibana does not start with fatal error URIError: URI malformed. Full error below.

Expected behavior:

Accept all special characters for kibana_system password.

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

  log   [11:32:11.244] [fatal][root] URIError: URI malformed
    at decodeURIComponent (<anonymous>)
    at getUsernameAndPassword (/home/imo/Cases/tmp/kibana-7.9.0-linux-x86_64/node_modules/@elastic/elasticsearch/index.js:251:19)
    at getAuth (/home/imo/Cases/tmp/kibana-7.9.0-linux-x86_64/node_modules/@elastic/elasticsearch/index.js:224:20)
    at new Client (/home/imo/Cases/tmp/kibana-7.9.0-linux-x86_64/node_modules/@elastic/elasticsearch/index.js:59:23)
    at configureClient (/home/imo/Cases/tmp/kibana-7.9.0-linux-x86_64/src/core/server/elasticsearch/client/configure_client.js:37:18)
    at new ClusterClient (/home/imo/Cases/tmp/kibana-7.9.0-linux-x86_64/src/core/server/elasticsearch/client/cluster_client.js:40:65)
    at ElasticsearchService.createClusterClient (/home/imo/Cases/tmp/kibana-7.9.0-linux-x86_64/src/core/server/elasticsearch/elasticsearch_service.js:135:12)
    at ElasticsearchService.start (/home/imo/Cases/tmp/kibana-7.9.0-linux-x86_64/src/core/server/elasticsearch/elasticsearch_service.js:104:24)
    at process._tickCallback (internal/process/next_tick.js:68:7)
  log   [11:32:11.248] [info][plugins-system] Stopping all plugins.

 FATAL  URIError: URI malformed

Any additional context:

Similar to https://github.com/elastic/kibana/issues/66412

[przemeqq]

same here

[Skoetting]

For me the error occurred after the Upgrade from 7.8.1 to 7.9.
After changing the kibana_system Users password to one without % it works now.
I done the Upgrade with the deb package

[workingninja]

I can also confirm this. Removing the % from elasticsearch.password resolved the issue.

By the way, here is how I reset my Elasticsearch user (foo) password:

curl -u foo -XPUT 'https://localhost:9200/_xpack/security/user/foo/_password?pretty -H 'Content-Type: application/json' -d'
{
  "password": "changeme"
}
'

Source: https://discuss.elastic.co/t/i-lost-the-password-that-has-been-changed/91867

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[legrego]

@elastic/kibana-platform could this be related to the new ES Client? (cc @delvedor)

[delvedor]

Hello! How are you passing the password to the client?
Via URL (eg https://user:[email protected]) or via the auth configuration option?
If you are passing the username and password inside the URL, it will be decoded as you can see here.
The username and password are then sent via the authorization header and encoded here.

[legrego]

@delvedor it looks like we're using the auth configuration option for connections made via the kibana_system user:

https://github.com/elastic/kibana/blob/1ad68fdef7ca9711bf5a6e39eccbc18bfd7960e5/src/core/server/elasticsearch/client/client_config.ts#L89-L94

[pgayvallet]

@delvedor as @legrego said, we are using the auth option. Should we manually escape/encode the values?

[delvedor]

@pgayvallet I don't think encoding is the problem, the only thing the client is doing is base64 encoding the username and password for the basic authentication header.
As you can see from the snippet below, the string gets encoded and decoded correctly.

> Buffer.from('username:pas%world').toString('base64')
'dXNlcm5hbWU6cGFzJXdvcmxk'
> Buffer.from('dXNlcm5hbWU6cGFzJXdvcmxk', 'base64').toString()
'username:pas%world'

[pgayvallet]

I don't think encoding is the problem, the only thing the client is doing is base64 encoding the username and password for the basic authentication header.

Hum, Looking at the stack it's not just using b64, but decodeURIComponent. Stacktrace leads to here: https://github.com/elastic/elasticsearch-js/blob/a064f0f357ea5797cb8a784671b85a6b0c88626d/index.js#L278

And decoding a plain % causes an error:

> decodeURIComponent('pass%word')
VM94:1 Uncaught URIError: URI malformed
    at decodeURIComponent (<anonymous>)
    at <anonymous>:1:1

@delvedor Maybe the user/password are not properly encoded when injected from options.auth to the node struct or string?

[delvedor]

I fear I am missing something, didn't you said that you are using the auth option?
If you are passing the credentials via the URL, so protcol://username:password@host:port, then the credentials should be URL encoded, yes.

[rudolf]

Maybe it's because of https://github.com/elastic/kibana/blob/1ad68fdef7ca9711bf5a6e39eccbc18bfd7960e5/src/core/server/elasticsearch/legacy/elasticsearch_client_config.ts#L140-L142

[rudolf]

We were too late to get a fix into v7.9.3 so the fix will be released as part of v7.10.0