Kibana: HTTP 302 loop - ERR_TOO_MANY_REDIRECTS

[nellicus]

Kibana version:
5.2.0

Elasticsearch version:
N/A

Server OS version:
Debian

Browser version:
curl

Browser OS version:
Debian 7

Original install method (e.g. download page, yum, from source, etc.):
tar.gz

Description of the problem including expected versus actual behavior:
Kibana expose 302 loop when ES is unreachable

Steps to reproduce:
configure Kibana to connect to imaginary cluster as per default docs

Errors in browser console (if relevant):
ERR_TOO_MANY_REDIRECTS

Provide logs and/or server output (if relevant):

spons [10:15:17.698]  GET / 302 4ms - 9.0B
  log   [10:15:18.048] [warning][admin][elasticsearch] Unable to revive connection: https://w530:9200/
  log   [10:15:18.049] [warning][admin][elasticsearch] No living connections
respons [10:15:18.606]  GET / 302 7ms - 9.0B
respons [10:15:18.614]  GET /login?next=%2F 302 2ms - 9.0B
respons [10:15:18.626]  GET / 302 6ms - 9.0B
respons [10:15:18.648]  GET /login?next=%2F 302 2ms - 9.0B
respons [10:15:18.660]  GET / 302 5ms - 9.0B
respons [10:15:18.667]  GET /login?next=%2F 302 2ms - 9.0B
respons [10:15:18.681]  GET / 302 5ms - 9.0B
respons [10:15:18.689]  GET /login?next=%2F 302 2ms - 9.0B
respons [10:15:18.704]  GET / 302 4ms - 9.0B
respons [10:15:18.712]  GET /login?next=%2F 302 1ms - 9.0B
respons [10:15:18.723]  GET / 302 5ms - 9.0B
respons [10:15:18.732]  GET /login?next=%2F 302 2ms - 9.0B
respons [10:15:18.751]  GET / 302 4ms - 9.0B
respons [10:15:18.763]  GET /login?next=%2F 302 1ms - 9.0B
respons [10:15:18.787]  GET / 302 5ms - 9.0B
respons [10:15:18.801]  GET /login?next=%2F 302 1ms - 9.0B
respons [10:15:18.819]  GET / 302 2ms - 9.0B
respons [10:15:18.828]  GET /login?next=%2F 302 1ms - 9.0B
respons [10:15:18.846]  GET / 302 5ms - 9.0B
respons [10:15:18.850]  GET / 302 4ms - 9.0B
respons [10:15:18.858]  GET /login?next=%2F 302 1ms - 9.0B
respons [10:15:18.873]  GET / 302 4ms - 9.0B
  log   [10:15:19.455] [warning][elasticsearch][monitoring-ui] Unable to revive connection: https://w530:9200/
  log   [10:15:19.456] [warning][elasticsearch][monitoring-ui] No living connections
  ops   [10:15:19.638]  memory: 96.6MB uptime: 0:02:09 load: [2.17 1.45 0.67] delay: 0.566

$ time curl https://w530:5601  -k  -L  --max-redirs 10000
curl: (47) Maximum (10000) redirects followed

real    0m13.535s
user    0m0.608s
sys 0m0.252s

user@w530 $ /opt/elk/prod/version/5.2.0/kibana-5.2.0-linux-x86_64/bin/kibana-plugin list
[email protected]

user@w530 $ egrep '^[^#]+' /opt/elk/prod/common/kibana-conf/kibana.yml
server.port: 5601
server.host: "w530"
server.name: "tonybana"
elasticsearch.url: "https://w530:9200"
elasticsearch.username: "kibana"
elasticsearch.password: "XXXXXX"
server.ssl.cert: /opt/elk/prod/common/kibana-conf/kibana-ssl/kibana.crt.pem
server.ssl.key: /opt/elk/prod/common/kibana-conf/kibana-ssl/kibana.key.pem
elasticsearch.ssl.cert: /opt/elk/prod/common/ssl/certgen_output/w530/w530.crt
elasticsearch.ssl.key: /opt/elk/prod/common/ssl/certgen_output/w530/w530.key
elasticsearch.ssl.ca: /opt/elk/prod/common/ssl/certgen_output/ca/ca.crt
elasticsearch.ssl.verify: true
logging.verbose: true
xpack.security.encryptionKey: "dsiajdiojoijeiojewiofhqhfuierhqguihreiupfhripfqhewpfhiowehqfpuhgpwuqihfuhat_least_32_characters"
xpack.reporting.encryptionKey: "dfkfkdsjfkldjsklfjdsfjioasjfiodsjfio;dsjf;idsjfo;idjsoi;fjdsio;fjdsa;iofhds;oiahf;iashdi;ofhsd;iah"
console.proxyConfig:
  - match:
      host: "*"
      port: "{9200..9202}"
    ssl:
      ca: "/opt/elk/prod/common/ssl/certgen_output/ca/ca.crt"

[junoteam]

Hello,
I have the same issue exactly.
kibana doing redirect loop with status code 302.
And this is happening after I change user's password for: elastic and kibana users trough the web interface of kibana.


199.203.151.53 - - [16/Feb/2017:15:36:40 +0000] "GET /login?next=%2F HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" "-"
199.203.151.53 - - [16/Feb/2017:15:36:40 +0000] "GET / HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" "-"
199.203.151.53 - - [16/Feb/2017:15:36:40 +0000] "GET /login?next=%2F HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" "-"
199.203.151.53 - - [16/Feb/2017:15:36:41 +0000] "GET / HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" "-"
199.203.151.53 - - [16/Feb/2017:15:36:41 +0000] "GET /login?next=%2F HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" "-"
199.203.151.53 - - [16/Feb/2017:15:36:41 +0000] "GET / HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" "-"
199.203.151.53 - - [16/Feb/2017:15:36:41 +0000] "GET /login?next=%2F HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" "-"
199.203.151.53 - - [16/Feb/2017:15:36:42 +0000] "GET / HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" "-"
199.203.151.53 - - [16/Feb/2017:15:36:42 +0000] "GET /login?next=%2F HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" "-"


[armandoltx]

After installing x-pack I changed the password of an existing user, "Kibana User" and since I did it, I cannot access to my app. When I uninstall the X-Pack I can access as always, but reinstall the plugin and the problem came back.

[luckydonald]

Can confirm, that forward loop started after changing the default password for elastic in the kibana web gui.
kibana.example.com redirected you too many times.

Running in docker, behind a reverse proxy.

$ docker version
Server:
 Version:      1.13.0
 API version:  1.25
 Go version:   go1.7.3
 Git commit:   49bf474
 Built:        Tue Jan 17 09:58:26 2017
 OS/Arch:      linux/amd64

Image: docker.elastic.co/kibana/kibana:5.3.0

[luckydonald]

In fact, changing the passwords back to changeme is a workaround.

I just executed:

POST _xpack/security/user/elastic/_password
{
  "password": "changeme"
}

[lukasolson]

Can you check your cookies to see if you have multiple cookies for the domain?

[m4rkw]

I had the same issue with Kibana 5.3.0. It seems if you change the default passwords you need to give kibana a password it can use to talk to the ES cluster in kibana.yml:

elasticsearch.username: "kibana"
elasticsearch.password: "pass"

after doing this it seems to work. not sure why it can't just use the credentials provided in the web interface when you log in though. maybe someone can clarify?

[lukasolson]

Usually what causes this is having multiple cookies for the same "domain" and "name", but with different values for "path". If you open the developer tools in Chrome, then click on the Application tab, then expand the Cookies section, and click on the domain, do you have multiple cookies with the name "sid"? If so, you can fix this issue by clearing all of them.

[luckydonald]

@lukasolson After kibana didn't work any more, I openend it in a new browser (Safari) where I never used this instance of kibana before. Got the same issue there too.
I had no time to redo it while monitoring my cookies.
@m4rkw yeah, which is strange. After using kibanas "change password" field, I expected same kibana to pick up the entered changes.

[andrewmclagan]

Also seeing this on 5.1.1. Cleared browser cache, used different browsers

[chrissound]

Same issue here.

[Panblack]

{"type":"log","@timestamp":"2017-05-27T09:45:56Z","tags":["warning","elasticsearch","admin"],"pid":19343,"message":"No living connections"}

For me, this error caused 302 loop.
Edit /config/kibana.yml and fix elasticsearch.url: ,the problem solved.

[chrissound]

If it's helpful at all, this issue occurred to me when using 2 elastic search nodes. When I just had 1 the error stopped.

Here is the working docker-compose.yaml I ended up with:

version: '2'
services:
  kibanna:
    image: docker.elastic.co/kibana/kibana:5.4.0
    ports:
      - 5601:5601
    networks:
      - esnet
    environment:
       ELASTICSEARCH_URL: http://elasticsearch1
  elasticsearch1:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.4.0
    container_name: elasticsearch1
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
    volumes:
      - esdata1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet
volumes:
  esdata1:
    driver: local

networks:
  esnet:

[caperea]

one seemingly workaround is using http://domain:port/login?next=/login

[shijij]

This bug appeared when I add node.local: true to the config file and restart ES.
Then even when I remove that and restart won't solve the problem.
Have to clear all cookies and goto /login?next=/login to end the loop. as @caperea mentioned

[chrowe]

I am having this issue as well. I use Chrome usually and have been testing with Incognito mode to avoid issues with cookies but that does not resolve the issue.

I am using version 5.4.1 of everything.

@lukasolson just to be sure, I tried your suggestion and see no cookies.

[dili91]

same as @chrowe

[chrowe]

My issue was solved by https://discuss.elastic.co/t/cant-access-kibana-after-changing-password/90701/2

This is the same solution as https://github.com/elastic/kibana/issues/10335#issuecomment-297319742

[chrowe]

Has something changed in 5.4.3 that would fix this?

After https://github.com/chrowe/docker-elk/commit/8bf9c62f6c292c0cb8a2cef703b2e200978408c1 I was able to just change the password in the Kibana GUI and did not have this issue.

[nellicus]

closing as per the above

[luckydonald]

@nellicus which do you mean?

[jbrant]

This is still an issue in elasticsearch 5.5.1.

[aclowkey]

6.7 - still happens!

[c4milo]

We manually patched Kibana 5.5.3 to fix this problem, here is the patch just in case anyone else find it useful:

diff --git a/src/server/config/schema.js b/src/server/config/schema.js
index e49033f4..f23d195b 100644
--- a/src/server/config/schema.js
+++ b/src/server/config/schema.js
@@ -59,7 +59,9 @@ module.exports = () => _joi2.default.object({
     uuid: _joi2.default.string().guid().default(),
     name: _joi2.default.string().default(_os2.default.hostname()),
     host: _joi2.default.string().hostname().default('localhost'),
+    address: _joi2.default.string().hostname().default('localhost'),
     port: _joi2.default.number().default(5601),
+    uri: _joi2.default.string(),
     maxPayloadBytes: _joi2.default.number().default(1048576),
     autoListen: _joi2.default.boolean().default(true),
     defaultRoute: _joi2.default.string().default('/app/kibana').regex(/^\//, `start with a slash`),
diff --git a/src/server/http/setup_connection.js b/src/server/http/setup_connection.js
index cbb522f6..2bc98e83 100644
--- a/src/server/http/setup_connection.js
+++ b/src/server/http/setup_connection.js
@@ -10,10 +10,14 @@ exports.default = function (kbnServer, server, config) {
   // this mixin is used outside of the kbn server, so it MUST work without a full kbnServer object.
   kbnServer = null;

+  const address = config.get('server.address');
+  const uri = config.get('server.uri');
   const host = config.get('server.host');
   const port = config.get('server.port');

   const connectionOptions = {
+    address,
+    uri,
     host,
     port,
     state: {

And, In kibana.yml we configure those 2 parameters like this:

# Tell kibana to bind to all network addresses
server.address: "0.0.0.0"
server.host: "mykibana.com"
server.port: 5601
# This is what gets used to build redirect URIs
server.uri: "https://mykibana.com"

[chrissound]

This issue appeared when I deleted all the elasticsearch indexes with curl -XDELETE "http://127.0.0.1:9200/_all".