Kibana: Kibana sometimes sends HTTP requests to Elasticsearch without credentials

[sscarduzio]

Kibana version: 5.1.1

Elasticsearch version: 5.1.1

Server OS version:OSX 10.11.6

Browser version: Chrome 54

Browser OS version: OSX 10.11.6

Original install method (e.g. download page, yum, from source, etc.): unzipped the tarball

Description of the problem including expected versus actual behavior: Kibana server sends some HTTP request without authorization header.

Steps to reproduce:

1. Have an Elasticsearch v5.1.1 instance running allowing only requests with HTTP Basic Auth credentials.
2. Configure said credentials in kibana.yml as "elasticsearch.username" and "elasticsearch.password".
3. Wait for ES to reach green status, and Kibana to find Elasticsearch.
4. Create some dummy data points

curl -X POST -H "Content-Type: application/json" -H "Authorization: Basic bG9nc3Rhc2g6bG9nc3Rhc2g=" -H "Cache-Control: no-cache"  -d '{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:28.998Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.267Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.267Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.267Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.268Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.268Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.268Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.268Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.269Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.269Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.269Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.269Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.269Z","host":"hilbert.local"}' "http://localhost:9200/_bulk"

1. Switch on Wireshark.
2. Start exploring and searching with Kibana
3. Observe in wireshark that Kibana skips sending the credentials for some requests, like in this screenshot:

See how Authorization header is not present!**

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

 error  [09:12:03.308]  Error: Authentication Exception
    at respond (/me/tmp/kibana-5.1.1-darwin-x86_64/node_modules/elasticsearch/src/lib/transport.js:289:15)
    at checkRespForFailure (/me/tmp/kibana-5.1.1-darwin-x86_64/node_modules/elasticsearch/src/lib/transport.js:248:7)
    at HttpConnector.<anonymous> (/me/tmp/kibana-5.1.1-darwin-x86_64/node_modules/elasticsearch/src/lib/connectors/http.js:164:7)
    at IncomingMessage.wrapper (/me/tmp/kibana-5.1.1-darwin-x86_64/node_modules/elasticsearch/node_modules/lodash/lodash.js:4994:19)
    at emitNone (events.js:91:20)
    at IncomingMessage.emit (events.js:185:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickDomainCallback (internal/process/next_tick.js:122:9)

[jbudz]

The .kibana/config requests without auth will be fixed by https://github.com/elastic/kibana/issues/9442 in the next release. Are you seeing this for more endpoints? Looking into it too.

[sscarduzio]

This is the set of actions I had to explicitly allow to be requested without authentication (for any index) to make Kibana work

      actions: ["indices:data/read/field_stats", "indices:admin/mappings/fields/get", "indices:admin/get", "indices:data/read/msearch"]

[sscarduzio]

@jbudz if you have Kibana builds to make me test, I have all set up on the ES side to test and I'm more than happy to help out!

[jbudz]

@sscarduzio awesome, thanks. We have snapshots here.

[sscarduzio]

Nice, is there already something I can test about this bug in the 5.1.2 snapshot?

[jbudz]

Yeah, https://github.com/elastic/kibana/pull/9446 should fix the auth requests to .kibana/config

[sscarduzio]

Hello! I noticed this is labelled as "neeeds details", do you need more details?

[epixa]

These endpoints were fixed in 5.1.2.

[sscarduzio]

@epixa Kibana with x-pack installed has this problem again, cannot load the /app/monitoring (Marvel).

I see a POST /.monitoring-data-2/_count (with no body) coming in without valid Authorization header.

[sscarduzio]

Oh and when I click on Timelion, I see GET /.kibana/config/5.2.0, again without valid Authorization. Although this is not fatal to the browser's user experience.

[epixa]

@sscarduzio Can you give some steps to reproduce the issue you're having with monitoring? Monitoring is loading without issue for me, and requests are properly authorized.

There are circumstances where Kibana will attempt to query for the config prior to authorization being available, and in those cases it will gracefully handle the authorization error that is returned from Elasticsearch, which is probably what you saw in the latter case.

[sscarduzio]

Sure, here is a step by step how to reproduce this with screenshots.

1. Download fresh ES and run it as is.
2. Download Kibana, install xpack plugin an use this configuration:

xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.graph.enabled: false
xpack.reporting.enabled: false
elasticsearch.username: "kibana"
elasticsearch.password: "kibanapassword"

1. Fire up wireshark
2. run Kibana, navigate to monitoring app
3. Search Wireshark for string ".monitoring-data-2/_count"

4. Follow HTTP traffic in Wireshark, note absence of Authorization header in the POST request.
   

5. Take another sample of the many other kibana HTTP connections you see in the same capture in Wireshark and see the presence of Authorization headers.

[gitime]

@epixa the issue still exists with elasticstack 5.2.0, why closed?

[gitime]

@sscarduzio why the elastic team taking so much time to resolve and fix this important issue?

[epixa]

@sscarduzio To clarify, the requests that you listed in your last comment that do not include an authorization header do get rejected by Elasticsearch as unauthorized, correct?

[epixa]

@gitime We're still trying to figure out what the issue even is. We're having no authorization issues in any of our testing, neither in our manual testing nor in our automated testing. As a Kibana end-user, how is this issue affecting you?

[epixa]

I think I've reproduced the issue, but I'd appreciate it if either of you could confirm on your end.

When security is enabled on Elasticsearch and disabled on Kibana, Monitoring fails to work due to access denied errors by default. The search profiler (via dev tools) also fails (as do reporting requests behind the scenes, but that's disabled for this reproduction). In both cases, the authorization headers are not being sent, which is causing the failed requests. If you go to discover or something like that, the requests work and have the authorization header.

Now this is where it gets really strange: if you go to discover and then generate a short URL via the "share" menu, you'll see that that request does properly have an authorization header, and the short url is created. Now, if you go to search profiler and execute a search, the authorization header is sent and the request works this time. Also at this time, if you manually navigate to /app/monitoring (to make sure you're bypassing the access denied page), monitoring works and all the requests are sending the authorization header.

Edit: And taking that one step further, once the authorization header starts sending for xpack requests, even refreshing the page doesn't stop the requests from working. Starting a new incognito window will get us back into a failing state though.

[sscarduzio]

@epixa I really appreciate your time for digging into this, and man, I know browsers' "magic" in handling HTTP basic auth is a source of major confusion while dealing with this kind of issues.
:)

I just reproduced the additional instance of unauthenticated request when you click on dev tools:

GET /_aliases
GET /_mapping

The way I've been detecting these issues

In hope it will be useful, I can share my initial testing setup (before inspecting with Wireshark). I believe it offers a great framework for pin-pointing rogue unauthenticated requests.

My original test involved installing readonlyrest plugin and configuring it to accept only requests carrying HTTP basic auth credentials (user=kibana, pass=kibana).
This would be my elasticsearch.yml

xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.graph.enabled: false
xpack.watcher.enabled: false
readonlyrest:
  enable: true
  access_control_rules:
    - name: Kibana
      auth_key: kibana:kibanapassword
      type: allow

I then set up Kibana to use those credentials too

xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.graph.enabled: false
xpack.reporting.enabled: false
elasticsearch.username: "kibana"
elasticsearch.password: "kibanapassword"

Then I opened my browser and gave the same credentials from the browser too.

At this stage I knew:

• if the browser fails to send credentials, I'll see a login prompt (after receiving a HTTP 401).
• If Kibana daemon fails to send credentials, I'd see an HTML error banner in the browser and/or a red line in the readonlyrest logs stating the request type, action, involved indices, headers and path.

Tell me if I can help any further.

_Simone

[epixa]

@sscarduzio Thanks for all of that info and your time so far in debugging this. We've figured it out. The problem is that certain requests in Kibana and X-Pack are not returning a 401 when Elasticsearch does. Browsers rely on that 401 error for their basic auth process, even when basic auth has already been completed once on that domain. The thing is, once basic auth credentials for a given realm are stored in the browser, the browser will automatically resend newly-authorized requests that 401 with the same realm that is stored, but browser dev tools merge these requests together and treat them as a single thing. This is convenient in most cases, but it made it particularly hard to track down an issue like this without a lower level tool like wireshark.

One thing that's worth mentioning again is that this is not a security vulnerability since the requests are being rejected as they should be, but it is definitely not the behavior we want.

To help illustrate the problem for anyone that still doesn't follow, let's talk about what is happening at the request level in the Really Strange™ behavior I described in a previous comment.

1. The initial request to /app/kibana 401s with the realm "security", and no stored credentials exist for that realm, so it prompts for a username and password, which we input. This 401 error appears in the dev console because it was specifically prompted for input. The browser will now send the authorization header on all requests to /app/*, since /app/ is the base path of this request that was just authenticated, which is why you can navigate between the various applications in Kibana without getting 401 errors.
2. Requests to monitoring result in 403 errors instead of 401 errors so the browser does not attempt to authorize them at all.
3. Attempting to use the search profiler results in a 200 request with the error information embedded into the response body, so the browser doesn't attempt to authorize it.
4. The short URL service makes a request to /shorten, which 401s with the realm "security". Since credentials for this realm were already stored, the same request is repeated with the authorization header. In the browser dev tools, this appears as a single successful request and includes the authorization headers, but a lower level tool like wireshark will show otherwise. Like in the first step, the browser will now send the authorization header to any request matching the base path of this request, which just happens to be / for this particular endpoint. This effectively means that all requests to Kibana will now include the authorization header by default, which is why everything begins to work at that point onward.

The solution here is to go through all authenticated endpoints in Kibana and X-Pack to make sure they are properly returning 401 errors for missing credentials. We're tracking this effort in an internal ticket, but I'm going to open this one so we have a public place to update on status and to which we can link pull requests.

[epixa]

/cc @spalger @ycombinator

[nan008]

@epixa any news on this problem?

[epixa]

@nan008 No news at this point other than that we're still auditing all the possible endpoints. The effort stalled a little due to Elastic{ON}, but we'll get back into it soon.

[changi67]

@epixa any news ?

[epixa]

@changi67 Unfortunately, I have no news here. I was hoping to get back into it after Elastic{ON}, but we got derailed by a couple of other projects as we march toward the last 5.x feature release and 6.0. I'm sorry for setting the wrong expectations before.

We do still intend to fix this, but it's just not on deck just yet.

[neuroticnetworks]

@epixa you mentioned an internal document where you're tracking these problems. Any chance you can share your notes so folks out here can start hacking at it?

[neuroticnetworks]

following up on

The short URL service makes a request to /shorten, which 401s with the realm "security". Since credentials for this realm were already stored, the same request is repeated with the authorization header. In the browser dev tools, this appears as a single successful request and includes the authorization headers, but a lower level tool like wireshark will show otherwise. Like in the first step, the browser will now send the authorization header to any request matching the base path of this request, which just happens to be / for this particular endpoint. This effectively means that all requests to Kibana will now include the authorization header by default, which is why everything begins to work at that point onward.

Would it be possible to throw an API call behind the scenes to /shorten right after a user sucessfully logs in at /app/kibana?

It's hacky, but this is kind of an annoying bug that renders the product unusable in mysterious ways. I'm not a Node dev but I'll carve out some time to learn it and help you guys out if I need to. It would be helpful if you guys could share the notes you have. Looks like you were going to do that earlier? Maybe you already have? Or were you referring to this Github issue?

[epixa]

@evanv These requests are primarily to xpack, so there is no open source code for people to contribute to in this case.

[epixa]

Would it be possible to throw an API call behind the scenes to /shorten right after a user sucessfully logs in at /app/kibana?

That's an interesting idea. I don't think abusing the /shorten endpoint is the right approach here, but if this is something that could be solved for all possible endpoints simply by adding a new top level endpoint that we need to hit when loading Kibana, that would be a potentially good solution from a practical perspective. If you throw together a quick proof of concept, I'd be happy to take a look!

[neuroticnetworks]

I can confirm similar odd behavior in Kibana pointed at Nginx fronted ES that isn't running any X-Pack services., fwiw.

Specifically I've noticed issues around short urls (eg https://github.com/elastic/kibana/issues/10456) and I'm also running into issues where users are randomly prompted for logins again. These random prompts do appear to stop happening after folks generate a short url, but I don't have enough data there to be confident it will solve things.

I'm with you on not abusing the /shorten endpoint but maybe replacing it with some top level one that gets hit behind the scenes, at least as a workaround. I'm trying to figure out the codebase and your contributing guide. Hopefully I'll have something in the next week though.

[drakkhen]

+1

[immustafa]

+1

[brettahale]

+1

[aliostad]

+1

[anandeur]

+1

[olddanmer]

+1

[aellisv]

+1

[MarcusCaepio]

Anything new here?
Have the same experience with readonlyrest

[askids]

Wow. Nearing 300 days and still open :+1:

[neuroticnetworks]

@battleroid did you and/or @lusis and/or Sam figure out a workaround for this? If so, any chance you can explain what you did?

It feels to me like abusing the /shorten endpoint might be preferable at this point. On the one hand it's gross but it's arguably much less gross that allowing Really Strange™ behavior that isn't understood continue to cause a bad experience for end users

[Battleroid]

@evanv We ended up creating a service account with limited read access and forward the credentials for it on the paths that were giving us trouble. It's not ideal, but it does work and it's only a single additional location block within nginx to fix the issue.

[danielkasen]

this is still occuring with kibana 6.1 btw. I stumbled onto it in my audit logs trying to hit
/_xpack/security/_authenticate and triggering anonymous_access_denied errors. Stopping kibana and those go away.

[amitripshtos]

Having that issue always for _field_caps for kibana 6.0.1

[epixa]

This is a much more obvious problem now in 6.x because of the changes we've made to introduce field_caps. We can probably fix that specific endpoint to return a 401 properly, but we really need to introduce a more holistic fix to this so that we're not just chasing down bugs like this forever.

[amitripshtos]

How can we push that issue? I really want to help but I don't quite understand how configured $http in angular can decide if to send / not send creds

[gobengo]

@Battleroid will you share more about the nginx conf that works for you, so I can do the same?

it's only a single additional location block within nginx to fix the issue.

[epixa]

@amitripshtos Without X-Pack, Kibana itself does not handle adding authorization headers - we rely entirely on the browser's native basic auth mechanism for this. So fixing this issue requires an understanding of how that basic auth mechanism works in relation to Kibana.

Further up on this issue, I summarized why this issue was occurring in relation to the browser's basic auth mechanism: https://github.com/elastic/kibana/issues/9583#issuecomment-280338002

And a possible solution to this might be Kibana hitting a new top level endpoint that pings ES and responds with a 401 if ES does, as mentioned here: https://github.com/elastic/kibana/issues/9583#issuecomment-301174001

Alternatively, all endpoints everywhere in Kibana could return a 401 when ES does, but this is becoming less likely as endpoints on the Kibana server continue to diverge from corresponding endpoints in Elasticsearch, and I suspect we'll end up chasing down errors like this in perpetuity if that's the approach we take now.

[Mythra]

I am unable to even setup an index pattern to start looking at data in Kibana 6.1. Due to the fact that when I do I get a 401 on field_caps. The fact that I can't use Kibana for anything on this newly setup cluster without creating a completely separate user, and essentially making any request without auth headers be this shared user (which seems to be what others have done) is mind boggling to me.

Now I'm not seeing any way besides creating a shared user to fix this on my end. However, I wanna believe deep down a company like elastic wouldn't allow kibana 6.1 to go out without being able to perform such a basic function of their software. So is there perhaps a workaround for Kibana 6.1 that I'm missing, that I can perform in order to actually use the software besides looking at a whole bunch of screens with no data?

[Mythra]

To clarify I was able to get around this by installing a browser plugin to manually add the authorization header to every request that did not contain it. This kind of makes me sad that I had to do this to even get a working kibana, and hopefully there's just something I'm missing.

[Battleroid]

@gobengo

We created a basic user with limited permissions on the .kibana index, _mget, _msearch, */_field_stats paths and have a nginx configuration similar to the following:

location ~ ^/(goto|es_admin/.kibana|es_admin/_mget|elasticsearch/_msearch|elasticsearch/.*/_field_stats.*)/? {
    proxy_set_header Authorization "Basic <base64 encoded credentials>";
    proxy_pass http://whatever_upstream_for_kibana;
}

This let's us get past the initial issue, after which the user's credentials are required.

[thomasriley]

@epixa Hey, any idea when the fix that you mentioned so that field_caps returns a 401 correctly is due? Kibana/Elastic Search 6.x is near impossible to use without some sort of Nginx proxy as described above when relying on Basic Auth (with no third party auth plugins or X-Pack). Thanks.

[didierb78]

Same thing here. This is a critical issue.

[euprogramador]

Idem, receive same error

[euprogramador]

works,

added readonlyrest one rule:

• name: "public access monitoring"
  actions: ["indices:data/read/search"]
  indices: [".monitoring*"]

Kibana require auth, monitoring (x-pack), works.

wow!

[burandobata]

+1

[kobelb]

We're actively looking into a solution to this problem that will prevent us from chasing down end-points that aren't handling the WWW-Authenticate headers properly on a 401. Thanks for bringing this to our attention, and your patience. I'll update this issue as we know more.

[julienyu]

+1

[nsbk]

We just stumbled upon this. So far we are using the elasticsearch.customHeaders field on kibana.yml to make sure that auth is always present on the request.

elasticsearch.customHeaders: { Authorization: Basic base64auth }

Being base64auth base64 encoded username:password

[kobelb]

https://github.com/elastic/kibana/pull/17725 just merged, so the index patterns api will now be handling the WWW-Authenticate header properly on 401s.

A more holistic approach is explored in this POC PR: https://github.com/elastic/kibana/pull/17724

[parallelthought]

This behaviour is observed in Elastic v6.4.2 as well (especially for calls made to update monitoring index by x-pack plugin). The impact is more for use cases where an OSS plugin is used for security on top of ES and basic auth needs is enabled for Kibana.

[ygersie]

I'm using the following hack with nginx as a proxy in front of Kibana to trick the browser to store credentials for the entire hostname:

    location = / {
      # Hack to workaround https://github.com/elastic/kibana/issues/9583
      if ($http_authorization !~* "^Basic") {
        rewrite .* /app/kibana break;
      }
      proxy_pass http://localhost:5601;
    }

    location / {
      if ($http_authorization !~* "^Basic") {
        return 302 https://$host;
      }
      proxy_pass http://localhost:5601;
    }

If a client does not send an Authorization header I force a proxy_pass towards /app/kibana which will prompt for auth. After the credentials are stored with the correct uri (/) all subsequent requests will contain the authorization header.

[sscarduzio]

News about this
https://forum.readonlyrest.com/t/provide-kibana-login-page-for-ror-oss-version/1441