Kibana: Kibana 6.3 error writing settings

[PSiAU]

Kibana version:
6.3.0
Elasticsearch version:
6.3.0
Server OS version:
7.11
Browser version:
11.1.1
Browser OS version:
macOS 10.13.5
Original install method (e.g. download page, yum, from source, etc.):
From repository
Describe the bug:
Upon upgrading the ELK stack to 6.3, i'm seeing the following error at the top of Kibana:
Config: Request failed with status code: 403
I also don't seem to be able to get rid of the "Help us improve Elastic Stack......" message. I'm presuming it's trying to save some settings and is failing to do so. This problem only occured after the upgrade to 6.3.0.
kibana.stdout shows Kibana is getting a 403 when doing a POST to /api/kibana/settings
I do reverse proxy /kibana through Apache and use the server.basePath setting in kibana.yml accordingly, but everything was working fine up until 6.3.0.
Steps to reproduce:

1. Upgrade to ELK 6.3.0
2. Go to the Discover/Visualise/Dashboard panels in Kibana.

Expected behavior:
No error. Able to get rid of the "Help us improve Elastic Stack...." message.
Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:

[PSiAU]

Fixed the issue. For some reason, the setting "index.blocks.read_only_allow_delete" on the Kibana index became set to "true". I changed it to "false" and things started working properly again.
Could anyone shed some light as to what might have happened?

[bhavyarm]

@jbudz thanks?

[tylersmalley]

@PSiAU, you should have received a message with instructions when bringing up the UI. Was that not the case?

[PSiAU]

No I didnt see any instructions. Mind you I do use an adblocker (AdGuard) so that might have hidden it maybe.

[marius-dr]

@tylersmalley there is no message about this at all when you get the error. Reproduced it today on 6.3.2 like this:

1. Set up Kibana to be running.
2. Fill the disk in order to trigger the flood watermark in ES (95% usage or more). This will set all indices as read-only.
3. Any operation that tries to write to .kibana after this will get a 403 error, but no instructions on how to fix it.

The screenshot below is from when I tried to create an index pattern after the index was set to read only.

[Fak3]

i can confirm this on kibana and es 6.3.1.
After i ran into disk shortage, i got log message from ES that indices are read-only. So i freed up some space, and then i did set cluster.routing.allocation.disk.threshold_enabled=false ENV var for elasticsearch docker instance. This allowed me to write into my indices again, but creating new kibana index pattern fails.
Also after first time i restarted kibana, and unlike the screenshot seen in the above comment from @marius-dr, i have only seen yellow message line in the kibana, and not red one (no 403 error). But the index pattern failed to be created. After second restart of kibana and ES - i started to get those same red messages about 403 errors.

[Fak3]

Unlike @PSiAU wrote in the first comment of this thread - "index.blocks.read_only_allow_delete" setetting of .kibana index was set to "false"

[Fak3]

what helped me to workaround this - is to remove .kibana index comlpetely and restart the cluster and kibana

[tvollscw]

Same issue occurring for me on Kibana and Elasticsearch 6.4.0.
@Fak3 's previous comment helped fix the issue.
Only difference between my issue and the ones stated above is that my install was at 94% usage

[gregwjacobs]

Ran into this also with 6.4, however you can recover also via this:

PUT */_settings
{
  "index": {
  "blocks": {
  "read_only_allow_delete": "false"
  }
 }
}

```

[maristitovs]

Fixed the issue. For some reason, the setting "index.blocks.read_only_allow_delete" on the Kibana index became set to "true". I changed it to "false" and things started working properly again.
Could anyone shed some light as to what might have happened?

I had an issue on ELK stack 6.4.0 in which I couldn't create a new index pattern. It seemed to have started at some random time, since I was able to create a few before.

Thanks to this comment, I checked the settings on my .kibana indice, and sure enough the index.blocks.read_only_allow_delete was set to true. Setting it back to false (as it should be by default?) fixed the issue.

[kobelb]

This just happened in discuss where a user was unable to create/edit an index-pattern and the situation isn't surfaced in an easy to digest manner: https://discuss.elastic.co/t/unable-to-create-an-index-pattern-pattern-creation-hanging-up-or-failing/178687/7

Is there any potential to put Kibana into a "yellow" state when we detect this on the .kibana index?