Elasticsearch: Some cluster settings don't take effect if set from elasticsearch.yml, but work fine if set via API

[gwbrown]

Some cluster-level settings don't take effect unless they're set via the Cluster Settings API, and values specified in elasticsearch.yml are ignored. They'll show up in GET _cluster/settings?include_defaults=true but will be otherwise ignored.

This is caused by reading settings only from the cluster state - in order to be initialized in elasticsearch.yml, a variable needs to be set in memory and initialized from a different Settings object during startup and updated in a cluster state listener rather than being read directly from the cluster state.

So far, I've found these settings which don't respect values set in elasticsearch.yml:
cluster.blocks.read_only
~cluster.max_shards_per_node~ fixed in https://github.com/elastic/elasticsearch/pull/57234.

/cc @joegallo who pointed this out

[elasticmachine]

Pinging @elastic/es-core-infra

[dmitry-ee]

Can confirm bug.
Setting via docker environment -e "cluster.max_shards_per_node=3000" is not working.
Wasted 3hrs for understanding where that 1000 shards limit actually is.
Hopefully found that bug.

Part of exception:
Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1366]/[1000] maximum shards open

Current solution:
curl -X PUT localhost:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent": { "cluster.max_shards_per_node": "3000" } }'

[Pupkur]

.opendistro_security index does not exists, attempt to create it ... ERR: An unexpected IllegalArgumentException occured: Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [2670]/[1000] maximum shards open;
Trace:
java.lang.IllegalArgumentException: Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [2670]/[1000] maximum shards open;

And when I`m trying:

curl -k -X PUT https://SERVER_IP:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent": { "cluster.max_shards_per_node": "3000" } }'

Got error:
Open Distro Security not initialized.

[gwbrown]

@Pupkur That looks like an issue with OpenDistro, which we don't maintain and can't provide support for. We do our best to answer questions about our default distribution and core Elasticsearch functionality on our forums and address bugs here on GitHub, but for third-party plugins you'll need to get support from either the creator of the plugin or a community with knowledge about that plugin. Let's keep the discussion on this issue focused on the bug described in the first post.

[Pupkur]

Oops. Sorry. I hope this bug does not exist in elasticsearch.
Good day for all!

[gperrego]

We are having the same issue, we can work around this using the API but thats an issue. We are on 7.7. Is there a specific section of the yml that this should be in? When placing this setting in the yml we were not sure where to place it. I don't think that should matter but wanted to check.

[gwbrown]

The cluster.max_shards_per_node issue has been fixed in https://github.com/elastic/elasticsearch/pull/57234. Because cluster.blocks.read_only is highly unlikely to be set via the YAML config, I'm going to close this issue. We can reopen if we decided we want to address that in the future.

[M9k]

If someone work in Kibana curl:
PUT /_cluster/settings { "persistent": { "cluster.max_shards_per_node": "5000" } }

[DaveCTurner]

Because cluster.blocks.read_only is highly unlikely to be set via the YAML config

@gwbrown I don't think this is the right reasoning to use here. I think we shouldn't silently ignore settings like this, regardless of how likely they are to be used.

In this case, you might for instance expect to be able to use this setting to bring up a cluster in a purely read-only mode -- perhaps you want to do some forensic analysis on a snapshot of the cluster and be sure that the data will remain unchanged while you're doing it. I think that silently ignoring this setting would be surprising to users trying to do that.

I'm reopening this issue for that reason.