Charts: [elasticsearch] Setting `config` chart value breaks cluster

[tereschenkov]

Which chart:
elasticsearch-12.3.1

Describe the bug
Setting config chart value breaks cluster. It is mounted inside a container with, I assume, a read-only mode.

To Reproduce
Steps to reproduce the behavior:

1. Add any content to config helm value
2. Deploy cluster with multiple nodes
3. Open a shell inside master or data pod and check the content of /opt/bitnami/elasticsearch/config/elasticsearch.yml
4. Verify that it has only content of config value

Expected behavior
/opt/bitnami/elasticsearch/config/elasticsearch.yml file inside a pod should have custom settings + settings configured by the container itself

Additional context
I'm trying to configure reindex.remote.whitelist setting and tried to use config helm value for that but when it is set, the whole deployment breaks. It happens because the content of config value is mounted inside the pod and I think it has a read-only mode, so configuration script cannot configure other settings, like name, node type and so on

[tereschenkov]

Some updates on my side. I tried to achieve the same using custom init scripts and succeeded only after upgraded docker version from 6.8.8-debian-10-r41 to 6.8.9-debian-10-r10 so maybe that's the problem with the mentioned issue as well.

I'll try to check it later this week

[marcosbc]

Hi @tereschenkov, note that the current approach requires you to specify the entire elasticsearch.yml configuration file if you set the config property, it does not append to the file. The idea is for you to be able to customize the most amount of settings for your deployment.

But as you mentioned, we added support for init scripts recently, and this may be valid for your approach of appending configuration to elasticsearch.yml without overriding the entire file.

[tereschenkov]

Hi @marcosbc, thanks for clarifying it. I think I understand why it's done this way. So yeah, using custom scripts is the best way to customize config in my case. I'll close the issue

[ados1991]

Hi, i am facing the same issue, @tereschenkov and @marcosbc can you show me how you proceed with the custom scripts ?

please give an example. it is an issue in production

best regards

[marcosbc]

@ados1991 You have several ways to specify init scripts, it is documented in the Elasticsearch chart README.

The easiest way is to specify initScripts in values.yaml:

## initdb scripts
## Specify dictionary of scripts to be run at first boot
## Alternatively, you can put your scripts under the files/docker-entrypoint-initdb.d directory
## For example:
## initScripts:
##   my_init_script.sh: |
##      #!/bin/sh
##      echo "Do something."

Alternatively, specify initScriptsSecret or initScriptsCM and pass the name of a secret or configmap, respectively, which contains an init script.

The init script needs to be a .sh script, the values.yaml contains the example included above.

[ados1991]

Hi @marcosbc Thanks for your answer, it was helpfull for my fix in production. Also i have a question about monitoring for my elasticsearch cluster.

In the documentation i see this (metrics.enabled | Enable prometheus exporter |) to enable prometheus to scrape the metrics but i also see this metrics.serviceMonitor.enabled for monitoring. Which one should i choose and what is this difference between the both ? Can i choose the both ? If i choose the both what does each of them perform ?

Sorry for this too many questions, i deployed in production and in order to avoid possibles issues in future, i want to implement monitoring.

Thanks in advance and Best Regards

[marcosbc]

@ados1991 This looks unrelated to the current issue, could you open a new Issue if, despite my reply, you have more questions?

In the documentation i see this (metrics.enabled | Enable prometheus exporter |) to enable prometheus to scrape the metrics but i also see this metrics.serviceMonitor.enabled for monitoring. Which one should i choose and what is this difference between the both ? Can i choose the both ? If i choose the both what does each of them perform ?

The service monitor allows to monitor your services, not only components (i.e. pods). You can read more about it here: https://coreos.com/blog/the-prometheus-operator.html

You should be able to use both without issues. See an example tutorial where we use MariaDB chart and enable both here: https://docs.bitnami.com/tutorials/create-multi-cluster-monitoring-dashboard-thanos-grafana-prometheus/