Elasticsearch: Elasticsearch-plugin fills tmp with empty dirs

[PereBal]

Elasticsearch version (bin/elasticsearch --version): 5.6.2 to 6.4.1 (at least)

Plugins installed: []

JVM version (java -version): 1.8

OS version (uname -a if on a Unix-like system): CentOs7

Description of the problem including expected versus actual behavior: Running elasticsearch-plugin list creates a directory under /tmp every run. If the directory is necessary for the run I expect the tool to either reuse it or clean it up after usage.

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.

1. List dirs under /tmp matching elasticsearch.: ls /tmp | grep 'elasticsearch\.'
2. Run plugin list: bin/elasticsearch-plugin list
3. List dirs under /tmp matching elasticsearch. again, a new dir should be there every execution

Provide logs (if relevant):

[.@host /tmp]$ ls -l | grep 'elasticsearch\.'
[.@host /tmp]$ sudo ES_PATH_CONF=/etc/elasticsearch /usr/share/elasticsearch/bin/elasticsearch-plugin list --verbose
Plugins directory: /usr/share/elasticsearch/plugins
[.@host /tmp]$ ls -l | grep 'elasticsearch\.'
drwx------ 2 root          root          4096 Oct 15 11:02 elasticsearch.yurOJFCO
[.@host /tmp]$ sudo ES_PATH_CONF=/etc/elasticsearch /usr/share/elasticsearch/bin/elasticsearch-plugin list
[.@host /tmp]$ ls -l | grep 'elasticsearch\.'
drwx------ 2 root          root          4096 Oct 15 11:02 elasticsearch.v9avpNCr
drwx------ 2 root          root          4096 Oct 15 11:02 elasticsearch.yurOJFCO

[elasticmachine]

Pinging @elastic/es-core-infra

[spinscale]

I can reproduce this under Linux (not under osx though) using 6.4.2.

This should indeed be cleaned up.

[rjernst]

This happens because elasticsearch-env (which is called by all of our scripts, including bin/elasticsearch) calls mktemp. We might want to only to set the temp dir like this when running elasticsearch? /cc @jasontedor

[jasontedor]

We can look into improving this. However, what is the concrete user-facing problem here? Knowing this will help us prioritize the effort. Modern systemd-based systems will clean /tmp periodically, and it will clean itself on reboot.

[PereBal]

@jasontedor we have a very heterogeneous cluster topology and use elasticsearch-plugin list to decide what is the connection method for our health checks (mostly if xpack is installed then https else http) .

As you can guess, even if the Os might rotate _/tmp_ and it should be mounted as tmpfs, a new dir every execution and frequent executions have caused us inode starvation issues.

I can imagine your thoughts for not prioritizing this, but in that case, what would be a good alternative for us?

[jasontedor]

@PereBal Thanks for clarifying.

As a workaround, for now you can set ES_TMPDIR explicitly to a specific temporary directory and each invocation of elasticsearch-plugin will reuse that temporary directory.