Elasticsearch: Cannot remove plugin from docker container

[tback]

It seems uninstalling plugins from the official docker image is broken.

My colleagues were unable to reproduce this error on debian stretch with btrfs.

I'm opening this issue in response to @jasontedor https://github.com/elastic/elasticsearch/pull/23548#issuecomment-295793139

Elasticsearch version:
5.3.1

Plugins installed: [x-pack]

JVM version:

docker run -ti --rm --name es docker.elastic.co/elasticsearch/elasticsearch:5.3.1 /usr/lib/jvm/java-1.8-openjdk/bin/java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (IcedTea 3.3.0) (Alpine 8.121.13-r0)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

OS version:
macos sierra 10.12.3 (16D32)
Docker Version 17.03.1-ce-mac5 (16048, b18e2a50cc)

Description of the problem including expected versus actual behavior:
I expect to successfully remove x-pack like any other plugin from the official docker image. I get an error instead.
I'd much rather get an official image without x-pack altogether, but thats a business decision I guess. Making x-pack hard to uninstall will drive me to community images for sure.

Steps to reproduce:

1. Create a Dockerfile
   ```FROM docker.elastic.co/elasticsearch/elasticsearch:5.3.1

RUN eval ${ES_JAVA_OPTS:-} &&
elasticsearch-plugin remove --silent x-pack

 2. Try to build it
```docker build .```
 3. Trace
```docker build .
Sending build context to Docker daemon 2.048 kB
Step 1/2 : FROM docker.elastic.co/elasticsearch/elasticsearch:5.3.1
5.3.1: Pulling from elasticsearch/elasticsearch
ec37562cf8fa: Pull complete
3f5b02e577c3: Pull complete
e4c7c0eb9289: Pull complete
e8008e68933d: Pull complete
1cfca4b45f64: Pull complete
77148a7a523b: Pull complete
c93eabd5eff3: Pull complete
d82d8a15579f: Pull complete
0f2b8d9540a0: Pull complete
5c9aa0d9a460: Pull complete
Digest: sha256:94ebca9d6316902596ceef6934161be39cca04885fcf07c8ab3feb04c02b4933
Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:5.3.1
 ---> 94062e7e850e
Step 2/2 : RUN eval ${ES_JAVA_OPTS:-} &&     elasticsearch-plugin remove --silent x-pack
 ---> Running in c7a27bb18817
Exception in thread "main" java.nio.file.DirectoryNotEmptyException: /usr/share/elasticsearch/plugins/x-pack
    at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:498)
    at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
    at java.nio.file.Files.move(Files.java:1395)
    at org.elasticsearch.plugins.RemovePluginCommand.execute(RemovePluginCommand.java:91)
    at org.elasticsearch.plugins.RemovePluginCommand.execute(RemovePluginCommand.java:56)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:58)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
    at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:69)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
    at org.elasticsearch.cli.Command.main(Command.java:88)
    at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)
The command '/bin/sh -c eval ${ES_JAVA_OPTS:-} &&     elasticsearch-plugin remove --silent x-pack' returned a non-zero code: 1

[tback]

This is also easily reproduced on the vagrant base box debian/jessie64.

I suspect this is related to aufs.

Elasticsearch version:
5.3.1

Plugins installed: [x-pack]

JVM version:
(Same base image, same jvm version)
```docker run -ti --rm --name es docker.elastic.co/elasticsearch/elasticsearch:5.3.1 /usr/lib/jvm/java-1.8-openjdk/bin/java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (IcedTea 3.3.0) (Alpine 8.121.13-r0)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

**OS version:**
debian jessie 64bit

uname -a
Linux jessie 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1 (2016-12-30) x86_64 GNU/Linux
docker version
Client:
Version: 17.03.1-ce
API version: 1.27
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:07:28 2017
OS/Arch: linux/amd64

Server:
Version: 17.03.1-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:07:28 2017
OS/Arch: linux/amd64
Experimental: false

**Steps to reproduce**

vagrant init debian/jessie64
vagrant up
vagrant ssh

Then, on the machine
install docker

sudo apt-get install
apt-transport-https
ca-certificates
curl
gnupg2
software-properties-common
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
sudo add-apt-repository
"deb [arch=amd64] https://download.docker.com/linux/debian
$(lsb_release -cs)
stable"
sudo apt-get update
sudo apt-get install docker-ce
sudo adduser vagrant docker

logout && login again (so the new group becomes effective

Create Dockerfile with same contents as above:

FROM docker.elastic.co/elasticsearch/elasticsearch:5.3.1

RUN eval ${ES_JAVA_OPTS:-} &&
elasticsearch-plugin remove --silent x-pack

Run the docker build

vagrant@jessie:~$ docker build .

And get the trace:

Sending build context to Docker daemon 13.31 kB
Step 1/2 : FROM docker.elastic.co/elasticsearch/elasticsearch:5.3.1
5.3.1: Pulling from elasticsearch/elasticsearch
ec37562cf8fa: Pull complete
3f5b02e577c3: Pull complete
e4c7c0eb9289: Pull complete
e8008e68933d: Pull complete
1cfca4b45f64: Pull complete
77148a7a523b: Pull complete
c93eabd5eff3: Pull complete
d82d8a15579f: Pull complete
0f2b8d9540a0: Pull complete
5c9aa0d9a460: Pull complete
Digest: sha256:94ebca9d6316902596ceef6934161be39cca04885fcf07c8ab3feb04c02b4933
Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:5.3.1
---> 94062e7e850e
Step 2/2 : RUN eval ${ES_JAVA_OPTS:-} && elasticsearch-plugin remove --silent x-pack
---> Running in b6e5800a9b42
Exception in thread "main" java.nio.file.DirectoryNotEmptyException: /usr/share/elasticsearch/plugins/x-pack
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:498)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at org.elasticsearch.plugins.RemovePluginCommand.execute(RemovePluginCommand.java:91)
at org.elasticsearch.plugins.RemovePluginCommand.execute(RemovePluginCommand.java:56)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:58)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:69)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
at org.elasticsearch.cli.Command.main(Command.java:88)
at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)
The command '/bin/sh -c eval ${ES_JAVA_OPTS:-} && elasticsearch-plugin remove --silent x-pack' returned a non-zero code: 1
```

[jasontedor]

Thanks for the report @tback. I've found the source of the issue; it's a combination of renaming not working on the lower layer of a union filesystem and a bug in the JDK. Briefly the JDK tries to do the following:

• if it's an atomic move, try to rename the source to target and if that fails with EXDEV, throw atomic move not supported
• if it's not an atomic move, try to rename the source except this time do not fail if EXDEV occurred
• when the previous rename fails, copy source to target and now try to rmdir the source; the bug is that rmdir requires the source to be empty but the JDK never cleans up the source so this will fail 100% of the time

I'd much rather get an official image without x-pack altogether, but thats a business decision I guess. Making x-pack hard to uninstall will drive me to community images for sure.

Your feedback here is very much appreciated. I want you to know that we are not intentionally making it difficult to uninstall X-Pack, we will prepare a fix for this for 5.4.0.

[jasontedor]

We will prepare a fix for this for 5.4.0.

I've opened #24252. We are going to try to get this into a 5.3.2 release (do not take that as a guarantee). I've built a new Docker image locally based off this change and the issue no longer reproduces.

[dliappis]

Thanks for taking care of this @jasontedor and reporting it @tback. I've taken a note to add an acceptance test for removing plugins in our docker image building acceptance tests. This will most likely land in 5.4.0 where we have introduced a more flexible acceptance test framework.