Elasticsearch: Elastic 6.2.1 in a Windows 2016 container crashes on startup if path.data is a volume

[chrisvanderpennen]

Elasticsearch version (bin/elasticsearch --version):
Version: 6.2.1, Build: 7299dc3/2018-02-07T19:34:26.990113Z, JVM: 9.0.4 (just in case)
Version: 6.2.1, Build: 7299dc3/2018-02-07T19:34:26.990113Z, JVM: 1.8.0_162

Plugins installed: []

JVM version (java -version):
Oracle server JRE windows/x64, versions:

java version "9.0.4"
Java(TM) SE Runtime Environment (build 9.0.4+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.4+11, mixed mode)

java version "1.8.0_162"
Java(TM) SE Runtime Environment (build 1.8.0_162-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)



md5-8f432ac76127ff854a46e1657c2a24f1



Platform Version
-------- -------
 Win32NT 10.0.14393.0



md5-ae93eb02ebac5d9fce274c9dba534b85



Platform ServicePack Version      VersionString
-------- ----------- -------      -------------
 Win32NT             10.0.14393.0 Microsoft Windows NT 10.0.14393.0



md5-a2c521d82c6278053ebd5021e26884fb



[2018-02-09T12:05:14,409][INFO ][o.e.n.Node               ] [057F70EA3551] initializing ...
[2018-02-09T12:05:14,466][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [057F70EA3551] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[G:\db\elastic]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.1.jar:6.2.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.1.jar:6.2.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.1.jar:6.2.1]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[G:\db\elastic]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:244) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.node.Node.<init>(Node.java:264) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.node.Node.<init>(Node.java:246) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.1.jar:6.2.1]
        ... 6 more
Caused by: java.io.IOException: failed to obtain lock on G:\db\nodes\0
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:223) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.node.Node.<init>(Node.java:264) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.node.Node.<init>(Node.java:246) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.1.jar:6.2.1]
        ... 6 more
Caused by: java.nio.file.NoSuchFileException: G:\db\nodes\0
        at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:79) ~[?:?]
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) ~[?:?]
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) ~[?:?]
        at sun.nio.fs.WindowsLinkSupport.getFinalPath(WindowsLinkSupport.java:82) ~[?:?]
        at sun.nio.fs.WindowsFileStore.create(WindowsFileStore.java:94) ~[?:?]
        at sun.nio.fs.WindowsFileSystemProvider.getFileStore(WindowsFileSystemProvider.java:482) ~[?:?]
        at java.nio.file.Files.getFileStore(Files.java:1461) ~[?:1.8.0_162]
        at org.elasticsearch.env.Environment.getFileStore(Environment.java:306) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.env.NodeEnvironment$NodePath.<init>(NodeEnvironment.java:104) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:210) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.node.Node.<init>(Node.java:264) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.node.Node.<init>(Node.java:246) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.1.jar:6.2.1]
        ... 6 more

[colings86]

@jasontedor I'm assigning to you but there may well be someone else better suited to look at this so feel free to pass it on to them. Also maybe @Mpdreamz has some thoughts about what might be going on here?

[jasontedor]

At this time we do not support running in a Windows container.

[sixeyed]

One day you might support Windows :) Confirming that this issue is with the latest releases of Elasticsearch. Version 5.6.11 works correctly in Windows containers. If you install 6.4.0 instead then you get this problem.

If you're interested you can test this from my repo. The Dockerfile defaults to installing 5.6.11 which gives you a working container. Build the same Dockerfile with --build-arg ES_VERSION=6.4.0 and you'll get the file not found issue.

[rifaterdemsahin]

any updates on this one ?

[a4chet]

I have Elasticsearch working in a nanoserver (1803) (docker pull mcr.microsoft.com/powershell:nanoserver) + OpenJDK11, Windows 10 host environment (1809) using process isolation. The recent fixes to the symbolic links in 1809 fix this problem.

Here are my dockerfiles & docker-compose yaml for my sample test. I use a locally hosted artifactory instance to proxy our docker images. Substitute with docker hub or your repo appropriately.

Dockerfile for building an openJDK:

FROM docker.artifactory.org:8080/microsoft/nanoserver:28Jan19_ps1803
SHELL ["pwsh", "-Command", "$ErrorActionPreference = 'Stop';"]

ADD http://artifactory:8081/artifactory/system-thirdparty/net/java/openjdk/11/openjdk-11-x64.zip openjdk.zip

ENV JAVA_HOME=C:\\jdk-11 \
       PATH=C:\\jdk-11\\bin;C:\\windows\\system32${PATH}

RUN       Expand-Archive -Path c:/openjdk.zip -DestinationPath c:/ ;\
      Remove-Item c:/openjdk.zip -Force     

Here is elastic search dockerfile based on the OpenJDK image above:

FROM docker.artifactory.org:8080/openjdk/openjdk:11_28_win1803
SHELL ["pwsh", "-Command", "$ErrorActionPreference = 'Stop';"]

ADD http://artifactory:8081/artifactory/system-thirdparty/com/elastic/elasticsearch/6.0.0/elasticsearch-6.0.0.zip /elasticsearch.zip

RUN Expand-Archive -Path c:/elasticsearch.zip -DestinationPath c:/ ;\
        Rename-Item c:/elasticsearch-6.0.0 c:/elasticsearch ;\
        Remove-Item c:/elasticsearch.zip -Force ;\
        New-Item -Path c:/temp -ItemType "directory"


# These mapped drive letters are no longer needed on Win10/1809!
# Need to mount external volumes as their own drive letters
# This is a bug in Java (sun.nio.fs.WindowsLinkSupport.getRealPath(WindowsLinkSupport.java:242))
# running in Docker for Windows. Java cannot resolve the Docker sym link mounted in the container.
#RUN Set-ItemProperty -path 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\DOS Devices' -Name 'L:' -Value '\??\C:\elasticsearch\logs' -Type String
#RUN Set-ItemProperty -path 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\DOS Devices' -Name 'D:' -Value '\??\C:\elasticsearch\data' -Type String
#RUN Set-ItemProperty -path 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\DOS Devices' -Name 'X:' -Value '\??\C:\elasticsearch\config' -Type String

SHELL ["cmd", "/S", "/C"]

ENV ES_PATH_CONF='c:\\\\elasticsearch\\\\config'

# We will mount the configuration externally        
RUN del c:\elasticsearch\config /F /S /Q         

WORKDIR c:/elasticsearch/bin
CMD ["cmd",  "/C","elasticsearch.bat"]

Snippet of docker-compose yaml:

version: '3.7'
services:
  elasticsearch:
    image: test/elasticsearch
    isolation: process
    build:
      context: ./docker/elasticsearch
    volumes:
      - "./docker/elasticsearch/config:c:/elasticsearch/config"
      - "./docker/elasticsearch/data:c:/elasticsearch/data"
      - "./docker/elasticsearch/logs:c:/elasticsearch/logs"
    ports:
      - "9200:9200"
    networks:
      - default      
networks:
  default:
    driver: nat

(Adding relevant elasticsearch.yml):

path.data: c:\elasticsearch\data
...
path.logs: c:\elasticsearch\logs

[nikfot]

I have Elasticsearch working in a nanoserver (1803) (docker pull mcr.microsoft.com/powershell:nanoserver) + OpenJDK11, Windows 10 host environment (1809) using process isolation. The recent fixes to the symbolic links in 1809 fix this problem.

Here are my dockerfiles & docker-compose yaml for my sample test. I use a locally hosted artifactory instance to proxy our docker images. Substitute with docker hub or your repo appropriately.

Dockerfile for building an openJDK:

FROM docker.artifactory.org:8080/microsoft/nanoserver:28Jan19_ps1803
SHELL ["pwsh", "-Command", "$ErrorActionPreference = 'Stop';"]

ADD http://artifactory:8081/artifactory/system-thirdparty/net/java/openjdk/11/openjdk-11-x64.zip openjdk.zip

ENV JAVA_HOME=C:\\jdk-11 \
       PATH=C:\\jdk-11\\bin;C:\\windows\\system32${PATH}

RUN       Expand-Archive -Path c:/openjdk.zip -DestinationPath c:/ ;\
      Remove-Item c:/openjdk.zip -Force     

Here is elastic search dockerfile based on the OpenJDK image above:

FROM docker.artifactory.org:8080/openjdk/openjdk:11_28_win1803
SHELL ["pwsh", "-Command", "$ErrorActionPreference = 'Stop';"]

ADD http://artifactory:8081/artifactory/system-thirdparty/com/elastic/elasticsearch/6.0.0/elasticsearch-6.0.0.zip /elasticsearch.zip

RUN Expand-Archive -Path c:/elasticsearch.zip -DestinationPath c:/ ;\
        Rename-Item c:/elasticsearch-6.0.0 c:/elasticsearch ;\
        Remove-Item c:/elasticsearch.zip -Force ;\
        New-Item -Path c:/temp -ItemType "directory"


# These mapped drive letters are no longer needed on Win10/1809!
# Need to mount external volumes as their own drive letters
# This is a bug in Java (sun.nio.fs.WindowsLinkSupport.getRealPath(WindowsLinkSupport.java:242))
# running in Docker for Windows. Java cannot resolve the Docker sym link mounted in the container.
#RUN Set-ItemProperty -path 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\DOS Devices' -Name 'L:' -Value '\??\C:\elasticsearch\logs' -Type String
#RUN Set-ItemProperty -path 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\DOS Devices' -Name 'D:' -Value '\??\C:\elasticsearch\data' -Type String
#RUN Set-ItemProperty -path 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\DOS Devices' -Name 'X:' -Value '\??\C:\elasticsearch\config' -Type String

SHELL ["cmd", "/S", "/C"]

ENV ES_PATH_CONF='c:\\\\elasticsearch\\\\config'

# We will mount the configuration externally        
RUN del c:\elasticsearch\config /F /S /Q         

WORKDIR c:/elasticsearch/bin
CMD ["cmd",  "/C","elasticsearch.bat"]

Snippet of docker-compose yaml:

version: '3.7'
services:
  elasticsearch:
    image: test/elasticsearch
    isolation: process
    build:
      context: ./docker/elasticsearch
    volumes:
      - "./docker/elasticsearch/config:c:/elasticsearch/config"
      - "./docker/elasticsearch/data:c:/elasticsearch/data"
      - "./docker/elasticsearch/logs:c:/elasticsearch/logs"
    ports:
      - "9200:9200"
    networks:
      - default      
networks:
  default:
    driver: nat

(Adding relevant elasticsearch.yml):

path.data: c:\elasticsearch\data
...
path.logs: c:\elasticsearch\logs

Interesting workaround. Did you build it exactly the way you mention? Because I tried it, but i had a problem with java_home on the elasticsearch image
could not find java in JAVA_HOME or bundled at C:\elasticsearch\jdk
Any ideas?

[a4chet]

Most likely your Java unzip location is not the same as mine so your ENV JAVA_HOME is incorrect

[nikfot]

Most likely your Java unzip location is not the same as mine so your ENV JAVA_HOME is incorrect

Sorry, my bad. I forgot to mention I tried elasticsearch 7.1.1 instead of 6. I just took a look at the ES 6 elasticsearch.bat and it has these commands at the end:

%JAVA% %ES_JAVA_OPTS% -Delasticsearch -Des.path.home="%ES_HOME%" -Des.path.conf="%ES_PATH_CONF%" -cp "%ES_CLASSPATH%" "org.elasticsearch.bootstrap.Elasticsearch" !newparams!

While the elasticsearch.bat in ES7 looks like this:
%JAVA% %ES_JAVA_OPTS% -Delasticsearch -Des.path.home="%ES_HOME%" -Des.path.conf="%ES_PATH_CONF%" -Des.distribution.flavor="%ES_DISTRIBUTION_FLAVOR%" -Des.distribution.type="%ES_DISTRIBUTION_TYPE%" -Des.bundled_jd="%ES_BUNDLED_JDK%" -cp "%ES_CLASSPATH%" "org.elasticsearch.bootstrap.Elasticsearch" !newparams!

[BewotecDaVinci]

After spending some days now to get a elastic cluster running in a windows docker based scenario, trying all various configurations to solve node lock issues and already been quite frustrated I came across this GIT Issue-Entry which just states "At this time we do not support running in a Windows container.".

Since elastic search supports windows based operating systems it is not quite obvious that windows docker based images are not supported, and there is not any single comment about this on the "Install elastic search with docker" site. Would be great to add this in some way to prevent others for stumbling into the same trap.

Anyway, here are my experiences with various setups:

• it doesn't really matter which version I tried, elastic search 5.6.11, any version 6.X or version 7.6.0
• it doesn't matter if I use volume mounts or bind mounts
• windows container version also does not really matter

Everything runs smoothly after the first cluster startup. Data is filled in and returned if searched, no issues.
But after shutting down the elastic cluster by sending a docker-compose down or a docker-compose up while the containers are rebuilded the following error occurs on one or more nodes:

failed to obtain node locks, tried [[C:dockerdata\data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?

So while investigating various forum entries regarding this isue I came across some suggestions to either increase max_local_storage_nodes, which will not be supported anymore in future versions, or just remove the node.lock file during the next cluster startup. Removing the file works, but leads directly to the next issue coming from existing write.lock files. After also removing the write.lock files everything starts up but the indices data is lost / corrupted and not accessible.

For some reason it looks like the data is not persisted correctly during docker shutdown. Even after adding a delay to the docker shutdown, by using a separated process which starts elastic search and waits several seconds after receiving the docker SIGTERM/SIGKILL notification, the elastic nodes are in an invalid state.

Would greatly appreciate to have this fixed in any upcoming elastic search version. If you are interested in further details please let me know.

Regards