Elasticsearch: Elasticsearch 6.2.2 fails to run as a windows service

[LucidObscurity]

Describe the feature:
Running Elasticsearch as a windows service

Windows service fails to start
Elasticsearch version (bin/elasticsearch --version):
6.2.2
Plugins installed: []
none
JVM version (java -version):
1.8.0_141
OS version (uname -a if on a Unix-like system):
Windows Server 2008 R2 SP1
Description of the problem including expected versus actual behavior:
Unable to run elasticsearch as a windows service. Attempting to start fails with error 4. Logs show: "The system could not find the environment option that was entered". Running from console works fine both elevated and non-elevated. JAVA_HOME environment variable is set for the SYSTEM. Rolling back to elasticsearch v6.1.1 alleviates the problem. After upgrading to 6.2.2 the windows service was "remove"d and re"install"ed. I also tried to delete and re-add the environment variable JAVA_HOME with reboots just for good measure.
Steps to reproduce:

• Upgrade to Elasticsearch 6.2.2 from Elasticsearch 6.1.1
• remove existing service and install as new
• try to start service
  Please include a minimal but complete recreation of the problem, including
  (e.g.) index creation, mappings, settings, query etc. The easier you make for
  us to reproduce it, the more likely that somebody will take the time to look at it.

1.
2.
3.

Provide logs (if relevant):

[javanna]

@elastic/microsoft any ideas?

[javanna]

cc @elastic/es-core-infra

[Mpdreamz]

@LucidObscurity are you using the MSI or zip distribution?

[LucidObscurity]

zip

[LucidObscurity]

I'll also add the deployment process deletes everything except the data directory then copies the contents of the package. I noticed some differences in the jvm.options from the one we deploy and replaced that with the one from the package with the same results.

[jasontedor]

This can be due to a missing system environment variables or due to the presence of the -server flag from an old jvm.options file (the default jvm.options file no longer includes this flag and the startup script no longer filters it out).

[LucidObscurity]

Are there other variables which is 6.2.2 requires other than JAVA_HOME which 6.1.1 did not. As I stated 6.1.1 is currently happily running as a windows service.

[LucidObscurity]

C:\Windows\system32>set
ALLUSERSPROFILE=C:\ProgramData
APPDATA=C:\Users[redacted]\AppData\Roaming
ASPNETCORE_ENVIRONMENT=DevMon
CommonProgramFiles=C:\Program FilesCommon Files
CommonProgramFiles(x86)=C:\Program Files (x86)Common Files
CommonProgramW6432=C:\Program FilesCommon Files
COMPUTERNAME=[redacted]
ComSpec=C:\Windows\system32\cmd.exe
FP_NO_HOST_CHECK=NO
HOMEDRIVE=[redacted]
HOMEPATH=\
HOMESHARE=[redacted]
JAVA_HOME=D:\JAVA
LOCALAPPDATA=C:\Users[redacted]\AppData\Local
LOGONSERVER=\[redacted]
NUMBER_OF_PROCESSORS=1
OS=Windows_NT
Path=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32
\WindowsPowerShell\v1.0\;C:\Program Files\dotnet\;C:\Program Files\Microsoft\Web
Platform Installer\;C:\Program Files (x86)\dotnet\;C:\Windows\System32\WindowsP
owerShell\v1.0\
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_ARCHITECTURE=AMD64
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 26 Stepping 5, GenuineIntel
PROCESSOR_LEVEL=6
PROCESSOR_REVISION=1a05
ProgramData=C:\ProgramData
ProgramFiles=C:\Program Files
ProgramFiles(x86)=C:\Program Files (x86)
ProgramW6432=C:\Program Files
PROMPT=$P$G
PSModulePath=C:\Windows\system32\WindowsPowerShell\v1.0\Modules\
PUBLIC=C:\Users\Public
SystemDrive=C:
SystemRoot=C:\Windows
TEMP=C:\Users[redacted]\AppData\Local\Temp
TMP=C:\Users[redacted]\AppData\Local\Temp
USERDNSDOMAIN=[redacted]
USERDOMAIN=[redacted]
USERNAME=[redacted]
USERPROFILE=C:\Users[redacted]
windir=C:\Windows

[svenso]

Why has this issue been closed without a proper solution?
I had exactly the same issue.
The following article pointed me in the right direction:
https://confluence.atlassian.com/confkb/tomcat-fails-to-start-313463519.html

_The issue can happen when you try to specify the -server property when the /jre/bin/server/jvm.dll is being used as the JVM. Technically these two settings are redundant, thus the -server parameter is rejected and startup fails._

I manually removed the -server parameter from the registry:
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\elasticsearch-service-x64\Parameters\Java\Options

Afterwards, I was able to start the service.
The link provided by @jasontedor in the earlier reply is not pointing to a matching resolution at all (and even might lead to create other problems too).
--> How to reopen the case, that the -server variable is removed for windows?

[jasontedor]

@svenso The default jvm.options file packaged with Elasticsearch does not include the -server flag since Elasticsearch 6.0.0.

[jasontedor]

@svenso Thanks for the feedback here, I pushed feb5f9dd8bf83d9b7426bc81f63ed8c7481cdbed. I hope this helps.

[LucidObscurity]

-server was the problem I was having. Thanks for the follow up.

[svenso]

interesting. basically our deployment process should use the default jvm.options and the service is removed and recreated everytime. I will have to check that tomorrow. But I was not able to reproduce it on a clean system. Never mind, thanks all :)

[rmacmorran]

I had a similar issue, and fixing jvm.options only worked once I had run elasticsearch-service.bat -REMOVE and then elasticsearch-service.bat -INSTALL again.