Beats: [Windows] Beats service fail to start when keystore is used

[adriansr]

For confirmed bugs, please report:

• Version: 6.7.1 - 7.1.0
• Operating System: Windows
• Steps to Reproduce:

• Install any Beat following Windows setup instructions.
• Setup Beat service PS> .\install-XXXX.ps1.
• The Beat service starts fine. CMD> sc start filebeat. Check with services.msc and no errors in Event Viewer.
• Setup a keystore and a pass: xxxbeat keystore create xxxbeat keystore add PASS.
• Edit configuration to use PASS, for example in output.elasticsearch.password.
• Restart Beat service.
• The service fails to start as it terminates immediately.
• Move the xxxbeat.keystore file from C:\Program Files\xxxbeat\data to C:\Program Files\xxxbeat\.
• The Beat service now starts successfully.

I've confirmed this bug with 6.7.1 and 7.1.0.

[ph]

It makes more sense to have the keystore defined in data dir vs the config dir. So lets make sure we make the necessary changes to keep it in the data directory.

[narph]

@adriansr, @ph , I am getting different results here based on the steps above.
All good until the Move the xxxbeat.keystore file from C:\Program Files\xxxbeat\data to C:\Program Files\xxxbeat\. step.

The beat service will not start successfully when moving the keystore files to the C:\Program Files\xxxbeat\ folder, it will start successfully when I move them to C:\ProgramData\xxxbeat folder.
This is most likely explained by the configuration settings in the install-service-xxxbeat.ps1 file:

# Create the new service.
New-Service -name metricbeat `
  -displayName Metricbeat `
  -binaryPathName "`"$workdir\metricbeat.exe`" -c `"$workdir\metricbeat.yml`" -path.home `"$workdir`" -path.data `"C:\ProgramData\metricbeat`" -path.logs `"C:\ProgramData\metricbeat\logs`""

For example:
We are setting -path.data to C:\ProgramData\metricbeat so when starting the metricbeat service it will try to find the keystore file in the data folder which is set to C:\ProgramData\metricbeat.

But, when creating the keystore file we are running xxxbeat keystore create command which will run independently of what the service configuration is set, so, the keystore files will end up in C:\Program Files\xxxbeat\data (working directory/data).

An obvious temp workaround here would be to set the keystore location manually when creating it: xxxbeat keystore create -E keystore.path= C:\ProgramData\xxxbeat

Also, why is the data path pointing to C:\ProgramData\xxxbeat and not C:\ProgramData\xxxbeat\data in the install-service-xxxbeat.ps1 file?

No mutable files should live in the ProgramFiles directory so we should make sure the docs don't default here.
Should the approach be to store the keystore files in the ProgramData\xxxbeat\data(?) by default?
At the moment the paths are relative to the working directory of the xxxbeat.

[adriansr]

@narph you're completely right.

I must've done some wrong while testing it. The keystore is expected in C:\programdata\metricbeat.

Shall we still consider this a bug, that metricbeat keystore create places it under C:\program files\metricbeat\data instead of c:\programdata\metricbeat, or the user is expected to pass the correct -path.data argument?

[ph]

Not sure if I understand correctly the problem, but lets assume the following.

Scenario 1: Run filebeat directly from the zipped directory.

In that case the keystore will be located in a data directory which is next to the binary.

Scenario 2: Install Filebeat as a service using our installer script.

In that case the keystore is expected to be in the c:\programdata\xxxbeat directory.
Now if you go the beat directory and use any keystore subcommands it will not use that directory by default.

On Linux system when you install the beat from a package we will also install a special shell scripts in your path that will setup the correct path for the beats before executing any commands.

Maybe we should provide a similar mechanism for user when they install beats as a service on Windows.

[narph]

@adriansr, I do not consider this to be a bug as it does exactly as instructed.
We could inform/warn the user (in the powershell script) to add the keystore.path when running the keystore create command (-E keystore.path= C:\ProgramData\xxxbeat).
But that won't help if the user creates the keystore files before running the install script.

Another alternative is the user modifying the xxxbeat.yml file and setting the path.data to C:\ProgramData\xxxbeat but I personally do not recommend this as the user should not modify files residing in that directory (C:\Program Files). This might be a separate topic for discussion.

@ph, the scenarios are correct, regarding:

Now if you go the beat directory and use any keystore sub-commands it will not use that directory by default.

this is because the argument path.data is encoded only in the service startup arguments so, in this case, it will use the working directory/data folder instead, (when called from the command line).

Maybe we should provide a similar mechanism for user when they install beats as a service on Windows.

Not sure there is a similar mechanism on windows, the only thing that comes to mind is adding an environment variable with the data folder location when running the install script and try to access it when creating the keystore file.
But again, what if the user runs the keystore sub-command before running the install script?

The most windows friendly solution would be to have an MSI for system wide installs and keep the zip file for local installations.
@elastic/microsoft, are there any future plans for a beats installer?

[ph]

@narph I also think the MSI is the way to go, right now we have an hybrid solution which is less than ideal. The environment variable would work, but it still an hack.

[philkra]

@elastic/microsoft, are there any future plans for a beats installer?

We hired a dedicated windows installer engineer, he will start on June 17. I'll set up a meeting for us in order to compile issues like this and make sure that he can hit the ground running. Sounds like a plan?

[narph]

@ph, I would not go on the environment variable route as it doesn't look as a resilient fix that could cover all use cases and since there are future plans here for an MSI maybe the way to go is to add more information/notes in the keystore related documentation about the location.

[KaamilW]

Hey,

I am facing the same problem with usage of keystore for metricbeat on windows. Besically, it works fine once the metricbeat is running in the console. However if I want to install it as a service he is not reading keystore variables correctly and I get errors related to authorization.

Was any solution found already to that issue? Or maybe a workaround?

Regards,
Kamil

[russcam]

@KaamilW The workaround for now is to specify where the keystore should be created when creating it, to match the location where the service expects it to be

xxxbeat keystore create -E keystore.path= C:\ProgramData\xxxbeat

[KaamilW]

One more question, how should I be then referencing to the value set in .yml? Lets assume that i have named my variable ES_PWD_TEST.

Should it be smth like:
output.elasticsearch.password: "${ES_PWD_TEST}"
output.elasticsearch.password: "/${ES_PWD_TEST}"
output.elasticsearch.password: "\${ES_PWD_TEST}"

No matter what I try i keep getting errors once trying to turn on metricbeat service. What might be wrong still??

[narph]

@KaamilW , in the configuration file you should be able to use
output.elasticsearch.password: "${ES_PWD_TEST}" (more here)
What are the errors you are encountering when starting the metricbeat service?

[adrwh]

Anyone know when this will work out of the box?

The default Winlogbeat path.data is ${path.home}/data, documented here https://www.elastic.co/guide/en/beats/winlogbeat/current/directory-layout.html.

However, the default service installer install-service-winlogbeat.ps1 is programmed to set -path.data to "C:\ProgramData\winlogbeat" instead. Is this intentional, and why?

.\winlogbeat.exe keystore create creates the file ${path.home}/data/winlogbeat.keystore as documented here https://www.elastic.co/guide/en/beats/winlogbeat/current/keystore.html

In winlogbeat.yml i configure cloud.id: "${CLOUD_ID}" and cloud.auth: "${ES_PWD}"

Now when i go Start-Service Winlogbeat, tries to start, sets the path.data to C:\programdata\winlogbeat\ and fails because it isn't seeing my keystore items and provides the error ERROR instance/beat.go:877 Exiting: error initializing publisher: missing required field accessing 'output.elasticsearch.hosts.

If i copy the winlogbeat.keystore to C:\programdata\winlogbeat\winlogbeat.keystore which is where the service has path.data configured, it starts up as expected and the packets flow.

[ygel]

I am working on MSI for Beats and considered setting home directory to ProgramData and installing bundled components (kibana, security, ...) there as well. Based on the discussion above, this is how I'll proceed. Since when beats are run from CLI, they read yaml for paths, I might also consider modifying default yaml with correct paths. This will also reduce inconsistency in environment of how service and CLI run.

[Udbv]

filebeat 7.4.0 keystore fix
by default keystore saved in ~Filebeat/data. in service it has another path.
edit install-service-filebeat.ps1 like that

-binaryPathName "`"$workdir\filebeat.exe`" -c `"$workdir\filebeat.yml`" -path.home `"$workdir`" -path.data `"$workdir\data`" -path.logs `"$workdir\logs`" -E logging.files.redirect_stderr=true" 

[narph]

@adriansr , we have been encountering this issue a lot more often than expected, does It makes more sense to have the keystore defined in data dir from the start?

[ygel]

@narph official Beats MSI takes care of that by exposing CLI shim that reads the same config files as the service. Please consider recommending using the MSI. One caveat is current beta doesn't support config file migration yet, so back up config/data and then place it in the new locations over default ones.

[narph]

@ygel thanks, is there any documentation at the moment regarding the new installers, maybe a mention here or steps https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-installation.html.
Also, we might need to add a warning/note in the Create keystore docs for windows users to check where their data folder will be (not sure how to word this). @dedemorton what do you think?

[dedemorton]

@narph I've created a doc issue to track the update. Thanks!

[romain-chanu]

It seems that the way forward with this issue will be to use the MSI files (c.f https://github.com/elastic/beats/issues/12315#issuecomment-604540810).

On a side note, regarding the previous workaround, keystore.path expects the full path to the keystore file. For example:

• .\metricbeat.exe -E keystore.path="C:\ProgramData\metricbeat\metricbeat.keystore" keystore create

In this context, keystore.path must also be specified when adding a secret into the keystore. For example:

• .\metricbeat.exe -E keystore.path="C:\ProgramData\metricbeat\metrictbeat.keystore" keystore add MY_SECRET