Beats: [BEATCM Elasticsearch 6.7, Metricbeat 6.7, Filebeat 6.7]

[toddferg]

Versions:
Elasticsearch 6.7.0
Metricbeat 6.7.0
Filebeat 6.7.0

ALL installed through the repo, RPM/YUM

Operating System:
CentOS Linux release 7.4.1708 (Core)

Environment:
Internal Certificate authority issued certificates

This occurs for metricbeat and Filebeat
Ways to reproduce:
When attempting to enroll a beat, Documentation doesn't specify what directory to run the command from:

When I run this command, it will create the metricbeat.keystore inside current working directory

metricbeat enroll https://welk1.toddhome.local:5601 theAccessCode -E "management.kibana.ssl.certificate_authorities=/etc/metricbeat/chain.pem"

If in /root, it will create the metricbeat.keystore in /root

Then

When running the command in the directory /etc/metricbeat, the file is created in there (/etc/metricbeat), but when attempting to launch the service, metricbeat gives the error.

2019-03-27T00:19:49.163-0500 INFO instance/beat.go:612 Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat] 2019-03-27T00:19:49.165-0500 INFO instance/beat.go:619 Beat UUID: c787e946-fe88-4bf8-a5ff-1bf3563069cf 2019-03-27T00:19:49.166-0500 INFO instance/beat.go:347 metricbeat stopped. 2019-03-27T00:19:49.169-0500 ERROR instance/beat.go:907 Exiting: parsing central management settings: missing field accessing 'management.access_token' (source:'/etc/metricbeat/metricbeat.yml')

Service: 

[Unit]
Description=Metricbeat is a lightweight shipper for metrics.
Documentation=https://www.elastic.co/products/beats/metricbeat
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml -path.home /usr/share/metricbeat -path.config /etc/metricbeat -path.data /var/lib/metricbeat -path.$
Restart=always

[Install]
WantedBy=multi-user.target

systemctl status

[root@elk1 metricbeat]# systemctl status metricbeat
● metricbeat.service - Metricbeat is a lightweight shipper for metrics.
Loaded: loaded (/usr/lib/systemd/system/metricbeat.service; disabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Wed 2019-03-27 11:28:19 CDT; 297ms ago
Docs: https://www.elastic.co/products/beats/metricbeat
Process: 11788 ExecStart=/usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml -path.home /usr/share/metricbeat -path.config /etc/metricbeat -path.data /var/lib/metricbeat -path.logs /var/log/metricbeat (code=exited, status=1/FAILURE)
Main PID: 11788 (code=exited, status=1/FAILURE)

Mar 27 11:28:18 elk1 systemd[1]: Unit metricbeat.service entered failed state.
Mar 27 11:28:18 elk1 systemd[1]: metricbeat.service failed.
Mar 27 11:28:19 elk1 systemd[1]: metricbeat.service holdoff time over, scheduling restart.
Mar 27 11:28:19 elk1 systemd[1]: Stopped Metricbeat is a lightweight shipper for metrics..
Mar 27 11:28:19 elk1 systemd[1]: start request repeated too quickly for metricbeat.service
Mar 27 11:28:19 elk1 systemd[1]: Failed to start Metricbeat is a lightweight shipper for metrics..
Mar 27 11:28:19 elk1 systemd[1]: Unit metricbeat.service entered failed state.
Mar 27 11:28:19 elk1 systemd[1]: metricbeat.service failed.

**Workaround**

Screen /usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml -path.home /usr/share/metricbeat -path.config /etc/metricbeat -path.data /var/lib/metricbeat -path.logs /var/log/metricbeat
```

[ph]

@toddferg Fix is backported in https://github.com/elastic/beats/pull/11497

There is a workaround, add the following to your configuration:

Filebeat:

keystore.path: /var/lib/filebeat/filebeat.keystore

Metricbeat:

keystore.path: /var/lib/filebeat/metricbeat.keystore

The fix will be included in 6.7.1, I will make a note for the known issues of that release.

[matthenning]

I ran into a similar problem and noticed a possible mismatch in variable names.
This is probably related to this issue, so I'll just append it here.

According to the log filebeat tried to access management.access_token after running filebeat enroll:

# tail -f /var/log/filebeat/filebeat
2019-03-28T13:21:46.523+0100    INFO    instance/beat.go:612    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2019-03-28T13:21:46.524+0100    INFO    instance/beat.go:619    Beat UUID: a6c2a5c7-9fad-4f4a-bf75-252b3d4578d2
2019-03-28T13:21:46.524+0100    INFO    instance/beat.go:347    filebeat stopped.
2019-03-28T13:21:46.527+0100    ERROR   instance/beat.go:907    Exiting: parsing central management settings: missing field accessing 'management.access_token' (source:'/etc/filebeat/filebeat.yml')

But the config file and keystore reference a different variable:

# filebeat keystore list
management.accesstoken

management:
  access_token: ${management.accesstoken}

Notice the difference:

management.access_token
management.accesstoken

[ph]

@matthenning I've looked at the code, the name differs because of in reality, it is actually two different contexts, the management.access_token refer to the key in the YAML and the management.accesstoken refer to a value in the Keystore. I believe the names are different because that could create a cyclic reference.

The real problem is due that the beat cannot find the keystore to retrieve the values from, if you look at my previous comment the workaround will fix your problem.

[matthenning]

Can confirm.
I added the suggested configuration to the default filebeat.yml, ran filebeat enroll TOKEN and added it again to the newly generated config.
Filebeat is now properly communicating with CM.
Thank you.

[matthenning]

Compare your syntax:

password: "$(ES_PWD)"

password: "${KI_PWD}"

[bigor44]

Compare your syntax:

password: "$(ES_PWD)"

password: "${KI_PWD}"

Thans a lot, i deleted my comment, it was dumb. i tried so many things and it was this ...

[ph]

merged in 6.7