Beats: Document privileges required for index lifecycle management

Created on 30 Jan 2019  路  11Comments  路  Source: elastic/beats

As a follow up to https://github.com/elastic/beats/pull/9263, we need to document the privileges required to set up Beats to work with index life cycle management.

Note that the security docs indicate that manage_ilm is required, but the Kibana UI does not show the setting as available. I've been unable to get ILM working with security enabled and need additional input from dev.

blocker docs
Source
picture dedemorton
👍3

Most helpful comment

I think I hit an issue with this today. It resulted in:

2019-04-26T18:03:22.337+0200    ERROR   pipeline/output.go:100  Failed to connect to backoff(elasticsearch(https://XXX)): Connection marked as failed because the onConnect callback failed: failed to check for alias 'metricbeat': (status=403) : 403 Forbidden:

I fixed it by adjusting the index name to metricbeat* rather than metricbeat-* and adding the manage privilege.
The alias is called metricbeat so the permission does not match with the dash at the end.

picture jakommo on 26 Apr 2019
👍4

All 11 comments

I was able to successfully load the ILM policy, but ran into problems when I enabled ilm in the Metricbeat config.

When I run Metricbeat with security enabled, but ILM disabled, Metricbeat ships events to Elasticsearch as expected. However, when I enable set ilm.enabled: true, I see " failed to check for alias: 403 Forbidden" errors:

2019-01-29T18:28:39.285-0800 INFO instance/beat.go:616 Home path: [/Users/dedemorton/BuildTesting/6.6.0_GA/metricbeat-6.6.0-darwin-x86_64] Config path: [/Users/dedemorton/BuildTesting/6.6.0_GA/metricbeat-6.6.0-darwin-x86_64] Data path: [/Users/dedemorton/BuildTesting/6.6.0_GA/metricbeat-6.6.0-darwin-x86_64/data] Logs path: [/Users/dedemorton/BuildTesting/6.6.0_GA/metricbeat-6.6.0-darwin-x86_64/logs] 2019-01-29T18:28:39.287-0800 INFO instance/beat.go:623 Beat UUID: d961ca44-ef7b-4753-a44f-f0c4626a5969 2019-01-29T18:28:39.287-0800 INFO [beat] instance/beat.go:936 Beat info {"system_info": {"beat": {"path": {"config": "/Users/dedemorton/BuildTesting/6.6.0_GA/metricbeat-6.6.0-darwin-x86_64", "data": "/Users/dedemorton/BuildTesting/6.6.0_GA/metricbeat-6.6.0-darwin-x86_64/data", "home": "/Users/dedemorton/BuildTesting/6.6.0_GA/metricbeat-6.6.0-darwin-x86_64", "logs": "/Users/dedemorton/BuildTesting/6.6.0_GA/metricbeat-6.6.0-darwin-x86_64/logs"}, "type": "metricbeat", "uuid": "d961ca44-ef7b-4753-a44f-f0c4626a5969"}}} 2019-01-29T18:28:39.287-0800 INFO [beat] instance/beat.go:945 Build info {"system_info": {"build": {"commit": "2c385a0764bdc537b6dc078a1d9bf11bb6d7bd95", "libbeat": "6.6.0", "time": "2019-01-24T10:38:21.000Z", "version": "6.6.0"}}} 2019-01-29T18:28:39.287-0800 INFO [beat] instance/beat.go:948 Go runtime info {"system_info": {"go": {"os":"darwin","arch":"amd64","max_procs":8,"version":"go1.10.8"}}} 2019-01-29T18:28:39.288-0800 INFO [beat] instance/beat.go:952 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-01-22T11:08:42.135577-08:00","name":"Rhodas-MBP.hsd1.or.comcast.net","ip":["127.0.0.1/8","::1/128","fe80::1/64","fe80::143e:6265:dc42:297e/64","10.0.0.81/24","2601:1c0:7001:9df:1084:cae3:fdd0:8e21/64","2601:1c0:7001:9df:8884:ca22:5365:71ac/64","2601:1c0:7001:9df::eb2c/64","2601:1c0:7001:9df:5c01:c673:a8d3:d2d/64","fe80::c4bf:a5ff:fe61:1927/64","fe80::4b17:95f6:4b1d:8c88/64","fe80::b5f5:6b71:c879:2670/64"],"kernel_version":"16.5.0","mac":["a0:99:9b:08:ea:df","6a:00:00:67:8f:a0","6a:00:00:67:8f:a1","6a:00:00:67:8f:a0","02:99:9b:08:ea:df","c6:bf:a5:61:19:27","0a:00:27:00:00:00","0a:00:27:00:00:01"],"os":{"family":"darwin","platform":"darwin","name":"Mac OS X","version":"10.12.4","major":10,"minor":12,"patch":4,"build":"16E195"},"timezone":"PST","timezone_offset_sec":-28800,"id":"3793E7AD-D0FB-5BAB-ACFB-D6CC2B1F4AA5"}}} 2019-01-29T18:28:39.288-0800 INFO [beat] instance/beat.go:981 Process info {"system_info": {"process": {"cwd": "/Users/dedemorton/BuildTesting/6.6.0_GA/metricbeat-6.6.0-darwin-x86_64", "exe": "./metricbeat", "name": "metricbeat", "pid": 39195, "ppid": 36935, "start_time": "2019-01-29T18:28:39.164-0800"}}} 2019-01-29T18:28:39.288-0800 INFO instance/beat.go:281 Setup Beat: metricbeat; Version: 6.6.0 2019-01-29T18:28:39.288-0800 WARN [cfgwarn] instance/beat.go:793 BETA: Index lifecycle management is enabled which is in beta. 2019-01-29T18:28:39.288-0800 INFO instance/beat.go:850 Set setup.template.name to 'metricbeat-6.6.0' as ILM is enabled. 2019-01-29T18:28:39.288-0800 INFO instance/beat.go:856 Set setup.template.pattern to 'metricbeat-6.6.0-*' as ILM is enabled. 2019-01-29T18:28:39.288-0800 INFO instance/beat.go:863 Set settings.index.lifecycle.rollover_alias in template to metricbeat-6.6.0 as ILM is enabled. 2019-01-29T18:28:39.288-0800 INFO instance/beat.go:868 Set settings.index.lifecycle.name in template to beats-default-policy as ILM is enabled. 2019-01-29T18:28:39.288-0800 INFO instance/beat.go:806 Set output.elasticsearch.index to 'metricbeat-6.6.0' as ILM is enabled. 2019-01-29T18:28:39.289-0800 INFO elasticsearch/client.go:165 Elasticsearch url: http://localhost:9200 2019-01-29T18:28:39.296-0800 INFO elasticsearch/client.go:721 Connected to Elasticsearch version 6.6.0 2019-01-29T18:28:42.329-0800 INFO add_cloud_metadata/add_cloud_metadata.go:319 add_cloud_metadata: hosting provider type not detected. 2019-01-29T18:28:42.330-0800 INFO elasticsearch/client.go:165 Elasticsearch url: http://localhost:9200 2019-01-29T18:28:42.331-0800 INFO [publisher] pipeline/module.go:110 Beat name: Rhodas-MBP.hsd1.or.comcast.net 2019-01-29T18:28:42.331-0800 INFO instance/beat.go:403 metricbeat start running. 2019-01-29T18:28:42.331-0800 INFO [monitoring] log/log.go:117 Starting metrics logging every 30s 2019-01-29T18:28:42.333-0800 INFO cfgfile/reload.go:150 Config reloader started 2019-01-29T18:28:42.336-0800 INFO cfgfile/reload.go:205 Loading of config files completed. 2019-01-29T18:28:43.338-0800 INFO pipeline/output.go:95 Connecting to backoff(elasticsearch(http://localhost:9200)) 2019-01-29T18:28:43.344-0800 INFO elasticsearch/client.go:721 Connected to Elasticsearch version 6.6.0 2019-01-29T18:28:43.345-0800 ERROR instance/ilm.go:80 Failed to check for alias: 403 Forbidden: : 2019-01-29T18:28:45.204-0800 ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://localhost:9200)): Connection marked as failed because the onConnect callback failed: failed to check for alias: 403 Forbidden: 2019-01-29T18:28:45.204-0800 INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 1 reconnect attempt(s) 2019-01-29T18:28:45.205-0800 INFO elasticsearch/client.go:721 Connected to Elasticsearch version 6.6.0 2019-01-29T18:28:45.211-0800 INFO template/load.go:83 Loading template for Elasticsearch version: 6.6.0 2019-01-29T18:28:45.211-0800 INFO template/load.go:85 Existing template will be overwritten, as overwrite is enabled. 2019-01-29T18:28:45.356-0800 INFO template/load.go:146 Elasticsearch template with name 'metricbeat-6.6.0' loaded 2019-01-29T18:28:45.356-0800 INFO instance/beat.go:894 Template successfully loaded. 2019-01-29T18:28:45.357-0800 ERROR instance/ilm.go:80 Failed to check for alias: 403 Forbidden: : 2019-01-29T18:28:47.992-0800 ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://localhost:9200)): Connection marked as failed because the onConnect callback failed: failed to check for alias: 403 Forbidden: 2019-01-29T18:28:47.992-0800 INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 2 reconnect attempt(s) 2019-01-29T18:28:47.993-0800 INFO elasticsearch/client.go:721 Connected to Elasticsearch version 6.6.0 2019-01-29T18:28:47.997-0800 INFO template/load.go:83 Loading template for Elasticsearch version: 6.6.0 2019-01-29T18:28:47.997-0800 INFO template/load.go:85 Existing template will be overwritten, as overwrite is enabled. 2019-01-29T18:28:48.142-0800 INFO template/load.go:146 Elasticsearch template with name 'metricbeat-6.6.0' loaded 2019-01-29T18:28:48.142-0800 INFO instance/beat.go:894 Template successfully loaded. 2019-01-29T18:28:48.143-0800 ERROR instance/ilm.go:80 Failed to check for alias: 403 Forbidden: : 2019-01-29T18:28:52.644-0800 ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://localhost:9200)): Connection marked as failed because the onConnect callback failed: failed to check for alias: 403 Forbidden: 2019-01-29T18:28:52.644-0800 INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 3 reconnect attempt(s) 2019-01-29T18:28:52.646-0800 INFO elasticsearch/client.go:721 Connected to Elasticsearch version 6.6.0 2019-01-29T18:28:52.651-0800 INFO template/load.go:83 Loading template for Elasticsearch version: 6.6.0 2019-01-29T18:28:52.651-0800 INFO template/load.go:85 Existing template will be overwritten, as overwrite is enabled. 2019-01-29T18:28:52.801-0800 INFO template/load.go:146 Elasticsearch template with name 'metricbeat-6.6.0' loaded 2019-01-29T18:28:52.801-0800 INFO instance/beat.go:894 Template successfully loaded. 2019-01-29T18:28:52.802-0800 ERROR instance/ilm.go:80 Failed to check for alias: 403 Forbidden: : 2019-01-29T18:29:02.257-0800 ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://localhost:9200)): Connection marked as failed because the onConnect callback failed: failed to check for alias: 403 Forbidden: 2019-01-29T18:29:02.257-0800 INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 4 reconnect attempt(s) 2019-01-29T18:29:02.258-0800 INFO elasticsearch/client.go:721 Connected to Elasticsearch version 6.6.0 2019-01-29T18:29:02.262-0800 INFO template/load.go:83 Loading template for Elasticsearch version: 6.6.0 2019-01-29T18:29:02.262-0800 INFO template/load.go:85 Existing template will be overwritten, as overwrite is enabled. 2019-01-29T18:29:02.404-0800 INFO template/load.go:146 Elasticsearch template with name 'metricbeat-6.6.0' loaded 2019-01-29T18:29:02.404-0800 INFO instance/beat.go:894 Template successfully loaded. 2019-01-29T18:29:02.405-0800 ERROR instance/ilm.go:80 Failed to check for alias: 403 Forbidden: : 2019-01-29T18:29:12.335-0800 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":149,"time":{"ms":149}},"total":{"ticks":1016,"time":{"ms":1016},"value":1016},"user":{"ticks":867,"time":{"ms":867}}},"info":{"ephemeral_id":"0bca9f4e-8e2d-47de-9765-eabc7b4940a2","uptime":{"ms":33098}},"memstats":{"gc_next":12916816,"memory_alloc":9750840,"memory_total":419584472,"rss":43175936}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"read":{"bytes":3479},"type":"elasticsearch","write":{"bytes":355037}},"pipeline":{"clients":6,"events":{"active":83,"filtered":1,"published":83,"retry":132,"total":84}}},"metricbeat":{"system":{"cpu":{"events":3,"success":3},"filesystem":{"events":4,"success":4},"fsstat":{"events":1,"success":1},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":42,"success":42},"process":{"events":24,"success":24},"process_summary":{"events":3,"success":3},"uptime":{"events":1,"success":1}}},"system":{"cpu":{"cores":8},"load":{"1":2.4341,"15":2.5713,"5":2.6079,"norm":{"1":0.3043,"15":0.3214,"5":0.326}}}}}} 2019-01-29T18:29:21.657-0800 ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://localhost:9200)): Connection marked as failed because the onConnect callback failed: failed to check for alias: 403 Forbidden: 2019-01-29T18:29:21.657-0800 INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 5 reconnect attempt(s) 2019-01-29T18:29:21.658-0800 INFO elasticsearch/client.go:721 Connected to Elasticsearch version 6.6.0 2019-01-29T18:29:21.664-0800 INFO template/load.go:83 Loading template for Elasticsearch version: 6.6.0 2019-01-29T18:29:21.664-0800 INFO template/load.go:85 Existing template will be overwritten, as overwrite is enabled. 2019-01-29T18:29:21.809-0800 INFO template/load.go:146 Elasticsearch template with name 'metricbeat-6.6.0' loaded 2019-01-29T18:29:21.809-0800 INFO instance/beat.go:894 Template successfully loaded. 2019-01-29T18:29:21.809-0800 ERROR instance/ilm.go:80 Failed to check for alias: 403 Forbidden: :

dedemorton picture dedemorton on 30 Jan 2019

Here's the role I created for my testing (it didn't work):

POST _xpack/security/role/metricbeat_writer { "cluster": ["manage_index_templates","monitor","manage_ilm"], "indices": [ { "names": [ "metricbeat-*" ], "privileges": ["write","create_index","manage_ilm"] } ] }

Here's my output config:

output.elasticsearch: # Array of hosts to connect to. hosts: ["localhost:9200"] username: "metricbeat_internal" password: "MYPASSWORD" ilm.enabled: true setup.template.overwrite: true

dedemorton picture dedemorton on 30 Jan 2019

I suspect that you may also need "view_index_metadata" in your indices privileges. We can confirm on our side too.

jakelandis picture jakelandis on 30 Jan 2019

@jakelandis Adding view_index_metadata got me a more complete message, but still errors:

ERROR instance/ilm.go:103 Error creating alias with write index: 403 Forbidden: {"error": {"root_cause":[{"type":"security_exception","reason":"action [indices:admin/aliases] is unauthorized for user [metricbeat_internal]"}],"type":"security_exception","reason":"action [indices:admin/aliases] is unauthorized for user [metricbeat_internal]"},"status":403},....

I tried granting the manage index privilege, and and that seemed to work (haven't verified that the rollover index gets created, but I suspect it will). Not sure manage is restrictive enough, tho.

dedemorton picture dedemorton on 30 Jan 2019

Lee recommended these privileges via email:

````
POST _xpack/security/role/ilm
{
"cluster": [
// To allow creation or deleteion of policies
"manage_ilm"
],
"indices": [
{
"names": [
"ilm-",
// needed for accessing the shrunken indices post-shrink
"shrink-ilm-"
],
"privileges": [
// To actually create the index (initial creation, rollover, shrink)
"create_index",
// needed to manage aliases for rollover
// also for updating settings (allocation, read only, etc)
"manage"
// For writing to the index/alias
"write",
// For explain/retry/remove of policy
"manage_ilm"]
}
]
}

````

dedemorton picture dedemorton on 31 Jan 2019

I've labeled this as a blocker because without this users will find setting up beats correctly near impossible in 7.0

andrewvc picture andrewvc on 8 Feb 2019

FWIW I've had luck with

POST _xpack/security/role/heartbeat_writer
{
  "cluster": ["manage_index_templates", "monitor", "manage_ilm"],
  "indices": [
    {
      "names": [ "heartbeat-*" ], 
      "privileges": ["write","create_index"]
    }
  ]
}

I'm n ot an index rules expert though.

andrewvc picture andrewvc on 8 Feb 2019

@andrewvc There's already a PR open here. Someone just need to review it. https://github.com/elastic/beats/pull/10449

dedemorton picture dedemorton on 9 Feb 2019

Closed by #10449

dedemorton picture dedemorton on 11 Feb 2019

I think I hit an issue with this today. It resulted in:

2019-04-26T18:03:22.337+0200    ERROR   pipeline/output.go:100  Failed to connect to backoff(elasticsearch(https://XXX)): Connection marked as failed because the onConnect callback failed: failed to check for alias 'metricbeat': (status=403) : 403 Forbidden:

I fixed it by adjusting the index name to metricbeat* rather than metricbeat-* and adding the manage privilege.
The alias is called metricbeat so the permission does not match with the dash at the end.

jakommo picture jakommo on 26 Apr 2019
👍4

@jakommo Thanks for your input. I have a new security PR in progress that should be available for final review today. I'm pushing a bunch of changes to the PR today. I'll make sure the PR addresses your concerns. https://github.com/elastic/beats/pull/11329

dedemorton picture dedemorton on 26 Apr 2019
❤1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

exekias picture exekias  路  3Comments
geekpete picture geekpete  路  3Comments
ptrlv picture ptrlv  路  3Comments
nicpenning picture nicpenning  路  3Comments
tsg picture tsg  路  4Comments