Beats: Heartbeat RBAC for publishing requires additional priv

[DanRoscigno]

The required privs are published in docs, but it seems like one is missing (indices:admin/create) which I __think__ should not be required when using ILM.

• Version: 7.5.0

• Operating System: ESS for ES and Kibana, Debian for Heartbeat

• Steps to Reproduce:

  • Create roles for setup and publish
  • Create user for setup and publish
  • Edit heartbeat.yml to specify cloud.id, cloud.auth, and setup.ilm.check_exists: false
  • Run setup
  • Run sudo heartbeat -e

Error message:

(status=403): {
"type":"security_exception","reason":"action [indices:admin/create] is unauthorized for user [beat
-writer]"}

Relevant heartbeat.yml config:

setup.ilm.check_exists: false

cloud.id: "Spaces750:dXMtd2VzdDExxxNwLmNsb3VkLmVzLmlvJDc0NTljZGRmZDE5NDQ2YjRhODE1M2YxMmQ4YmU1YWY0JDRmOTNiMzY5ZjJkNjQyMGU4NTE1NjgzZGJhMjc0ZGJj"
cloud.auth: "${ES_USER}:${ES_USER_PASSWORD}"

Role for publishing user:

{
  "beats_writer" : {
    "cluster" : [
      "monitor",
      "cluster:admin/ingest/pipeline/get"
    ],
    "indices" : [
      {
        "names" : [
          "filebeat-*",
          "metricbeat-*",
          "heartbeat-*",
          "packetbeat-*",
          "auditbeat-*"
        ],
        "privileges" : [
          "create_doc"
        ],
        "field_security" : {
          "grant" : [
            "*"
          ],
          "except" : [ ]
        },
        "allow_restricted_indices" : false
      }
    ],
    "applications" : [ ],
    "run_as" : [ ],
    "metadata" : { },
    "transient_metadata" : {
      "enabled" : true
    }
  }
}

User beat-writer:

{
  "beat-writer" : {
    "username" : "beat-writer",
    "roles" : [
      "beats_writer"
    ],
    "full_name" : "Beat writer",
    "email" : "",
    "metadata" : { },
    "enabled" : true
  }
}

[faec]

Thanks for reporting! It looks like the privilege you're talking about is already in the documentation you linked? In the final row of the table of privileges needed for the writer role, I see "create_index on heartbeat-* indices", but this doesn't seem to be present in the role configuration you include here. I believe this is the privilege the indices:admin/create is asking for. You might want to double check the entries in that table, as others are missing too (e.g. read_ilm).

[DanRoscigno]

Hi @faec , yes, but: The docs say "Create daily indices when connecting to clusters that do not support ILM. Not needed when using ILM." and I am using ILM, ILM is on by default since 7.0. The Filebeat docs say the same thing, and create_index is NOT needed for Filebeat.

read_ilm should never be needed in my use case, as I have setup.ilm.check_exists: false as noted above.

[DanRoscigno]

@faec Maybe this is related to another issue I am seeing with heartbeat. Running heartbeat setup should probably be setting up the index pattern heartbeat-*. Instead, it is setting up the index pattern UptimeIndexPattern which does not seem to work with anything other than the Uptime UI (for example, you cannot use it in Discover or Lens). Maybe since the index pattern is "wrong" the Beat does not realize that ILM is enabled, and it is trying to manage the indices itself?

[faec]

Hmm, you're right... I'm not sure if this is a bug in the beat or the docs, or if some other context might be affecting it. I'm going to check with someone who knows more about Heartbeat internals.

[DanRoscigno]

Thanks @faec , I feel strongly that this is a code bug, and not a doc bug. In my opinion ILM and RBAC config should be common code across the Beats, so Heartbeat and Filebeat should work the same. You might include this Metricbeat bug in your discussion, it is very similar (and is mis-categorized as a doc issue!!)

[dedemorton]

@DanRoscigno I did a quick test in 7.5.1 (default configuration) with setup.ilm.check_exists: false and the documented privileges worked for me. I was able to write events to the heartbeat index.

[DanRoscigno]

You are correct @dedemorton , the docs are right. It appears that there is a difference in the behavior when running beatname setup between heartbeat and the rest. Filebeat and Metricbeat do not seem to skip creating an index during setup if the config file has setup.ilm.check_exists set to false, but Heartbeat does. When this happens (or doesn't happen!) and Heartbeat is run to publish data there is no index, and none is created because the writer user does not have the indices:admin/create priv.

To make things work I have to:

• create the setup role and setup user
• create the publish role and publish user (remembering to not edit the config when I read the instructions to set setup.ilm.check_exists: false)
• run setup
• edit the config file to set setup.ilm.check_exists: false
• run heartbeat

Is this the pattern that you followed to get it to work?

[andrewvc]

@DanRoscigno interesting, we don't tend to use setup very often because heartbeat no longer provides dashboards or similar. Interestingly our setup command must go through an alternate code path than just running heartbeat directly against ES (which does setup ILM correctly). Will dive in later today.

[DanRoscigno]

Thanks @andrewvc , how do you get the index setup if you do not run setup? When you do have a look, could you please look into creating the heartbeat-* index pattern instead of UptimeIndexPattern (or in addition to if you need UptimeIndexPattern)?

I wonder if altering the standard code path to skip the dashboards etc. for Heartbeat would be more future proof?

[dedemorton]

@andrewvc The Add data tutorial still recommends running the setup command, so perhaps you should change that if you don't expect users to run the command. I like the idea of maintaining consistency with other beats, though, because I think uses will expect the commands to work the same (even if they don't necessarily need to run them).

From the add data tutorial:

Start Heartbeat

The setup command loads the Kibana index pattern.

./heartbeat setup
./heartbeat -e

[DanRoscigno]

Interestingly our setup command must go through an alternate code path than just running heartbeat directly against ES (which does setup ILM correctly). Will dive in later today.

@andrewvc If you mean running heartbeat without setup first configures ILM properly, I never tried that, but it requires that heartbeat be run with more privs than the beat writer role, right? I am trying to get people to stop using a superuser to run Beats as it is a huge security issue.

[andrewvc]

Yeah, it does. It's what we tend to do in dev/test, hence why we never
noticed this. It's on my list to tackle :)

On Fri, Jan 17, 2020 at 1:05 PM Dan Roscigno [e-mail]
wrote:

Interestingly our setup command must go through an alternate code path
than just running heartbeat directly against ES (which does setup ILM
correctly). Will dive in later today.

@andrewvc https://github.com/andrewvc If you mean running heartbeat
without setup first configures ILM properly, I never tried that, but it
requires that heartbeat be run with more privs than the beat writer role,
right? I am trying to get people to stop using a superuser to run Beats as
it is a huge security issue.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/elastic/beats/issues/15341?email_source=notifications&email_token=AABACY3HVPHWV5VWPMVUZG3Q6H6QFA5CNFSM4KDIAFKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJIVGMI#issuecomment-575755057,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABACY442ZU2W2F72DW4NUDQ6H6QFANCNFSM4KDIAFKA
.

[DanRoscigno]

If you want help with an automated build of a test env for k8s let me know.

[elasticmachine]

Pinging @elastic/uptime (:uptime)

[andrewvc]

@DanRoscigno / @urso am I correct in saying after investigation that this is a libbeat issue and not heartbeat specific? If so, do you mind if I remove the heartbeat/uptime label?

[urso]

The setting setup.ilm.check_exists interoperates with setup.ilm.overwrite. Setting up ILM takes multiple steps. If overwrite is true, these steps are always executed. If overwrite is false we need to check via ES API if the alias, policy, and template already exists. If the setup.ilm.check_exists is set to false, then Beats assumes that the ILM is already configured. Setting check_exists and overwrite to false should force Beats to not take any action (no ILM policy, ailas, template will be configured). The setup command will respect the setup.ilm.overwrite setting with Beats 7.6.0. Before 7.6.0 overwrite was always set to true by <beatname> setup by accident. Which means I would expect all Beats behave as we're seeing here.

For the scenario of separate user for setup and publishing each, personally (I practice this myself) I would recommend to have different configuration files. One for setup and one for running Beats (I normally put output configs on a separate config file). But as most people drive Beats from one single config file one might prefer to use -E.

The config file should state:

setup.ilm.check_exists: false   # Beats assume ILM is already configured
setup.ilm.overwrite: false      # Beats must not update an existing ILM,alias setup

This ensures that no ILM, policy, alias, template related API will be used by Beats.

With these settings one __MUST__ always use either setup.ilm.check_exists: true or setup.ilm.overwrite: true with setup. If overwrite is true we always overwrite all resources. Normally you don't want this. Better use setup.ilm.check_exists: true. If both settings are false, then setup will do nothing and we will get aforementioned error. Maybe we should clarify in the docs section that users should run: <beatname> setup -E setup.ilm.check_exists=true.

[DanRoscigno]

I have been behind, I will test this next week and then write a doc PR.

[DanRoscigno]

Based on what I read above from @urso and what I am seeing I would propose:

1. Add a new file filebeat.setup.yml that contains:

#============================== Setup ILM =====================================
# Disable the check for an existing lifecycle policy. The default is false. If
# you disable this check, set setup.ilm.overwrite: true so the lifecycle policy
# can be installed.
setup.ilm.check_exists: true

# Overwrite the lifecycle policy at startup. The default is false.
setup.ilm.overwrite: false

# Configure index lifecycle management (ILM). These settings create a write
# alias and add additional settings to the index template. When ILM is enabled,
# output.elasticsearch.index is ignored, and the write alias is used to set the
# index name.

# Enable ILM support. Valid values are true, false, and auto. When set to auto
# (the default), the Beat uses index lifecycle management when it connects to a
# cluster that supports ILM; otherwise, it creates daily indices.
#setup.ilm.enabled: auto

# Set the prefix used in the index lifecycle write alias name. The default alias
# name is 'filebeat-%{[agent.version]}'.
#setup.ilm.rollover_alias: "filebeat"

# Set the rollover index pattern. The default is "%{now/d}-000001".
#setup.ilm.pattern: "{now/d}-000001"

# Set the lifecycle policy name. The default policy name is
# 'filebeat-%{[agent.version]}'.
#setup.ilm.policy_name: "mypolicy"

# The path to a JSON file that contains a lifecycle policy configuration. Used
# to load your own lifecycle policy.
#setup.ilm.policy_file:

#============================= Elastic Cloud ==================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
# cloud.id: 

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
# cloud.auth: 

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

1. In the docs specify that running setup is separate from sending records into the Elastic Stack and should be treated differently. A separate configuration file and a separate user with different privileges than the user that publishes logs and metrics to the stack should be used to setup Filebeat. The user should edit the file filebeat.setup.yml to configure setup. In this file they may enable (default) or disable ILM, include a custom ILM policy, change the index name that Filebeat will write to, etc. After editing the file setup can be run using this command:

filebeat setup  -c ./filebeat.setup.yml

Note: Using the Filebeat keystore or passing parameters on the commandline is documented at .....

1. We should modify the default filebeat.yml to include:

# Disable the check for an existing lifecycle policy. The default is false. If
# you disable this check, set setup.ilm.overwrite: true so the lifecycle policy
# can be installed.
#setup.ilm.check_exists: false

1. And we should also modify the Filebeat code to set setup.ilm.check_exists: false in the code (I think that at least in 7.5 it is actually true by default, even though filebeat.reference.yml says false See config.go#L122)

[DanRoscigno]

Once I finish testing the above on Filebeat I will then do Metricbeat and Heartbeat and the rest. BTW, here are my roles and users for completeness:
Filebeat-7.5.2-Roles-Users.txt

[DanRoscigno]

:( I had to give the beat-writer role cluster:admin/ingest/pipeline/put . I wonder if this is because my filebeat.setup.yml does not specify a modules dir?

[DanRoscigno]

@urso A new question:

Note: I am using Elasticsearch and Kibana 7.6.0-SNAPSHOT and Filebeat 7.5.2

My filebeat.yml

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

processors:
  - add_host_metadata:
      netinfo.enabled: true
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

I used these commands:

sudo ./filebeat modules enable system

sudo ./filebeat setup -E setup.ilm.check_exists="true" \
  -E output.elasticsearch.hosts="[localhost:9200]" \
  -E setup.kibana.host="localhost:5601" \
  -E output.elasticsearch.username="beat-setup" \
  -E output.elasticsearch.password="beat-setup"

sudo ./filebeat -e -E setup.ilm.check_exists="false" \
  -E output.elasticsearch.hosts="[localhost:9200]" \
  -E output.elasticsearch.username="beat-writer" \
  -E output.elasticsearch.password="beat-writer" \
  -c ~/Downloads/Urso/filebeat.yml

Everything worked fine, but I realized that I needed to enable another module and I then ran:

sudo ./filebeat modules enable auditd

sudo ./filebeat -e -E setup.ilm.check_exists="false" \
  -E output.elasticsearch.hosts="[localhost:9200]" \
  -E output.elasticsearch.username="beat-writer" \
  -E output.elasticsearch.password="beat-writer" \
  -c ~/Downloads/Urso/filebeat.yml

I am now seeing this error:

"security_exception","reason":"action [cluster:admin/ingest/pipeline/put] is unauthorized for user [beat-writer]"},"status":403}

I think it makes sense if the original run of setup only loaded ingest pipelines for enabled modules. Is that correct?

If so, then I would add a note to the text about running setup and say something like "setup has to be re-run after enabling new modules"

[DanRoscigno]

this is fixed in 7.6.0, shall I close it @andrewvc ?

[andrewvc]

Yes, closing