Beats: Elastic Agent: enrolling multiple agents at once fail

Created on 7 May 2020 · 12Comments · Source: elastic/beats

I can't enroll 20 agents at once (docker-compose scale):

elasticsearch_1     | {"@timestamp":"2020-05-07T10:54:32.045Z", "log.level": "WARN", "message":"Authentication to realm default_native failed - Password authentication failed for fleet_enroll", "service.name":"ES_ECS","process.thread.name":"elasticsearch[f72ed16d6744][get][T#6]","log.logger":"org.elasticsearch.xpack.security.authc.AuthenticationService","type":"server","cluster.uuid":"NF8Jlw4-RUqKMjCZJxFw9g","node.id":"2zAg0k0kSs6WnwTnXcFnkQ","node.name":"f72ed16d6744","cluster.name":"docker-cluster"}
elastic-agent_5     |  * base: centos.hitme.net.pl
elastic-agent_5     |  * epel: epel.mirror.serveriai.lt
elastic-agent_5     |  * extras: mirror-pl.kielcetechnologypark.net
elastic-agent_5     |  * updates: mirror-pl.kielcetechnologypark.net
kibana_1            | {"type":"log","@timestamp":"2020-05-07T10:54:32+00:00","tags":["error","plugins","security","api-key"],"pid":6,"message":"Failed to create API key: [security_exception] unable to authenticate user [fleet_enroll] for REST request [/_security/api_key], with { header={ WWW-Authenticate={ 0=\"ApiKey\" & 1=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } }"}
elasticsearch_1     | {"@timestamp":"2020-05-07T10:54:32.075Z", "log.level": "INFO", "message":"updated role [fleet_enroll]", "service.name":"ES_ECS","process.thread.name":"elasticsearch[f72ed16d6744][management][T#4]","log.logger":"org.elasticsearch.xpack.security.action.role.TransportPutRoleAction","type":"server","cluster.uuid":"NF8Jlw4-RUqKMjCZJxFw9g","node.id":"2zAg0k0kSs6WnwTnXcFnkQ","node.name":"f72ed16d6744","cluster.name":"docker-cluster"}
kibana_1            | {"type":"error","@timestamp":"2020-05-07T10:54:30+00:00","tags":[],"pid":6,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:132:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:86:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:81:17)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:160:34)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/api/ingest_manager/fleet/setup","path":"/api/ingest_manager/fleet/setup","href":"/api/ingest_manager/fleet/setup"},"message":"Internal Server Error"}
100   334  100   278  100    56    138     27  0:00:02  0:00:02 --:--:--   138
elastic-agent_18    |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
elastic-agent_18    |                                  Dload  Upload   Total   Spent    Left  Speed
elasticsearch_1     | {"@timestamp":"2020-05-07T10:54:32.243Z", "log.level": "INFO", "message":"updated user [fleet_enroll]", "service.name":"ES_ECS","process.thread.name":"elasticsearch[f72ed16d6744][management][T#3]","log.logger":"org.elasticsearch.xpack.security.action.user.TransportPutUserAction","type":"server","cluster.uuid":"NF8Jlw4-RUqKMjCZJxFw9g","node.id":"2zAg0k0kSs6WnwTnXcFnkQ","node.name":"f72ed16d6744","cluster.name":"docker-cluster"}
100    22  100    22    0     0     68      0 --:--:-- --:--:-- --:--:--    68
elastic-agent_16    |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
elastic-agent_16    |                                  Dload  Upload   Total   Spent    Left  Speed
elasticsearch_1     | {"@timestamp":"2020-05-07T10:54:32.375Z", "log.level": "WARN", "message":"Authentication to realm default_native failed - Password authentication failed for fleet_enroll", "service.name":"ES_ECS","process.thread.name":"elasticsearch[f72ed16d6744][get][T#6]","log.logger":"org.elasticsearch.xpack.security.authc.AuthenticationService","type":"server","cluster.uuid":"NF8Jlw4-RUqKMjCZJxFw9g","node.id":"2zAg0k0kSs6WnwTnXcFnkQ","node.name":"f72ed16d6744","cluster.name":"docker-cluster"}
kibana_1            | {"type":"log","@timestamp":"2020-05-07T10:54:32+00:00","tags":["error","plugins","security","api-key"],"pid":6,"message":"Failed to create API key: [security_exception] unable to authenticate user [fleet_enroll] for REST request [/_security/api_key], with { header={ WWW-Authenticate={ 0=\"ApiKey\" & 1=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } }"}
elasticsearch_1     | {"@timestamp":"2020-05-07T10:54:32.470Z", "log.level": "INFO", "message":"updated role [fleet_enroll]", "service.name":"ES_ECS","process.thread.name":"elasticsearch[f72ed16d6744][management][T#2]","log.logger":"org.elasticsearch.xpack.security.action.role.TransportPutRoleAction","type":"server","cluster.uuid":"NF8Jlw4-RUqKMjCZJxFw9g","node.id":"2zAg0k0kSs6WnwTnXcFnkQ","node.name":"f72ed16d6744","cluster.name":"docker-cluster"}
kibana_1            | {"type":"error","@timestamp":"2020-05-07T10:54:32+00:00","tags":[],"pid":6,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:132:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:86:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:81:17)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:160:34)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/api/ingest_manager/fleet/enrollment-api-keys","path":"/api/ingest_manager/fleet/enrollment-api-keys","href":"/api/ingest_manager/fleet/enrollment-api-keys"},"message":"Internal Server Error"}
100   320  100   278  100    42    845    127 --:--:-- --:--:-- --:--:--   847
elastic-agent_2     |   Installing : oniguruma-5.9.5-3.el7.x86_64                                 1/2
elastic-agent_18    | {"isInitialized":true}{"statusCode":500,"error":"Internal Server Error","message":"[security_exception] unable to authenticate user [fleet_enroll] for REST request [/_security/api_key], with { header={ WWW-Authenticate={ 0=\"ApiKey\" & 1=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } }"}The Elastic Agent is currently in Experimental and should not be used in production
elastic-agent_11    | Resolving Dependencies
elastic-agent_11    | --> Running transaction check
elastic-agent_11    | ---> Package jq.x86_64 0:1.6-1.el7 will be installed
elastic-agent_11    | --> Processing Dependency: libonig.so.2()(64bit) for package: jq-1.6-1.el7.x86_64
package-registry_1  | 2020/05/07 10:54:32 source.ip: 127.0.0.1:51072, url.original: /
elasticsearch_1     | {"@timestamp":"2020-05-07T10:54:32.789Z", "log.level": "INFO", "message":"updated user [fleet_enroll]", "service.name":"ES_ECS","process.thread.name":"elasticsearch[f72ed16d6744][management][T#1]","log.logger":"org.elasticsearch.xpack.security.action.user.TransportPutUserAction","type":"server","cluster.uuid":"NF8Jlw4-RUqKMjCZJxFw9g","node.id":"2zAg0k0kSs6WnwTnXcFnkQ","node.name":"f72ed16d6744","cluster.name":"docker-cluster"}
elastic-agent_11    | --> Running transaction check
elastic-agent_11    | ---> Package oniguruma.x86_64 0:5.9.5-3.el7 will be installed
elastic-agent_11    | --> Finished Dependency Resolution
elasticsearch_1     | {"@timestamp":"2020-05-07T10:54:33.040Z", "log.level": "WARN", "message":"Authentication to realm default_native failed - Password authentication failed for fleet_enroll", "service.name":"ES_ECS","process.thread.name":"elasticsearch[f72ed16d6744][generic][T#8]","log.logger":"org.elasticsearch.xpack.security.authc.AuthenticationService","type":"server","cluster.uuid":"NF8Jlw4-RUqKMjCZJxFw9g","node.id":"2zAg0k0kSs6WnwTnXcFnkQ","node.name":"f72ed16d6744","cluster.name":"docker-cluster"}
elasticsearch_1     | {"@timestamp":"2020-05-07T10:54:33.042Z", "log.level": "WARN", "message":"Authentication to realm default_native failed - Password authentication failed for fleet_enroll", "service.name":"ES_ECS","process.thread.name":"elasticsearch[f72ed16d6744][generic][T#6]","log.logger":"org.elasticsearch.xpack.security.authc.AuthenticationService","type":"server","cluster.uuid":"NF8Jlw4-RUqKMjCZJxFw9g","node.id":"2zAg0k0kSs6WnwTnXcFnkQ","node.name":"f72ed16d6744","cluster.name":"docker-cluster"}
elasticsearch_1     | {"@timestamp":"2020-05-07T10:54:33.042Z", "log.level": "WARN", "message":"Authentication to realm default_native failed - Password authentication failed for fleet_enroll", "service.name":"ES_ECS","process.thread.name":"elasticsearch[f72ed16d6744][get][T#2]","log.logger":"org.elasticsearch.xpack.security.authc.AuthenticationService","type":"server","cluster.uuid":"NF8Jlw4-RUqKMjCZJxFw9g","node.id":"2zAg0k0kSs6WnwTnXcFnkQ","node.name":"f72ed16d6744","cluster.name":"docker-cluster"}
kibana_1            | {"type":"log","@timestamp":"2020-05-07T10:54:33+00:00","tags":["error","plugins","security","api-key"],"pid":6,"message":"Failed to create API key: [security_exception] unable to authenticate user [fleet_enroll] for REST request [/_security/api_key], with { header={ WWW-Authenticate={ 0=\"ApiKey\" & 1=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } }"}
kibana_1            | {"type":"log","@timestamp":"2020-05-07T10:54:33+00:00","tags":["error","plugins","security","api-key"],"pid":6,"message":"Failed to create API key: [security_exception] unable to authenticate user [fleet_enroll] for REST request [/_security/api_key], with { header={ WWW-Authenticate={ 0=\"ApiKey\" & 1=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } }"}
kibana_1            | {"type":"log","@timestamp":"2020-05-07T10:54:33+00:00","tags":["error","plugins","security","api-key"],"pid":6,"message":"Failed to create API key: [security_exception] unable to authenticate user [fleet_enroll] for REST request [/_security/api_key], with { header={ WWW-Authenticate={ 0=\"ApiKey\" & 1=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } }"}

It's easy to reproduce using the testing cluster with agent support: https://github.com/elastic/integrations/pull/10

beta1 Ingest Management bug

Source

mtojek

Most helpful comment

It shouldn't be a problem considering the suggestion above.

Yes it should not, it's more a question of concerns, I think the agent should only call the enroll API and checkin after.

nchaulet on 11 May 2020

👍2

All 12 comments

Pinging @elastic/ingest-management (Team:Ingest Management)

elasticmachine on 7 May 2020

I took a quick look at it, and looks like the docker elastic agent, is doing the fleet setup each time an agent boot, this call should not be made for each agent

nchaulet on 8 May 2020

yes it seems setup is enabled by default here https://github.com/elastic/observability-test-environments/blob/master/helm-charts/elastic-agent/values.yaml
it should not be with option to override.
should not be setup idempotent?

michalpristas on 11 May 2020

Agree that the setup should not be made each time. But even if it is, it should be idempotent as @michalpristas mentioned and skip it quickly.

ruflin on 11 May 2020

I think currently it's not a problem, if you call the setup multiple times sequentially, but you can have some race conditions if all the agents call it at the same time.

nchaulet on 11 May 2020

I think currently it's not a problem, if you call the setup multiple times sequentially, but you can have some race conditions if all the agents call it at the same time.

So it's not idempotent :)

mtojek on 11 May 2020

👍1

@nchaulet If the setup is called twice, will it create a different password the second time?

ruflin on 11 May 2020

Yes it will change the password of the fleet user

nchaulet on 11 May 2020

Assuming we will be in a world in the near future with an API key instead of a user, this would probably be unexpected. If a new API key is created, I would expect the old one to be invalidated which potentially would invalidate all existing API keys. So if the API key already exists, setup should just return already exists. If a user wants to reset / overwrite it, an additional param or different API endpoint should be needed.

ruflin on 11 May 2020

Yes it will change the password of the fleet user, we can make the change to not do it if the password is already set.

The way fleet setup is consumed in the UI, we first call
GET /fleet/setup -> isInitialized: true|false
and if it's not initialized we call POST /fleet/setup on user click

@ruflin Yes we can make the change to make POST more idempotent, but still the agent should avoid to call it multiple time

In the ideal scenario the docker setup should more something like:

setup kibana and ES
do ingest and fleet setup
enroll N agents

nchaulet on 11 May 2020

So if the API key already exists, setup should just return already exists. If a user wants to reset / overwrite it, an additional param or different API endpoint should be needed.

I think the API should just return same results. If the key exists, it shouldn't fail an error, but simply return the existing one. We don't need flags for that.