Beats: Packetbeat setup is throwing an internal error

[srikwit]

For confirmed bugs, please report:

• Version: 7.9.1
• Operating System: Ubuntu 16.04.6 LTS
• Discuss Forum URL: https://github.com/elastic/beats/issues/20716
• Steps to Reproduce:
• Setup Elastic security
• Configure Packetbeat with SSL enabled and verification none for both Elasticsearch as well as Kibana

Loading dashboards (Kibana must be running and reachable)
2020-09-12T19:49:10.966+0300    INFO    kibana/client.go:119    Kibana url: https://IP:5601
2020-09-12T19:49:10.966+0300    WARN    [tls]   tlscommon/tls_config.go:83  SSL/TLS verifications disabled.
2020-09-12T19:49:10.966+0300    WARN    [tls]   tlscommon/tls_config.go:83  SSL/TLS verifications disabled.
2020-09-12T19:49:11.225+0300    INFO    kibana/client.go:119    Kibana url: https://IP:5601
2020-09-12T19:49:11.225+0300    WARN    [tls]   tlscommon/tls_config.go:83  SSL/TLS verifications disabled.
2020-09-12T19:49:11.226+0300    WARN    [tls]   tlscommon/tls_config.go:83  SSL/TLS verifications disabled.
2020-09-12T19:49:11.373+0300    ERROR   instance/beat.go:951    Exiting: 1 error: error loading index pattern: returned 500 to import file: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}
Exiting: 1 error: error loading index pattern: returned 500 to import file: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[andrewkroh]

This is a HTTP 500 error code while communicating to Kibana. Please verify that Kibana is up and healthy then check your Kibana logs to see if there are more details about the error.

[srikwit]

Yes I can verify that the Kibana server is up and I had received the following error log in it:
{"type":"error","@timestamp":"2020-09-17T07:51:28Z","tags":[],"pid":17007,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:69:19)\n at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:170:34)\n at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"?force=true","query":{"force":"true"},"pathname":"/api/kibana/dashboards/import","path":"/api/kibana/dashboards/import?force=true","href":"/api/kibana/dashboards/import?force=true"},"message":"Internal Server Error"}

I do not understand why the protocol, auth, host, port are showing up as null because they are configured in the packetbeat.yml.

[nhova]

Hi @srikwit

Did you get this resolved?

I am facing the same issue today.

{"type":"error","@timestamp":"2020-09-29T04:26:10Z","tags":[],"pid":1430483,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:69:19)\n at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:170:34)\n at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"?force=true","query":{"force":"true"},"pathname":"/api/kibana/dashboards/import","path":"/api/kibana/dashboards/import?force=true","href":"/api/kibana/dashboards/import?force=true"},"message":"Internal Server Error"}

[srikwit]

Yes this did get resolved. I need to configure the Kibana options in my packetbeat. These are of the type setup.kibana.*

You can refer the following link for fixing it: https://www.elastic.co/guide/en/beats/packetbeat/current/setup-kibana-endpoint.html

[srikwit]

Sorry team, I would like to reopen this issue. I am still facing issues with this setup. The only reason it worked for me because I had used a superuser credential. The packetbeat user with the following permissions did not work for me:

Cluster - monitor, manage, manage_index_templates, manage_ingest_pipelines
Indices - create, create_index, manage, manage_ilm (I have even tried with all permissions).

[marc-gr]

@srikwit after configuring your kibana options under setup.kibana.*, are you getting the same 500 error, or is it a different one now?

[srikwit]

Yes @marc-gr , the error is the same. The difference was the permissions of the user. With superuser I have no issues but with the packetbeat user I am facing issues.

[andrewstucki]

@srikwit: See the docs about setting up packetbeat integration. The setup command requires admin-level permissions to create the index lifecycle management policies and index templates. I'm assuming your packetbeat user is the user intended for publishing? If so, I wouldn't be surprised if it was missing manage permissions on the index.

If for some reason your permissions appear to be set properly for the user, could you post them here? That said, that Kibana 500 error is pretty useless for diagnosing the root issue, so might need to open an issue in Kibana about it.

EDIT:

Actually, looking at the path in Kibana returning the 500, I'm thinking you're likely missing the built-in kibana_admin role for the user you're trying to run setup as.

[zbioe]

Nice, I just added the kibana_admin role as you said, and for me work's like a spell.
Thank's