Graylog2-server: Is there a way to use ElasticSearch from AWS with Graylog?

[fernandowiek]

I think is a dumb question but, is there a way to use ElasticSearch from AWS with Graylog installed in a EC2?
I dont know very well how this components work together... if anyone can explain me, i will apreciate.
Regards.

[LordMike]

You most likely can.
In the omnibus install of Graylog (the debian packages), I find my config here: /etc/graylog/server

In this config, theres a number of "elasticsearch" entries:

elasticsearch_analyzer = standard
elasticsearch_cluster_name = graylog2
elasticsearch_config_file = /etc/elasticsearch/elasticsearch.yml
elasticsearch_discovery_zen_ping_unicast_hosts = 127.0.0.1:9300
elasticsearch_index_prefix = graylog2
elasticsearch_max_number_of_indices = 250
elasticsearch_max_size_per_index = 1073741824
elasticsearch_replicas = 0
elasticsearch_shards = 4

The ones you want to meddle with are the elasticsearch_discovery_zen_ping_unicast_hosts and elasticsearch_cluster_name.

What will happen is this:

• Your AWS ES instance will have a number of nodes, in some cluster name (f.ex. graylog2).
• Your Graylog server instances, will all be Elasticsearch nodes too, only they won't store data.

Beware of security if you're going across the internet or on some other public network. Elasticsearch is unauthenticated, meaning if I get access to its interface (the :9300 port), I can DELETE all your data.

[joschi]

Graylog is currently joining the Elasticsearch cluster as a regular client (i. e. no master, no data node) which is not possible with the AWS Elasticsearch service as far as I know. We might come up with a solution to this in the future, but for now you'd have to setup and manage your own Elasticsearch cluster on AWS.

In the future, please post these kind of questions to our mailing list or to #graylog on Freenode. Thanks.

[LordMike]

Oh, AWS Elasticsearch is a higher level of management?
I assumed it was a simple ES endpoint - but then they probably skipped that due to the security issues.

[fernandowiek]

Thanks for all.
Regards.

[goya]

:(

[ktamas77]

How about Graylog 2.0?
Could it use AWS's ElasticSearch Service?

[joschi]

@ktamas77 No.

[davidchua]

Is this still an issue?

[joschi]

@davidchua Yes.

[iain17]

Sorry I don't follow. Why is this an issue?

[joschi]

@iain17 See https://github.com/Graylog2/graylog2-server/issues/1473#issuecomment-147110521

[schmod]

Is it correct to assume that the switch to the HTTP API in Graylog 2.3 will finally allow this to work?

[joschi]

@schmod Yes.

[sandeep89]

Can someone help me out here @joschi

1. We are talking about Graylog 2.2.3 and not Graylog 2.3?
2. Also I have done all the default setting in aws es and changed below

elasticsearch_discovery_zen_ping_unicast_hosts = <aws es host>
elasticsearch_http_enabled = true
elasticsearch_network_host = <aws es host>

It seems to be crashing, what did I miss?

[joschi]

@sandeep89 We are using GitHub issues for tracking bugs in Graylog itself, but this doesn't look like one. Please post this issue to our discussion forum or join the #graylog channel on freenode IRC.

Thank you!

[GunnarHelliesen]

@joschi Does your "yes" reply to @schmod above mean that Graylog 2.3 will be able to talk to the AWS Elasticsearch Service directly? If so, is there documentation (even source) you could point me to for how to configure this? I have 2.3RC2 installed and ready.

Thanks!

[joschi]

@GunnarHelliesen Yes.

Simply provide the URIs of your AWS Elasticsearch nodes in the elasticsearch_hosts setting.

Take note that the proprietary authentication mechanism of the AWS Elasticsearch Service is not supported by Graylog.

[GunnarHelliesen]

@joschi That was easy! Confirmed to be working, I have 2.3RC on an EC2 instance talking to the Amazon Elasticsearch Service. The authentication problem is easily worked around by writing an AWS policy based on IP addresses. Being able to use an AWS IAM key pair would be an awesome feature addition down the line, though.

Quick follow-up question: With this configuration, is there any need to install Elasticsearch locally on the Graylog node? I.e., is it still a prerequisite for Graylog when using the Amazon Elasticsearch Service?

Thanks!

[joschi]

Quick follow-up question: With this configuration, is there any need to install Elasticsearch locally on the Graylog node?

No.

[schmod]

The authentication problem is easily worked around by writing an AWS policy based on IP addresses.

Yeah, this seems to be one of the only ways to do it, given that AWS ES has absolutely no integration with VPC (which is kind of crazy for an Amazon service).

Another alternative would be to install a proxy service on your Graylog server (or somewhere inside the same private subnet) that will sign requests for you.

A quick search turns up two options (I've used the second):
https://github.com/abutaha/aws-es-proxy
https://www.npmjs.com/package/aws-es-kibana

[joschi]

@schmod @GunnarHelliesen Feel free and encouraged to contribute a guide to use Graylog with the AWS Elasticsearch Service on the Graylog Marketplace (akin to the syslog guides).

[GunnarHelliesen]

@joschi I might just do that, but for a new problem I've come across. Graylog displays a red banner with the following error whenever I go to the Graylog Search page, or click to view messages from an input:

Loading field information failed with status: cannot GET http://n.n.n.n:9000/api/system/fields (500)

When I try to load that URL, I get the following error from ES:

{"message":"Couldn't read cluster state for indices graylog_*\n\n{\"Message\":\"Your request: '/_cluster/state/metadata' is not allowed.\"}","details":["{\"Message\":\"Your request: '/_cluster/state/metadata' is not allowed.\"}"]}

I've googled and found that this is a problem with the AWS ES Service, it doesn't support this endpoint.

How critical is this? Does this mean that Graylog can't really work with the AWS ES Service after all?

Thanks!

[danvaida]

@GunnarHelliesen Although it doesn't have any particularities regarding AWS' ES service, I successfully used my Graylog Ansible role the past weekend. It works seamlessly with AWS ES & Graylog 2.3.0 RC2.
Hope that helps.
https://github.com/danvaida/graylog-ansible-role/tree/2.3.0

[bernd]

@GunnarHelliesen Make sure you use at least AWS ES version 5.3. AWS added access to a partial /_cluster/state/metadata endpoint which is enough to make the Graylog /api/system/fields call work.

[bernd]

I am locking this conversation. Please use our discussion forum if you have any further questions about AWS ES integration or open a new issue if you run into a bug. Thank you!