Graylog2-server: 2.0.0 failed to send join request to master connect_timeout 127.0.0.1:9350

[naisanza]

Problem description

I'm trying to figure out what's not configured right after going through the documentation for Manual Setup

I have Graylog, Mongodb, and Elasticsearch installed in its own separate container. Mongodb and Elasticsearch both are running, and there are no networking issues and are all in the same subnet.

The error in the log:

2016-04-28 12:31:19,577 INFO : org.elasticsearch.discovery.zen - [graylog-9e3b63b4-7946-4321-bd6c-6190b8a7d36e] failed to send join request to master [{Vertigo}{YEOqVxzVQdiHqVrw-gpx8g}{10.0.3.236}{10.0.3.236:9300}], reason [RemoteTransportException[[Vertigo][10.0.3.236:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-9e3b63b4-7946-4321-bd6c-6190b8a7d36e][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]

graylog.conf

node_id_file = /etc/graylog/server/node-id
password_secret = ***
root_password_sha2 = ***
plugin_dir = plugin
rest_listen_uri = http://127.0.0.1:12900/
web_listen_uri = http://0.0.0.0:9000/
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_cluster_name = graylog
elasticsearch_discovery_zen_ping_unicast_hosts = 10.0.3.236:9300
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = data/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://10.0.3.45/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_auto_load = grok-patterns.json

graylog.conf.txt
graylog.log.txt
elasticsearch-graylog.log.txt
mongodb.log.txt

Steps to reproduce the problem

1. OracleJava8.tgz extracted and running
2. Mongodb.tgz extracted and running
3. Elasticsearch.tgz extracted and running
4. Graylog.tgz copied; error occurs when ran
   

   Environment

• Graylog Version: 2.0.0
• Elasticsearch Version: 2.3.2
• MongoDB Version: 3.2.5
• Operating System: 14.04.4
• Browser version: Chrome

[joschi]

@naisanza Make sure that the embedded Elasticsearch node in Graylog is announcing the correct IP address of the system. The address 127.0.0.1:9350 is obviously only reachable from within the same machine.

Take a look at the elasticsearch_network_* settings in the Graylog configuration file: https://github.com/Graylog2/graylog2-server/blob/2.0.0/misc/graylog.conf#L187-L194

On a side note, your rest_listen_uri also looks wrong (the Graylog REST API must be accessible for your web browser for the web interface to work).

[naisanza]

@joschi I would have thought elasticsearch_discovery_zen_ping_unicast_hosts would have taken cared of that

Also, is the embedded Elasticsearch node essential, can it be disabled and just use the node provided by elasticsearch_discovery_zen_ping_unicast_hosts?

[joschi]

@naisanza elasticsearch_discovery_zen_ping_unicast_hosts and elasticsearch_network_host are two different things. The first one defines which Elasticsearch nodes Graylog should connect to, the second one defines which network interface to bind to so that _the other_ Elasticsearch nodes can connect to Graylog (or rather to the embedded Elasticsearch client node).

Also, is the embedded Elasticsearch node essential

Yes, it is. Graylog is running an embedded Elasticsearch client node (i. e. no master and no data) to connect to the Elasticsearch cluster.

[kroepke]

We are using GitHub issues for tracking bugs in Graylog itself, but this doesn't look like it. Please post this issue to our public mailing list or join the #graylog channel on freenode IRC.

Thank you!