Graylog2-server: Unable to access file data/elasticsearch/modules

[lasdem]

Problem description

graylog-server does not start after install

Steps to reproduce the problem

1. Installed all components from repositories with yum (RPMs)
2. graylog server.conf

is_master = true
node_id_file = /etc/graylog/server/node-id
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://127.0.0.1:12900/
web_enable = true
web_listen_uri = http://127.0.0.1:9000/
rotation_strategy = size
elasticsearch_max_size_per_index = 1073741824
elasticsearch_max_number_of_indices = 400
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog2
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_cluster_name = graylog
elasticsearch_node_master = false
elasticsearch_node_data = false
elasticsearch_discovery_zen_ping_multicast_enabled = false
elasticsearch_discovery_zen_ping_unicast_hosts = 127.0.0.1:9300
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog2
mongodb_max_connections = 100
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json

1. elasticsearch.yml

cluster.name: graylog
path.data: /data_elk/elasticsearch/data
path.work: /data_elk/elasticsearch/work
path.logs: /data_elk/elasticsearch/logs
path.plugins: /data_elk/elasticsearch/plugins
network.host: localhost
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["localhost:9300"]

Environment

• openjdk version: "1.8.0_71"
• Graylog Version: v2.0 Beta.1
• Elasticsearch Version: 2.2.1
• MongoDB Version: 3.2.4
• Operating System: Red Hat Enterprise Linux Server release 6.7 (Santiago)
• Browser version: N/A
  

  Log Output

/var/log/graylog-server/server.log

2016-03-25T15:01:18.928+01:00 INFO  [CmdLineTool] Loaded plugins: [Anonymous Usage Statistics 2.0.0-beta.1 [org.graylog.plugins.usagestatistics.UsageStatsPlugin], Pipeline Processor Plugin 1.0.0-alpha.6 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin], Collector 1.0.0-beta.1 [org.graylog.plugins.collector.CollectorPlugin], MapWidgetPlugin 1.0.0-beta.1 [org.graylog.plugins.map.MapWidgetPlugin], ArchivePlugin 1.0.0-beta.1 [com.graylog.plugins.archive.ArchivePlugin]]
2016-03-25T15:01:19.034+01:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Dgraylog2.installation_source=rpm
2016-03-25T15:01:21.310+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2016-03-25T15:01:21.608+01:00 INFO  [LogManager] Loading logs.
2016-03-25T15:01:21.615+01:00 INFO  [LogManager] Logs loading complete.
2016-03-25T15:01:21.690+01:00 INFO  [LogManager] Created log for partition [messagejournal,0] in /var/lib/graylog-server/journal with properties {segment.index.bytes -> 1048576, file.delete.delay.ms -> 60000, segment.bytes -> 104857600, flush.ms -> 60000, delete.retention.ms -> 86400000, index.interval.bytes -> 4096, retention.bytes -> 5368709120, min.insync.replicas -> 0, cleanup.policy -> delete, unclean.leader.election.enable -> true, segment.ms -> 3600000, max.message.bytes -> 2147483647, flush.messages -> 1000000, min.cleanable.dirty.ratio -> 0.5, retention.ms -> 43200000, segment.jitter.ms -> 0}.
2016-03-25T15:01:21.691+01:00 INFO  [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2016-03-25T15:01:21.710+01:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2016-03-25T15:01:21.747+01:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
2016-03-25T15:01:21.780+01:00 INFO  [cluster] No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, all=[ServerDescription{address=localhost:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2016-03-25T15:01:21.823+01:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:16}] to localhost:27017
2016-03-25T15:01:21.824+01:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 4]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=585965}
2016-03-25T15:01:21.830+01:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:17}] to localhost:27017
2016-03-25T15:01:22.072+01:00 INFO  [NodeId] Node ID: 6a3c182b-f24a-48a9-8442-d0be1db7cbf0
2016-03-25T15:01:22.167+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] version[2.2.1], pid[7744], build[d045fc2/2016-03-09T09:38:54Z]
2016-03-25T15:01:22.168+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] initializing ...
2016-03-25T15:01:22.183+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] version[2.2.1], pid[7744], build[d045fc2/2016-03-09T09:38:54Z]
2016-03-25T15:01:22.183+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] initializing ...
2016-03-25T15:01:22.192+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] version[2.2.1], pid[7744], build[d045fc2/2016-03-09T09:38:54Z]
2016-03-25T15:01:22.192+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] initializing ...
2016-03-25T15:01:22.200+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] version[2.2.1], pid[7744], build[d045fc2/2016-03-09T09:38:54Z]
2016-03-25T15:01:22.201+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] initializing ...
2016-03-25T15:01:22.293+01:00 INFO  [Version] HV000001: Hibernate Validator 5.2.4.Final
2016-03-25T15:01:22.446+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] version[2.2.1], pid[7744], build[d045fc2/2016-03-09T09:38:54Z]
2016-03-25T15:01:22.447+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] initializing ...
2016-03-25T15:01:22.460+01:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2016-03-25T15:01:24.572+01:00 INFO  [RulesEngineProvider] No static rules file loaded.
2016-03-25T15:01:24.589+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] version[2.2.1], pid[7744], build[d045fc2/2016-03-09T09:38:54Z]
2016-03-25T15:01:24.589+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] initializing ...
2016-03-25T15:01:24.592+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] version[2.2.1], pid[7744], build[d045fc2/2016-03-09T09:38:54Z]
2016-03-25T15:01:24.593+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] initializing ...
2016-03-25T15:01:24.596+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] version[2.2.1], pid[7744], build[d045fc2/2016-03-09T09:38:54Z]
2016-03-25T15:01:24.596+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] initializing ...
2016-03-25T15:01:24.599+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] version[2.2.1], pid[7744], build[d045fc2/2016-03-09T09:38:54Z]
2016-03-25T15:01:24.599+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] initializing ...
2016-03-25T15:01:24.620+01:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2016-03-25T15:01:24.622+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] version[2.2.1], pid[7744], build[d045fc2/2016-03-09T09:38:54Z]
2016-03-25T15:01:24.622+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] initializing ...
......
2016-03-25T15:01:27.503+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] version[2.2.1], pid[7744], build[d045fc2/2016-03-09T09:38:54Z]
2016-03-25T15:01:27.503+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] initializing ...
2016-03-25T15:01:27.504+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] version[2.2.1], pid[7744], build[d045fc2/2016-03-09T09:38:54Z]
2016-03-25T15:01:27.504+01:00 INFO  [node] [graylog-6a3c182b-f24a-48a9-8442-d0be1db7cbf0] initializing ...
2016-03-25T15:01:27.582+01:00 ERROR [CmdLineTool]

################################################################################

ERROR: Unable to access file data/elasticsearch/modules

Need help?

* Official documentation: http://docs.graylog.org/
* Community support: https://www.graylog.org/community-support/
* Commercial support: https://www.graylog.com/support/

Terminating. :(

################################################################################

[kroepke]

I cannot reproduce this error.
Does any of your Elasticsearch configuration files mention the path data/elasticsearch/modules?

[kroepke]

@lasdem Can you please check the user permissions for the data directory?
It looks like Elasticsearch cannot create or read its data files.

[schwing]

I ran into this moments ago, and I found that permissions were too strict for the /data directory on the server. Graylog was trying to access /data and failing because the graylog user didn't have read access to that directory--even though I have nothing configured to use /data. Allowing the graylog user read access to /data resolved it in my case. I haven't tested if the behavior is the same when the directory doesn't exist.

If you're running SELinux in enforcing mode (noticed you're running RHEL), that might complicate things further, but perhaps this is the same cause.

[kroepke]

Thanks @schwing for the workaround!

We will introduce a common data directory in a future release, see https://github.com/Graylog2/graylog2-server/issues/2057
In the mean time I'll close this as we cannot directly influence the permissions of the data directory right now (as it isn't explicitly configured at the moment).

[lasdem]

Thank you for the information.
My filesystem structure is like this
[root@graylog-test ~]# ls -alh /data
lrwxrwxrwx 1 root root 9 Jul 22 2015 /data -> /data_graylog

[root@graylog-test ~]# ls -alh /data_graylog
total 48K
drwxr-xr-x 9 graylog graylog 4.0K Mar 25 14:49 .
dr-xr-xr-x. 25 root root 4.0K Mar 22 17:17 ..
drwxr-x--- 6 elasticsearch elasticsearch 4.0K Mar 25 14:46 elasticsearch
drwxrwx--- 2 root root 4.0K Apr 13 15:59 install
drwx------ 2 root root 16K Apr 29 2015 lost+found
drwxr-x--- 4 mongod mongod 4.0K Aug 10 2015 mongodb
drwxr-x--- 3 nginx nginx 4.0K Aug 11 2015 nginx

With your feedback I could fix it by
chmod a+r /data/elasticsearch/
chmod a+x /data/elasticsearch/
mkdir /data/elasticsearch/modules
chown elasticsearch.elasticsearch /data/elasticsearch/modules
chmod a+r -R chmod a+r . -R

But I find it strange that graylog even looks in /data/elasticsearch, because there is no configuration to tell it to. Bug still persists in current version (graylog-server-2.0.0-4.beta.3.noarch), but it works thanks to workaround.

[joschi]

But I find it strange that graylog even looks in /data/elasticsearch, because there is no configuration to tell it to.

There are two undocumented settings (elasticsearch_path_home and elasticsearch_path_data, see https://github.com/Graylog2/graylog2-server/blob/2.0.0-beta.3/graylog2-server/src/main/java/org/graylog2/configuration/ElasticsearchConfiguration.java#L46-L50) which control where the embedded Elasticsearch node should write its data.

[lasdem]

Thank you, I will try to set these at my test installation to different paths, because I dont want to have elasticsearch and graylog use the same path.

[msteelepmp]

Another option is to add graylog user to the elasticsearch group.

In /etc/group:
elasticsearch:x:494:graylog

Note your groupid may be different.