Graylog2-server: new install - graylog just keeps on restarting....

Created on 10 May 2016  ·  7Comments  ·  Source: Graylog2/graylog2-server

Problem description

Fresh install via docs, just wanting to explore graylog a bit. despite following docs, I cant' get graylog to start - well actually stay started - as it keeps on just restarting itself. Not sure wtf is going on.

Environment

Centos 7.2 
 # java -version 
openjdk version "1.8.0_65"
OpenJDK Runtime Environment (build 1.8.0_65-b17)
OpenJDK 64-Bit Server VM (build 25.65-b01, mixed mode)
# rpm -qa | grep -E 'graylog|mongo|elastic'
mongodb-org-mongos-2.6.12-1.x86_64
mongodb-org-shell-2.6.12-1.x86_64
mongodb-org-server-2.6.12-1.x86_64
mongodb-mms-monitoring-agent-2.4.1.108-1.x86_64
graylog-server-2.0.0-6.noarch
graylog-2.0-repository-1-1.noarch
mongodb-org-tools-2.6.12-1.x86_64
elasticsearch-2.3.2-1.noarch
mongodb-mms-backup-agent-2.4.0.156-1.x86_64
mongodb-org-2.6.12-1.x86_64

output from /var/log/graylog-server/server.log

# systemctl restart graylog-server
# tail -f /var/log/graylog-server/server.log 
2016-05-10T16:01:10.798-04:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2016-05-10T16:01:10.830-04:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2016-05-10T16:01:10.866-04:00 INFO  [cluster] No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, all=[ServerDescription{address=localhost:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2016-05-10T16:01:10.901-04:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:191}] to localhost:27017
2016-05-10T16:01:10.902-04:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[2, 6, 12]}, minWireVersion=0, maxWireVersion=2, maxDocumentSize=16777216, roundTripTimeNanos=746486}
2016-05-10T16:01:10.908-04:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:192}] to localhost:27017
2016-05-10T16:01:11.098-04:00 INFO  [NodeId] Node ID: a524ff09-acf9-423c-84dd-f539ff75015a
2016-05-10T16:01:11.180-04:00 INFO  [node] [graylog-a524ff09-acf9-423c-84dd-f539ff75015a] version[2.3.1], pid[28564], build[bd98092/2016-04-04T12:25:05Z]
2016-05-10T16:01:11.180-04:00 INFO  [node] [graylog-a524ff09-acf9-423c-84dd-f539ff75015a] initializing ...
2016-05-10T16:01:11.185-04:00 INFO  [plugins] [graylog-a524ff09-acf9-423c-84dd-f539ff75015a] modules [], plugins [graylog-monitor], sites []
2016-05-10T16:01:12.495-04:00 INFO  [node] [graylog-a524ff09-acf9-423c-84dd-f539ff75015a] initialized
2016-05-10T16:01:12.562-04:00 INFO  [Version] HV000001: Hibernate Validator 5.2.4.Final
2016-05-10T16:01:12.694-04:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2016-05-10T16:01:14.505-04:00 INFO  [RulesEngineProvider] No static rules file loaded.
2016-05-10T16:01:14.634-04:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /tmp/GeoLite2-City.mmdb
2016-05-10T16:01:14.639-04:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2016-05-10T16:01:15.186-04:00 INFO  [ServerBootstrap] Graylog server 2.0.0 (2dc6c03) starting up
2016-05-10T16:01:15.186-04:00 INFO  [ServerBootstrap] JRE: Oracle Corporation 1.8.0_65 on Linux 3.10.0-327.el7.x86_64
2016-05-10T16:01:15.186-04:00 INFO  [ServerBootstrap] Deployment: rpm
2016-05-10T16:01:15.187-04:00 INFO  [ServerBootstrap] OS: CentOS Linux 7 (Core) (centos)
2016-05-10T16:01:15.187-04:00 INFO  [ServerBootstrap] Arch: amd64
2016-05-10T16:01:15.191-04:00 WARN  [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2016-05-10T16:01:15.210-04:00 INFO  [PeriodicalsService] Starting 24 periodicals ...
2016-05-10T16:01:15.211-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2016-05-10T16:01:15.214-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
2016-05-10T16:01:15.215-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2016-05-10T16:01:15.217-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [0s], polling every [20s].
2016-05-10T16:01:15.217-04:00 INFO  [node] [graylog-a524ff09-acf9-423c-84dd-f539ff75015a] starting ...
2016-05-10T16:01:15.219-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever.
2016-05-10T16:01:15.220-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2016-05-10T16:01:15.222-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2016-05-10T16:01:15.227-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2016-05-10T16:01:15.232-04:00 INFO  [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2016-05-10T16:01:15.233-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2016-05-10T16:01:15.233-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2016-05-10T16:01:15.234-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2016-05-10T16:01:15.235-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2016-05-10T16:01:15.235-04:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:194}] to localhost:27017
2016-05-10T16:01:15.235-04:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:193}] to localhost:27017
2016-05-10T16:01:15.235-04:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2016-05-10T16:01:15.236-04:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [300s].
2016-05-10T16:01:15.237-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2016-05-10T16:01:15.237-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2016-05-10T16:01:15.238-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2016-05-10T16:01:15.242-04:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:195}] to localhost:27017
2016-05-10T16:01:15.243-04:00 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2016-05-10T16:01:15.245-04:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:197}] to localhost:27017
2016-05-10T16:01:15.245-04:00 INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:196}] to localhost:27017
2016-05-10T16:01:15.246-04:00 INFO  [connection] Opened connection [connectionId{localValue:8, serverValue:198}] to localhost:27017
2016-05-10T16:01:15.283-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2016-05-10T16:01:15.284-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2016-05-10T16:01:15.284-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2016-05-10T16:01:15.287-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2016-05-10T16:01:15.287-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s].
2016-05-10T16:01:15.288-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s].
2016-05-10T16:01:15.290-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2016-05-10T16:01:15.364-04:00 INFO  [transport] [graylog-a524ff09-acf9-423c-84dd-f539ff75015a] publish_address {127.0.0.1:9350}, bound_addresses {[::1]:9350}, {127.0.0.1:9350}
2016-05-10T16:01:15.369-04:00 INFO  [discovery] [graylog-a524ff09-acf9-423c-84dd-f539ff75015a] graylog/At-r2-4YTiSoJCD2HXFBjg
2016-05-10T16:01:15.550-04:00 INFO  [AbstractJerseyService] Enabling CORS for HTTP endpoint
2016-05-10T16:01:18.229-04:00 ERROR [ServiceManager] Service WebInterfaceService [FAILED] has failed in the STARTING state.
javax.ws.rs.ProcessingException: Failed to start Grizzly HTTP server: Address already in use
    at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpServerFactory.createHttpServer(GrizzlyHttpServerFactory.java:299) ~[graylog.jar:?]
    at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpServerFactory.createHttpServer(GrizzlyHttpServerFactory.java:163) ~[graylog.jar:?]
    at org.graylog2.shared.initializers.AbstractJerseyService.setUp(AbstractJerseyService.java:158) ~[graylog.jar:?]
    at org.graylog2.initializers.WebInterfaceService.startUp(WebInterfaceService.java:46) ~[graylog.jar:?]
    at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:60) [graylog.jar:?]
    at com.google.common.util.concurrent.Callables$3.run(Callables.java:100) [graylog.jar:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]
Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_65]
    at sun.nio.ch.Net.bind(Net.java:433) ~[?:1.8.0_65]
    at sun.nio.ch.Net.bind(Net.java:425) ~[?:1.8.0_65]
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[?:1.8.0_65]
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) ~[?:1.8.0_65]
    at org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler.bindToChannelAndAddress(TCPNIOBindingHandler.java:131) ~[graylog.jar:?]
    at org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler.bind(TCPNIOBindingHandler.java:88) ~[graylog.jar:?]
    at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:248) ~[graylog.jar:?]
    at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:228) ~[graylog.jar:?]
    at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:219) ~[graylog.jar:?]
    at org.glassfish.grizzly.http.server.NetworkListener.start(NetworkListener.java:714) ~[graylog.jar:?]
    at org.glassfish.grizzly.http.server.HttpServer.start(HttpServer.java:278) ~[graylog.jar:?]
    at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpServerFactory.createHttpServer(GrizzlyHttpServerFactory.java:296) ~[graylog.jar:?]
    ... 6 more
2016-05-10T16:01:18.234-04:00 ERROR [InputSetupService] Not starting any inputs because lifecycle is: Uninitialized [LB:DEAD]
2016-05-10T16:01:18.242-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.AlertScannerThread].
2016-05-10T16:01:18.243-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.AlertScannerThread] complete, took <0ms>.
2016-05-10T16:01:18.243-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread].
2016-05-10T16:01:18.243-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] complete, took <0ms>.
2016-05-10T16:01:18.243-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ClusterHealthCheckThread].
2016-05-10T16:01:18.243-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ClusterHealthCheckThread] complete, took <0ms>.
2016-05-10T16:01:18.243-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexerClusterCheckerThread].
2016-05-10T16:01:18.243-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexerClusterCheckerThread] complete, took <0ms>.
2016-05-10T16:01:18.243-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRetentionThread].
2016-05-10T16:01:18.243-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRetentionThread] complete, took <0ms>.
2016-05-10T16:01:18.243-04:00 WARN  [BufferSynchronizerService] Elasticsearch is unavailable. Not waiting to clear buffers and caches, as we have no healthy cluster.
2016-05-10T16:01:18.244-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRotationThread].
2016-05-10T16:01:18.244-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRotationThread] complete, took <0ms>.
2016-05-10T16:01:18.244-04:00 INFO  [LogManager] Shutting down.
2016-05-10T16:01:18.244-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.VersionCheckThread].
2016-05-10T16:01:18.244-04:00 INFO  [node] [graylog-a524ff09-acf9-423c-84dd-f539ff75015a] stopping ...
2016-05-10T16:01:18.244-04:00 INFO  [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput
2016-05-10T16:01:18.244-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.VersionCheckThread] complete, took <0ms>.
2016-05-10T16:01:18.244-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ThrottleStateUpdaterThread].
2016-05-10T16:01:18.244-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, took <0ms>.
2016-05-10T16:01:18.244-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2016-05-10T16:01:18.244-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventPeriodical] complete, took <0ms>.
2016-05-10T16:01:18.244-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventCleanupPeriodical].
2016-05-10T16:01:18.244-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventCleanupPeriodical] complete, took <0ms>.
2016-05-10T16:01:18.244-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical].
2016-05-10T16:01:18.246-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical] complete, took <0ms>.
2016-05-10T16:01:18.246-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical].
2016-05-10T16:01:18.246-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] complete, took <0ms>.
2016-05-10T16:01:18.246-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical].
2016-05-10T16:01:18.246-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] complete, took <0ms>.
2016-05-10T16:01:18.246-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread].
2016-05-10T16:01:18.246-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] complete, took <0ms>.
2016-05-10T16:01:18.258-04:00 INFO  [node] [graylog-a524ff09-acf9-423c-84dd-f539ff75015a] stopped
2016-05-10T16:01:18.258-04:00 INFO  [node] [graylog-a524ff09-acf9-423c-84dd-f539ff75015a] closing ...
2016-05-10T16:01:18.260-04:00 INFO  [JournalReader] Stopping.
2016-05-10T16:01:18.263-04:00 INFO  [node] [graylog-a524ff09-acf9-423c-84dd-f539ff75015a] closed
2016-05-10T16:01:18.268-04:00 INFO  [LogManager] Shutdown complete.
2016-05-10T16:01:18.372-04:00 WARN  [discovery] [graylog-a524ff09-acf9-423c-84dd-f539ff75015a] waited for 3s and no initial state was set by the discovery
2016-05-10T16:01:18.374-04:00 ERROR [ServiceManager] Service IndexerSetupService [FAILED] has failed in the STOPPING state.
java.lang.IllegalStateException: Can't move to started state when closed
    at org.elasticsearch.common.component.Lifecycle.canMoveToStarted(Lifecycle.java:114) ~[graylog.jar:?]
    at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:62) ~[graylog.jar:?]
    at org.elasticsearch.node.Node.start(Node.java:291) ~[graylog.jar:?]
    at org.graylog2.initializers.IndexerSetupService.startUp(IndexerSetupService.java:114) ~[graylog.jar:?]
    at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:60) [graylog.jar:?]
    at com.google.common.util.concurrent.Callables$3.run(Callables.java:100) [graylog.jar:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]
2016-05-10T16:01:18.375-04:00 WARN  [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2016-05-10T16:01:22.363-04:00 ERROR [ServiceManager] Service RestApiService [FAILED] has failed in the STOPPING state.
javax.ws.rs.ProcessingException: Failed to start Grizzly HTTP server: Address already in use
    at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpServerFactory.createHttpServer(GrizzlyHttpServerFactory.java:299) ~[graylog.jar:?]
    at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpServerFactory.createHttpServer(GrizzlyHttpServerFactory.java:163) ~[graylog.jar:?]
    at org.graylog2.shared.initializers.AbstractJerseyService.setUp(AbstractJerseyService.java:158) ~[graylog.jar:?]
    at org.graylog2.shared.initializers.RestApiService.startUp(RestApiService.java:65) ~[graylog.jar:?]
    at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:60) [graylog.jar:?]
    at com.google.common.util.concurrent.Callables$3.run(Callables.java:100) [graylog.jar:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]
Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_65]
    at sun.nio.ch.Net.bind(Net.java:433) ~[?:1.8.0_65]
    at sun.nio.ch.Net.bind(Net.java:425) ~[?:1.8.0_65]
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[?:1.8.0_65]
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) ~[?:1.8.0_65]
    at org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler.bindToChannelAndAddress(TCPNIOBindingHandler.java:131) ~[graylog.jar:?]
    at org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler.bind(TCPNIOBindingHandler.java:88) ~[graylog.jar:?]
    at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:248) ~[graylog.jar:?]
    at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:228) ~[graylog.jar:?]
    at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:219) ~[graylog.jar:?]
    at org.glassfish.grizzly.http.server.NetworkListener.start(NetworkListener.java:714) ~[graylog.jar:?]
    at org.glassfish.grizzly.http.server.HttpServer.start(HttpServer.java:278) ~[graylog.jar:?]
    at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpServerFactory.createHttpServer(GrizzlyHttpServerFactory.java:296) ~[graylog.jar:?]
    ... 6 more
2016-05-10T16:01:22.364-04:00 INFO  [ServiceManagerListener] Services are now stopped.
2016-05-10T16:01:22.364-04:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {STARTING=[RestApiService [STARTING], IndexerSetupService [STARTING]], FAILED=[WebInterfaceService [FAILED]]}
    at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:713) ~[graylog.jar:?]
    at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:542) ~[graylog.jar:?]
    at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:299) ~[graylog.jar:?]
    at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:127) [graylog.jar:?]
    at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:209) [graylog.jar:?]
    at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]
2016-05-10T16:01:22.364-04:00 WARN  [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2016-05-10T16:01:22.364-04:00 INFO  [Server] SIGNAL received. Shutting down.
2016-05-10T16:01:22.370-04:00 INFO  [GracefulShutdown] Graceful shutdown initiated.
2016-05-10T16:01:22.370-04:00 INFO  [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2016-05-10T16:01:22.370-04:00 WARN  [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
^C

No other httpd are running 

# service httpd status 
Redirecting to /bin/systemctl status  httpd.service
● httpd.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)
[root@blackbeard ~]# systemctl status httpd.service
● httpd.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

curl elastic search

# curl -X GET http://localhost:9200
{
  "name" : "Super-Adaptoid",
  "cluster_name" : "graylog2",
  "version" : {
    "number" : "2.3.2",
    "build_hash" : "b9e4a6acad4008027e4038f6abed7f7dba346f94",
    "build_timestamp" : "2016-04-21T16:03:47Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}
# curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "graylog2",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
#
question
Source
picture DreadPirateRob
👍2

Most helpful comment

UPDATE: If anyone else ends up having this issue. They were correct. It was another instance of graylog running. Now its running just fine.

There is another possibility, in this example file https://github.com/Graylog2/graylog2-images/blob/07aaa4bfa0b4e7d8e99de53fc493e441202f84e4/docker/config/graylog.conf rest_listen_uri and web_listen_uri use the same port!

rest_listen_uri = http://0.0.0.0:9000/api/
web_listen_uri = http://0.0.0.0:9000/

if web and rest services use the same port then you have this error:

Failed to start Grizzly HTTP server: Address already in use
picture harobed on 30 Aug 2016
👍2

All 7 comments

@DreadPirateRob 

2016-05-10T16:01:18.229-04:00 ERROR [ServiceManager] Service WebInterfaceService [FAILED] has failed in the STARTING state.
javax.ws.rs.ProcessingException: Failed to start Grizzly HTTP server: Address already in use
[...]
2016-05-10T16:01:22.363-04:00 ERROR [ServiceManager] Service RestApiService [FAILED] has failed in the STOPPING state.
javax.ws.rs.ProcessingException: Failed to start Grizzly HTTP server: Address already in use

Make sure that only 1 instance of Graylog is running and that no other process blocks the ports for the web interface (9000) and the Graylog REST API (12900).

We are using GitHub issues for tracking bugs in Graylog itself, but this doesn't look like one. Please post this issue to our public mailing list or join the #graylog channel on freenode IRC.

Thank you!

joschi picture joschi on 11 May 2016

UPDATE: If anyone else ends up having this issue. They were correct. It was another instance of graylog running. Now its running just fine.

DreadPirateRob picture DreadPirateRob on 11 May 2016

UPDATE: If anyone else ends up having this issue. They were correct. It was another instance of graylog running. Now its running just fine.

There is another possibility, in this example file https://github.com/Graylog2/graylog2-images/blob/07aaa4bfa0b4e7d8e99de53fc493e441202f84e4/docker/config/graylog.conf rest_listen_uri and web_listen_uri use the same port!

rest_listen_uri = http://0.0.0.0:9000/api/
web_listen_uri = http://0.0.0.0:9000/

if web and rest services use the same port then you have this error:

Failed to start Grizzly HTTP server: Address already in use
harobed picture harobed on 30 Aug 2016
👍2

@harobed No. This has changed in Graylog 2.1.0 while this issue has been opened for Graylog 2.0.x.

joschi picture joschi on 30 Aug 2016

Hi

I have the same problem. The graylog server restarts and immediately shutdowns because the IndexerSetupService service is in a failed state.

2017-06-12T10:02:10.776+02:00 WARN  [discovery] [graylog-48217bdc-c511-4df1-bd29-bb4ac9cec964] waited for 3s and no initial state was set by the discovery
2017-06-12T10:02:10.778+02:00 ERROR [ServiceManager] Service IndexerSetupService [FAILED] has failed in the STOPPING state.
java.lang.IllegalStateException: Can't move to started state when closed
        at org.elasticsearch.common.component.Lifecycle.canMoveToStarted(Lifecycle.java:114) ~[graylog.jar:?]
        at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:62) ~[graylog.jar:?]
        at org.elasticsearch.node.Node.start(Node.java:291) ~[graylog.jar:?]
        at org.graylog2.initializers.IndexerSetupService.startUp(IndexerSetupService.java:114) ~[graylog.jar:?]
        at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
        at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
2017-06-12T10:02:10.788+02:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {STARTING=[IndexerSetupService [STARTING]], FAILED=[JerseyService [FAILED]]}
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:742) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:555) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:304) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:147) [graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:209) [graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]
2017-06-12T10:02:10.788+02:00 WARN  [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2017-06-12T10:02:10.789+02:00 INFO  [ServiceManagerListener] Services are now stopped.
2017-06-12T10:02:10.799+02:00 INFO  [Server] SIGNAL received. Shutting down.
2017-06-12T10:02:10.809+02:00 INFO  [GracefulShutdown] Graceful shutdown initiated.
2017-06-12T10:02:10.809+02:00 WARN  [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2017-06-12T10:02:10.810+02:00 INFO  [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2017-06-12T10:02:14.812+02:00 INFO  [GracefulShutdown] Goodbye.

My E.S version is :

 curl -X GET http://localhost:9200
{
  "name" : "node-1",
  "cluster_name" : "graylog2",
  "cluster_uuid" : "8C24wnUYSg-CUH2OKPQ19g",
  "version" : {
    "number" : "5.4.1",
    "build_hash" : "2cfe0df",
    "build_date" : "2017-05-29T16:05:51.443Z",
    "build_snapshot" : false,
    "lucene_version" : "6.5.1"
  },
  "tagline" : "You Know, for Search"
}

Thanks for your answer.

BR

cheucheu picture cheucheu on 12 Jun 2017

yum info graylog-server
Installed Packages
Name : graylog-server
Arch : noarch
Version : 2.2.3
Release : 1
Size : 103 M
Repo : installed
From repo : graylog
Summary : Graylog server
URL : https://www.graylog.org/
License : GPLv3
Description : Graylog server

cheucheu picture cheucheu on 12 Jun 2017

@cheucheu Graylog 2.2.x and earlier only supports Elasticsearch 2.x: http://docs.graylog.org/en/2.2/pages/installation.html#system-requirements

joschi picture joschi on 12 Jun 2017
Was this page helpful?
0 / 5 - 0 ratings

Related issues

eroji picture eroji  ·  4Comments
lennartkoopmann picture lennartkoopmann  ·  4Comments
jalogisch picture jalogisch  ·  4Comments
bpbp-boop picture bpbp-boop  ·  4Comments
mhaasEFD picture mhaasEFD  ·  4Comments