Influxdb: cannot start influxdb

[airyland]

The service just stop two days ago and I couldn't start or restart it. The status always returned FAIL. After I upgrade to 0.9.5, it still failed but was with some logs:

sudo service influxdb restart

2015/11/18 03:05:36 InfluxDB starting, version 0.9.5-rc3, branch 0.9.5, commit a589f941e93078a4a093806d7336ee903d32f23a, built 2015-11-17T00:42:10+0000
2015/11/18 03:05:36 Go version go1.4.2, GOMAXPROCS set to 2
2015/11/18 03:05:36 Using configuration at: /etc/influxdb/influxdb.conf
[metastore] 2015/11/18 03:05:36 Using data dir: /var/opt/influxdb/meta
[metastore] 2015/11/18 03:05:36 Skipping cluster join: already member of cluster: nodeId=1 raftEnabled=true peers=[localhost:8088]
[metastore] 2015/11/18 03:05:36 Node at localhost:8088 [Follower]
[metastore] 2015/11/18 03:05:37 Node at localhost:8088 [Leader]. peers=[localhost:8088]
[metastore] 2015/11/18 03:05:37 spun up monitoring for 1
[store] 2015/11/18 03:05:37 Using data dir: /var/opt/influxdb/data
[metastore] 2015/11/18 03:05:37 Updated node id=1 hostname=localhost:8088
[registration] 2015/11/18 03:05:39 registration service terminating
[retention] 2015/11/18 03:05:39 retention policy enforcement terminating
[monitor] 2015/11/18 03:05:39 shutting down monitor system
[handoff] 2015/11/18 03:05:39 shutting down hh service
[subscriber] 2015/11/18 03:05:39 closed service
run: open server: open tsdb store: failed to open shard 102: load metadata index: open /var/opt/influxdb/wal/bz_shop/default/102/000000.meta: permission denied

And when I ran influxd

influxd run --config /etc/influxdb/influxdb.conf

 8888888           .d888 888                   8888888b.  888888b.
   888            d88P"  888                   888  "Y88b 888  "88b
   888            888    888                   888    888 888  .88P
   888   88888b.  888888 888 888  888 888  888 888    888 8888888K.
   888   888 "88b 888    888 888  888  Y8bd8P' 888    888 888  "Y88b
   888   888  888 888    888 888  888   X88K   888    888 888    888
   888   888  888 888    888 Y88b 888 .d8""8b. 888  .d88P 888   d88P
 8888888 888  888 888    888  "Y88888 888  888 8888888P"  8888888P"

2015/11/18 03:14:01 InfluxDB starting, version 0.9.5-rc3, branch 0.9.5, commit a589f941e93078a4a093806d7336ee903d32f23a, built 2015-11-17T00:42:10+0000
2015/11/18 03:14:01 Go version go1.4.2, GOMAXPROCS set to 2
2015/11/18 03:14:01 Using configuration at: /etc/influxdb/influxdb.conf
[metastore] 2015/11/18 03:14:01 Using data dir: /var/opt/influxdb/meta
[metastore] 2015/11/18 03:14:01 Node at localhost:8088 [Follower]
[metastore] 2015/11/18 03:14:01 Skipping cluster join: already member of cluster: nodeId=1 raftEnabled=true peers=[localhost:8088]
[metastore] 2015/11/18 03:14:02 Node at localhost:8088 [Leader]. peers=[localhost:8088]
[metastore] 2015/11/18 03:14:02 spun up monitoring for 1
[store] 2015/11/18 03:14:02 Using data dir: /var/opt/influxdb/data
[metastore] 2015/11/18 03:14:02 Updated node id=1 hostname=localhost:8088
[wal] 2015/11/18 03:14:04 WAL starting with 30720 ready series size, 0.50 compaction threshold, and 52428800 partition size threshold
[wal] 2015/11/18 03:14:04 WAL writing to /var/opt/influxdb/wal/bz_shop/default/102
[wal] 2015/11/18 03:14:05 WAL starting with 30720 ready series size, 0.50 compaction threshold, and 52428800 partition size threshold
[wal] 2015/11/18 03:14:05 WAL writing to /var/opt/influxdb/wal/bz_shop/default/1539
[wal] 2015/11/18 03:14:06 WAL starting with 30720 ready series size, 0.50 compaction threshold, and 52428800 partition size threshold
[wal] 2015/11/18 03:14:06 WAL writing to /var/opt/influxdb/wal/bz_shop/default/2473
.
.
.
Killed

[otoolep]

@benbjohnson -- any idea why this would happen? Apart from some simple permission problem?

@airyland -- can you show us your directory and file tree at that point? With full permission details etc?

[benbjohnson]

@otoolep It looks like /var/opt/influxdb/wal/bz_shop/default/102/000000.meta doesn't have read & write permission for the user running influxd.

[otoolep]

Yep, it looks as simple as that.

@airyland -- any chance you ran influxd as a different user at some point?

[airyland]

@otoolep Actually I only use user root.. Before the error came out there was no any log for the failure of starting.

[airyland]

I use "chmod" to fix the permission and still failed. So why did it just terminate the service without any reason?

[metastore] 2015/11/18 11:06:15 Node at localhost:8088 [Follower]
[metastore] 2015/11/18 11:06:17 Node at localhost:8088 [Leader]. peers=[localhost:8088]
[metastore] 2015/11/18 11:06:17 spun up monitoring for 1
[store] 2015/11/18 11:06:17 Using data dir: /var/opt/influxdb/data
[metastore] 2015/11/18 11:06:17 Updated node id=1 hostname=localhost:8088
[registration] 2015/11/18 11:06:17 registration service terminating
[retention] 2015/11/18 11:06:17 retention policy enforcement terminating
[monitor] 2015/11/18 11:06:17 shutting down monitor system
[handoff] 2015/11/18 11:06:17 shutting down hh service
[subscriber] 2015/11/18 11:06:17 closed service

[airyland]

I upgraded to 0.9.6 and ran "influxd run ..." and it worked...
But service influxdb restart still failed.

So I don't know what happened..

[yuokada]

this issue happened in my enviroment.
My env is CnetOS6.7, InfluxDB v0.9.5-rc2

$ rpm -q  influxdb
influxdb-0.9.5-0.1.rc2.x86_64

I run influxdb by root user only, too.

[rossmcdonald]

@airyland I believe your permissions are incorrect due running InfluxDB as root. Can you send us the output from the following command?

sudo find /var/opt/influxdb -user root

It is also highly recommended that you _not_ run InfluxDB as the root user.

[yuokada]

In passing, my results is here.

$ sudo find /var/lib/influxdb -user root
/var/lib/influxdb/wal/_internal/monitor/10/000000.meta
/var/lib/influxdb/wal/_internal/monitor/1/000000.meta
/var/lib/influxdb/wal/stress/default/11/000000.meta
/var/lib/influxdb/wal/mydb/default/6/000000.meta
/var/lib/influxdb/wal/mydb/default/7/000000.meta
/var/lib/influxdb/wal/mydb/default/9/000000.meta
/var/lib/influxdb/wal/mydb/default/5/000000.meta
/var/lib/influxdb/wal/mydb/default/8/000000.meta

I change these files owner to influxdb and server running fine.

[airyland]

@rossmcdonald

root@iseekr:~# sudo find /var/opt/influxdb -user root
/var/opt/influxdb/wal/bz_bbs/default/49/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/33/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/9/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/23/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/7/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/25/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/514/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/18/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/1535/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/2484/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/32/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/20/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/26/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/16/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/24/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/8/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/1/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/2470/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/35/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/5/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/13/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/6/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/29/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/3/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/22/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/34/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/4/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/36/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/12/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/19/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/28/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/15/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/2/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/31/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/17/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/30/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/27/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/14/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/11/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/21/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/10/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/2551/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/2542/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/519/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/2529/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/2577/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/1540/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/96/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/2474/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/100/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/2488/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/2497/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/2565/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/94/000000.meta
/var/opt/influxdb/wal/mome/default/2505/000000.meta
/var/opt/influxdb/wal/mome/default/2503/000000.meta
/var/opt/influxdb/wal/mome/default/2504/000000.meta
/var/opt/influxdb/wal/mome/default/2511/000000.meta
/var/opt/influxdb/wal/mome/default/2506/000000.meta
/var/opt/influxdb/wal/mome/default/2533/000000.meta
/var/opt/influxdb/wal/mome/default/2510/000000.meta
/var/opt/influxdb/wal/mome/default/2507/000000.meta
/var/opt/influxdb/wal/mome/default/2508/000000.meta
/var/opt/influxdb/wal/mome/default/2509/000000.meta
/var/opt/influxdb/wal/mome/default/2544/000000.meta
/var/opt/influxdb/wal/bz_shop/default/2576/000000.meta
/var/opt/influxdb/wal/bz_shop/default/2528/000000.meta
/var/opt/influxdb/wal/bz_shop/default/2550/000000.meta
/var/opt/influxdb/wal/bz_shop/default/2496/000000.meta
/var/opt/influxdb/wal/bz_shop/default/2564/000000.meta
/var/opt/influxdb/wal/bz_shop/default/102/000000.meta
/var/opt/influxdb/wal/bz_shop/default/2473/000000.meta
/var/opt/influxdb/wal/bz_shop/default/1539/000000.meta
/var/opt/influxdb/wal/bz_shop/default/99/000000.meta
/var/opt/influxdb/wal/bz_shop/default/518/000000.meta
/var/opt/influxdb/wal/bz_shop/default/97/000000.meta
/var/opt/influxdb/wal/bz_shop/default/2487/000000.meta
/var/opt/influxdb/wal/bz_shop/default/2541/000000.meta
/var/opt/influxdb/wal/test/default/2555/000000.meta
/var/opt/influxdb/wal/test/default/39/000000.meta
/var/opt/influxdb/wal/test/default/52/000000.meta
/var/opt/influxdb/wal/test/default/2574
/var/opt/influxdb/wal/test/default/2574/000000.meta
/var/opt/influxdb/wal/_internal/monitor/2578/000000.meta
/var/opt/influxdb/wal/_internal/monitor/2571/000000.meta
/var/opt/influxdb/wal/_internal/monitor/2573/000000.meta
/var/opt/influxdb/wal/_internal/monitor/2572/000000.meta
/var/opt/influxdb/wal/bz_umeng/default/528/000000.meta
/var/opt/influxdb/wal/bz_umeng/default/529/000000.meta
/var/opt/influxdb/wal/mome_activity/default/2483/000000.meta
/var/opt/influxdb/wal/mome_activity/default/2482/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/59/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/62/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/63/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/65/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/516/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/2543/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/64/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/60/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/61/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/2495/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/2486/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/2563/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/1537/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/58/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/2472/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/2527/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/2552/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/2575/000001.meta
/var/opt/influxdb/wal/mome1/default/2514/000000.meta
/var/opt/influxdb/wal/mome1/default/2513/000000.meta
/var/opt/influxdb/wal/mome1/default/2515/000000.meta
/var/opt/influxdb/wal/mome1/default/2517/000000.meta
/var/opt/influxdb/wal/mome1/default/2519/000000.meta
/var/opt/influxdb/wal/mome1/default/2520/000000.meta
/var/opt/influxdb/wal/mome1/default/2516/000000.meta
/var/opt/influxdb/wal/mome1/default/2518/000000.meta
/var/opt/influxdb/wal/mome1/default/2512/000000.meta
/var/opt/influxdb/wal/game_pintu/default/2559/000000.meta
/var/opt/influxdb/wal/game_pintu/default/2556/000000.meta
/var/opt/influxdb/wal/game_pintu/default/2567/000000.meta
/var/opt/influxdb/data/test/default/2574

[rossmcdonald]

@airyland Your permissions are incorrect, as those files need to be owned by the influxdb user if you would like to run InfluxDB as a service. To fix, you should just be able to run:

chown -R influxdb:influxdb /var/opt/influxdb

[airyland]

@rossmcdonald I see. Thanks!