Influxdb: Error upgrading to InfluxDB version 1.4.0 and 1.4.1 from 1.3.7

[binary0111]

Bug report

__System info:__
InfluxDB Version: 1.3.7 -> 1.4.0 or 1.4.1
OS: Custom Linux Distro using standard Linux binaries for InfluxDB

__Steps to reproduce:__

1. Shut down existing InfluxDB instance
2. Install new binaries
3. Restart InfluxDB

__Expected behavior:__

When upgrading InfluxDB in the past, things just worked without requiring explicit backup and restore of the database. This included moving to the tsi index from the inmen index in 1.3.x.

__Actual behavior:__

The following errors show up in the log:

[I] 2017-11-14T05:51:25Z Failed to open shard: 39: [shard 39] incompatible tsi1 index MANIFEST service=store
[I] 2017-11-14T05:51:25Z Failed to open shard: 37: [shard 37] incompatible tsi1 index MANIFEST service=store
[I] 2017-11-14T05:51:25Z Failed to open shard: 44: [shard 44] incompatible tsi1 index MANIFEST service=store
[I] 2017-11-14T05:51:25Z Failed to open shard: 38: [shard 38] incompatible tsi1 index MANIFEST service=store
[I] 2017-11-14T05:51:25Z Failed to open shard: 40: [shard 40] incompatible tsi1 index MANIFEST service=store
[I] 2017-11-14T05:51:25Z Failed to open shard: 41: [shard 41] incompatible tsi1 index MANIFEST service=store
[I] 2017-11-14T05:51:25Z Failed to open shard: 42: [shard 42] incompatible tsi1 index MANIFEST service=store
[I] 2017-11-14T05:51:25Z Failed to open shard: 43: [shard 43] incompatible tsi1 index MANIFEST service=store
.......
[I] 2017-11-14T05:52:01Z write failed for shard 44: [shard 44] incompatible tsi1 index MANIFEST service=write
[I] 2017-11-14T05:52:02Z write failed for shard 44: [shard 44] incompatible tsi1 index MANIFEST service=write
[I] 2017-11-14T05:52:03Z write failed for shard 44: [shard 44] incompatible tsi1 index MANIFEST service=write

__Additional info:__

I have been able to reproduce the exact same issue on multiple hosts, all using version 1.3.7 with the tsi index before the upgrade. Problem exists for both 1.4.0 and 1.4.1. If I delete the newest shard (in this case 44), things start working but historical data is still lost, which makes sense given the errors above. Other possibly relevant configuration:

cache-max-memory-size = "100m"
cache-snapshot-memory-size = "20m"

[benbjohnson]

@binary0111 I apologize for the issue. We missed a breaking change note in the CHANGELOG from this code change: https://github.com/influxdata/influxdb/pull/8857

We've upgraded our internal hash algorithm but unfortunately it is not backwards compatible with the previous TSI format. To upgrade you will need to rebuild your TSI indexes:

1. Stop influxd
2. Remove the index directory in your shard.
3. Run influx_inspect inmem2tsi using the shard's data and WAL directories for -datadir and -waldir, respectively.
4. Repeat for each TSI enabled shard.
5. Start influxd

[derkgort]

this could help some people in "mass" update:

in dir /var/lib/influxdb

remove old index dirs:

find -type d | grep index | xargs rm -Rvf

run on all dirs:

for i in data/*/*/*; do echo $i; done | awk -F '/' '{print $2,$3,$4}' | 
while read db rp sh; do 
echo $db; 
influx_inspect inmem2tsi -datadir data/$db/$rp/$sh -waldir wal/$db/$rp/$sh;  
done

[psargentatnewsbank]

Adding to derkgort's post above. After running those commands, run
chown -R influxdb:influxdb data