Elasticsearch: Upgrade From 1.4.2 to 1.5.1 (Exception: Could not find a state file to recover from)

[wumingwuan]

Hello,
When I upgrading our machine form 1.4.2 to 1.5.1.
copying data/config from 1.4.2 folder to 1.5.1 data/config folder.
when execute elasticsearch.bat, following exception throw and elasticsearch crash.

[2015-04-13 17:50:07,869][WARN ][bootstrap                ] jvm uses the client vm, make sure to run `java` with the server vm for best performance by adding `-server` to the command line
[2015-04-13 17:50:07,929][INFO ][node                     ] [bimgo] version[1.5.1], pid[3436], build[5e38401/2015-04-09T13:41:35Z]
[2015-04-13 17:50:07,929][INFO ][node                     ] [bimgo] initializing
 ...
[2015-04-13 17:50:07,988][INFO ][plugins                  ] [bimgo] loaded [mapper-attachments, analysis-smartcn], sites []
[2015-04-13 17:50:11,014][ERROR][gateway.local.state.shards] [bimgo] failed to read local state (started shards), exiting...
org.elasticsearch.ElasticsearchIllegalStateException: Could not find a state file to recover from among [[id:18, legacy:true, file:D:\elasticsearch-1.5.1\data\elasticsearchaa\nodes\0\indices\8934\1\_state\state-18], [id:8, legacy:false,file:D:\elasticsearch-1.5.1\data\elasticsearchaa\nodes\0\indices\8934\1\_state\state-8.st]]
        at org.elasticsearch.gateway.local.state.meta.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:300)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardStateInfo(LocalGatewayShardsState.java:188)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardsStateInfo(LocalGatewayShardsState.java:173)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.<init>(LocalGatewayShardsState.java:66)

We also try upgrading to other version too.
It's working upgrading from 1.4.2 to 1.4.4.
When upgrading from 1.4.2 to 1.5.0 , same exception also throw.

Best regards

[clintongormley]

@s1monw what should be done here? delete the legacy file? or can the non-legacy file not be read for some reason, so we should bump up logging?

[s1monw]

you are running windows here right? did you upgrade and downgrade again by any chance?

[s1monw]

or did you upgrade to 1.5.0 first for some reason?

[wumingwuan]

This my upgrade process.

1. backup 1.4.2 data/config folder.
   2.copy 1.4.2 data/config to extracted 1.5.0 folder
   3.execute 1.5.0/elasticsearch.bat => exception throw
   I thought maybe 1.5.1 fix this problem. so i download 1.5.1
   and copy 1.4.2 data/config to extracted 1.5.1 folder =>same exception throw.
   Finally, I download 1.4.4 for test.
   and copy 1.4.2 data/config to extracted 1.4.4 folder =>working.

We tried the last week 1.4.2 backup data in 1.5.1=> working .
maybe this week's UAT 1.4.2 data corrupt or something.
We will try this upgrade on our production machine , if same problem happen , we will let you know.
Thank you for your help :)

[nellicus]

I am seeing this in my test system on Linux/1.5.0, though not right after upgrade, 1.5.0 has been running for a while with several stop/restart.
It's my test box where I do lot of test/hacking so not sure about sequence of events :/

[HenrikOssipoff]

Seeing the same in an upgrade from 1.4.3 to 1.5.2, running Linux.

In our process we actually upgraded to 1.5.2, but had to downgrade back to 1.4.3 (no indices were created while running 1.5.2, so theoretically this shouldn't have been an issue?).

After we settled an issue we had, we once again tried upgrading to 1.5.2, but we're now seeing this issue.

What's the best way to get around this issue once we're in this place?

[clintongormley]

In our process we actually upgraded to 1.5.2, but had to downgrade back to 1.4.3 (no indices were created while running 1.5.2, so theoretically this shouldn't have been an issue?).

Not true - any writes could have changed the index (and in fact it looks like it wrote a new state file during the failed upgrade).

@s1monw what can be done once an index is in this state?

[HenrikOssipoff]

Worth noting that no new indices were created while running 1.5.x and downgrading the node to 1.4.x and it runs without issues.

[s1monw]

you can just remove the state file in without the .st ending in this case to recover one you are on 1.5
ie. file:D:\elasticsearch-1.5.1\data\elasticsearchaa\nodes\0\indices\8934\1\_state\state-18

[HenrikOssipoff]

Worked perfectly, thank you.

[clintongormley]

thanks for letting us know @HenrikOssipoff - closing

[geekamit]

Thanks. Removing .st file worked perfectly.

[skylarmb]

Just in cast anyone comes across this in the future and the above solution does not work for you, here is what I did to fix it. DISCLAIMER: I am setting up a local dev machine, so all my data is replaceable and unimportant. I DO NOT know the effects of this on a server or other machine where you don't have a backup of this data in case it doesn't work...

I installed 2.1 and later found out the app I was trying to work on required 1.7.3 and tried to downgrade. I received this error

[ERROR][gateway.local.state.shards] [Lilandra Neramani] failed to read local state (started shards), exiting... org.elasticsearch.ElasticsearchException: unexpected field in shard state [index_uuid]

After trying many things that didnt work, including the above solution, I resorted to simply deleting the entire data folder for elasticsearch and rebuilding it. rm -rf /usr/local/var/elasticsearch. Then I just ran a reindex which rebuilt that folder rake search:reindex and everything worked just fine.

[dlamichhane]

+1

[cerealkill]

Thank you @skylarmb .

[ckc6cz]

Not an issue, just me being stupid. Sorry!

On Thu, Feb 11, 2016 at 3:59 AM, Pradeep Gowda [email protected]
wrote:

We lost all the data after removing .st file. Is this a know issue?

—
Reply to this email directly or view it on GitHub
https://github.com/elastic/elasticsearch/issues/10565#issuecomment-182772523
.

Chelsea Chandler
University of Virginia Class of 2016
B.A. Computer Science and Mathematics

[pradbk]

I did a upgrade from 1.4.4 to 1.7.1 in production. Ended up in same state (ElasticsearchIllegalStateException). removed all state file and I lost months worth of data! Do you guys have any fix for this? How to recover safely from this state?

[HenleyChiu]

I had this same problem.

I'm not sure what fixed it but I did these 2 things that I did NOT do before:
1) Sync-Flush the nodes running with old version: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-synced-flush.html
2) Shutdown the nodes properly rather than do a kill -9.
curl -XPOST 'http://localhost:9200/_cluster/nodes/_local/_shutdown'

Then when I copied the data files over, it was able to start successfully with the new version with no errors.
This is from 1.0.1 to 2.2