Consul: Electing a 0.8.x leader during an upgrade can cause a panic in older servers

[slackpad]

Haven't seen this in the wild, but this code could possibly cause older Consul servers to panic if a 0.8.x server gains leadership during an upgrade:

https://github.com/hashicorp/consul/blob/v0.8.0/consul/leader.go#L156-L162

The panic would occur when an older server gets a Raft log entry for the autopilot config, which it won't understand. To avoid this until fixed, make sure to upgrade the followers before updating the current leader.

This is probably pretty rare since most folks upgrade the leader last to avoid unnecessary elections, but the consequences are high enough to make it worth avoiding. We could have the autopilot loop skip out if not all servers are at least at the right version and have it create the config, so that way it's created quickly once all the servers are upgraded, even if there's not a leader transition.

[fieldju]

@slackpad, I am not sure how rare it is for people in the AWS / CloudFormation world who use immutable AMIs and autoscaling group update policies. If I use CloudFormation to push out a new AMI that has the new version of Consul then the order in which the existing EC2 instances are replaced is non-deterministic.

[slackpad]

@fieldju yeah we have a fix in work under #2897 and are planning on a quick release (ideally next Monday) to get this patched up.

[fieldju]

I am glad, I saw this issue before upgrading. I was just thinking of ways to force CloudFormation to do what I wanted, but I can wait till #2897 is done.