Venus: cluster has stopped mining

[phritz]

It is stuck on block 276. cc @nickle

1. Did we get an alert for this? If not we must follow up on why not.
2. Why?

[frrist]

Yes we got an alert(s) for this -- It started at 6UTC to 10pm PST last night meaning I didn't see it until this morning.

Seeing errors like these on all nodes in the cluster:

2018-12-11 08:28:02.520659 ERROR node node.go:482: Problem mining a block: get weight: weight numerator cannot be repr by uint64

2018-12-11 08:26:29.942149 DEBUG consensus.expected expected.go:280: TODO: return error; proof is invalid.

Based on these errors it appears that we are unable to generate a new block, and we are also unable to validate mining is occuring correctly.

@shannonwells it appears that you were the last person to edit the line with: TODO: return error; proof is invalid., could you shed some light on the nature of this message?

[whyrusleeping]

@ZenGround0 This appears to be related to the weighting issues...

[frrist]

Current Cluster Tipset:

[{"/":"zDPWYqFCt2ga4CZrT1M2ndP286HStv6VyQt7kpAZRVfYHzCcs1eD"},{"/":"zDPWYqFCvihZ3FEzKXFzFQwgGTkekwS2b88TJTmPG9ZRAFiNRfSa"}]

Current Chain head block:

zDPWYqFCvihZ3FEzKXFzFQwgGTkekwS2b88TJTmPG9ZRAFiNRfSa:

{"height":"lAI=","messageReceipts":[{"exitCode":0,"return":null},{"exitCode":0,"return":["HQ=="]}],"messages":[{"message":{"from":"fcqyrerzkmgpfmhjnuwaj88elewwjda6jzqwqghvh","method":"","nonce":"AA==","params":null,"to":"fcq5729q8yv4h6vdvfqu63w47d8zk9e04au9dtz9e","value":"1000"},"signature":null},{"message":{"from":"fcqlhz2zpy9yf65zux8r095vh0rm7ewcyqdjh9kd4","method":"addAsk","nonce":"Jw==","params":"gkmAgOiTkpjy2CJBBQ==","to":"fcqmj4zc94qc4qzrue8psqzazjddx74j54kce8je6","value":null},"signature":"FV1+fJVRur+aCmIZvIe53sHuWXRBvYDIdPInQyFhK39uJncwmwi0JWlrNQOsw25seEzLF0f+xBSV0eRd8Dq/cwE="}],"miner":"fcq5729q8yv4h6vdvfqu63w47d8zk9e04au9dtz9e","nonce":"AA==","parentWeightDenominator":"hciHpJ+zdA==","parentWeightNumerator":"zt31u4XQguA+","parents":[{"/":"zDPWYqFCz5rZhR21xxzrt5nEVbGxrcysY2W7iBbaSqd8VxqeFjjN"},{"/":"zDPWYqFD6bnrpF1ZBtdbP5UP6LvxiRyHpCrArNZD8ZFy8zUmJPxg"}],"proof":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"stateRoot":{"/":"zdpuAogwAVRTANsJ52ZGS87r558w7LGZAkdvav8cQM3wFX88e"},"ticket":"LURGdDfAGyrc9bW1cDpd2sJgRRRJ6CNupnTUDxhF9pM="}

zDPWYqFCt2ga4CZrT1M2ndP286HStv6VyQt7kpAZRVfYHzCcs1eD:

{"height":"lAI=","messageReceipts":[{"exitCode":0,"return":null}],"messages":[{"message":{"from":"fcqyrerzkmgpfmhjnuwaj88elewwjda6jzqwqghvh","method":"","nonce":"AA==","params":null,"to":"fcqet4zmyekew6zyvez34874w9ervweafps5yfk59","value":"1000"},"signature":null}],"miner":"fcqet4zmyekew6zyvez34874w9ervweafps5yfk59","nonce":"AA==","parentWeightDenominator":"hciHpJ+zdA==","parentWeightNumerator":"zt31u4XQguA+","parents":[{"/":"zDPWYqFCz5rZhR21xxzrt5nEVbGxrcysY2W7iBbaSqd8VxqeFjjN"},{"/":"zDPWYqFD6bnrpF1ZBtdbP5UP6LvxiRyHpCrArNZD8ZFy8zUmJPxg"}],"proof":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"stateRoot":{"/":"zdpuAxY7VMZ72fd75TqAPFFuSCSqqQkn5cxd7273kSnM3PWR3"},"ticket":"FKiPXYYQZNoJ6lIyhk8dkxYGPzzNoCWdUnlKoQJe6NA="}

[phritz]

Isn't it clear what the issue is? Overflow of the power numerator causes getWeight to fail.

[frrist]

@ZenGround0 and I were chatting in #fil-dev I think he may have some ideas here.

[ZenGround0]

@phritz is correct

This is more or less tracked in #655. Will work out story details and plan of action to keep cluster from coming down in the short/medium term.

[frrist]

@ZenGround0 not sure what the next steps are, if this is a duplicate of #655 my vote is we close this issue, and prioritize #655 to "In Progress", that sound reasonable?