Lighthouse: Missed Attestation despite lighthouse claiming otherwise

[vogelito]

Description

According to my lighthouse node, we successfully published our first attestation, but both beaconcha.in and beaconscan claim otherwise.

The lighthouse logs say:

Aug 10 15:16:12 medalla-testnet lighthouse[4836]: Aug 10 15:16:12.084 INFO Successfully published attestations     type: unaggregated, slot: 43880, committee_index: 5, head_block: 0x89ca

However, beaconcha.in and beaconscan say did-not-attest and missed, respectively.

Version

ether@medalla-testnet:~/lighthouse$ git branch
* master
ether@medalla-testnet:~/lighthouse$ git log --oneline | head -1
6206d8e7 Ensure .git is copied into docker (#1462)

Present Behaviour

Until the time of writing, new attestations are working.

Expected Behaviour

I would've expected the first attestation to have worked. New attestations have been working even though we didn't even touch the node.

Steps to resolve

I have no idea. I would love to understand why I got slashed (even if it was a very small amount).

[michaelsproul]

Hey @vogelito, thanks for the report!

What has happened here is that your attestation was signed and published by your node, but then the other nodes on the network failed to include it in a block (which is what the block explorer shows). This could happen for a number of reasons -- it could be that your node didn't have enough peers at the time, and so the attestation didn't make it to one of the (randomly selected) attestation aggregators, or it could have arrived too late, or been dropped by faulty aggregators. We are working hard to improve our attestation inclusion metrics, e.g. in https://github.com/sigp/lighthouse/pull/1415. At the moment it's normal to miss an attestation from time to time, but if you feel like you're missing lots, please let us know!

If you're satisfied with that resolution, I'll close this issue

[michaelsproul]

One more thing! You should update Lighthouse to v0.2.2 and keep an eye on the Discord announcements channel for important info -- things are moving quickly with improvements landing all the time!

[AgeManning]

Yep. Missing attestations is a complicated issue which is not isolated to just Lighthouse.

In a lighthouse-only network, we see 100% attestation inclusions. There are some more complicated network-dynamics happening involving multiple clients, various interoperability issues and the fact that we have 20,000+ validators all at once trying to be included in every block.

All the eth2 client developers are working together to try and get the attestation inclusion rate to 100% on the public testnet, and will likely be continuously improving until mainnet.

As @michaelsproul said, the best thing to do is to keep up to date with the improvements we will be continually adding to lighthouse.

[vogelito]

Understood, thank you very much.

We've missed out on a few attestations in the last 24 hours, but we'll continue to monitor the situation as the updates come.

I've subscribed to the rss feed at https://github.com/sigp/lighthouse/releases.atom and will keep lighthouse updated as per the releases there.

Question: is there a best practice to upgrade lighthouse? I assume it's something like:

1. build the new binary
2. shutdown old validator
3. shutdown old beacon
4. start new beacon
5. start new validator

Thanks!

[vogelito]

Now running Lighthouse/v0.2.2-48fd312/x86_64-linux

[michaelsproul]

Yep! That's a pretty good upgrade procedure!

Closing now, feel free to open another issue or ping us if you run into trouble ☺️

[vogelito]

Completely circumstantial, but I think my attestation effectiveness is actually down after the upgrade. I was at 96% before and now at 91%. I didn't miss any attestations during the upgrade, but many since.