Lighthouse: Disk usage accelerates under non-finality

[nickoneill]

Description

Since medalla started having finality issues on October 12th, disk usage for lighthouse has sharply increased.

Version

0.3.0, rustc 1.46.0 (04488afe3 2020-08-24)

Present Behaviour

See the grafana graph for available space on-disk below. Usage is steady until the 12th when it starts to accelerate. This can potentially make non-finality issues worse as nodes that do not anticipate rapidly increasing disk space requirements can be knocked offline, further decreasing participation rates.

Expected Behaviour

Disk usage should not significantly change during periods of non-finality.

[michaelsproul]

There's talk amongst the client teams of adopting a more aggressive pruning strategy to prune even without finality. The logic is that we can prune heads with very few votes, because they are unlikely to be built upon. If someone does try to build upon a pruned head, we can use the existing parent-block-not-found code to resync it

[paulhauner]

Disk usage should not significantly change during periods of non-finality.

I agree we could probably do better, but I suspect disk usage will always increase to some degree under finality.

[paulhauner]

I'm going to close this since "reducing DB usage" is a well-known goal and this issue doesn't identify a specific issue that we need to fix. Thank you for raising it and providing detail :)