Venus: Datastore is larger than expected

[ZenGround0]

Description

We have gotten reports from @whyrusleeping and others that the datastore is much bigger than expected. The expectation (I believe) is that chain data dominates the total amount of (non-sector) data stored by the filecoin node. However at present other data with size an order of magnitude greater than chain data is stored in the datastore.

Here are the relative sizes of the different sub-stores on filecoin devnets (thanks @travisperson for grabbing these)

user:

$ du -h /mnt/storage/filecoins/0
76K    /mnt/storage/filecoins/0/deals
13G    /mnt/storage/filecoins/0/badger
639M    /mnt/storage/filecoins/0/chain
24K    /mnt/storage/filecoins/0/snapshots
20K    /mnt/storage/filecoins/0/wallet
8.0K    /mnt/storage/filecoins/0/staging
8.0K    /mnt/storage/filecoins/0/keystore
13G    /mnt/storage/filecoins/0

test

$ du -h /mnt/storage/filecoins/0
28M    /mnt/storage/filecoins/0/chain
16K    /mnt/storage/filecoins/0/snapshots
76K    /mnt/storage/filecoins/0/deals
8.0K    /mnt/storage/filecoins/0/staging
1.2G    /mnt/storage/filecoins/0/badger
20K    /mnt/storage/filecoins/0/wallet
8.0K    /mnt/storage/filecoins/0/keystore
1.3G    /mnt/storage/filecoins/0

28M    /mnt/storage/filecoins/1/chain
257M    /mnt/storage/filecoins/1/sealed
16K    /mnt/storage/filecoins/1/snapshots
64K    /mnt/storage/filecoins/1/deals
257M    /mnt/storage/filecoins/1/staging
454M    /mnt/storage/filecoins/1/badger
20K    /mnt/storage/filecoins/1/wallet
8.0K    /mnt/storage/filecoins/1/keystore
995M    /mnt/storage/filecoins/1

28M    /mnt/storage/filecoins/2/chain
513M    /mnt/storage/filecoins/2/sealed
16K    /mnt/storage/filecoins/2/snapshots
76K    /mnt/storage/filecoins/2/deals
261M    /mnt/storage/filecoins/2/staging
460M    /mnt/storage/filecoins/2/badger
20K    /mnt/storage/filecoins/2/wallet
8.0K    /mnt/storage/filecoins/2/keystore
1.3G    /mnt/storage/filecoins/2

28M    /mnt/storage/filecoins/3/chain
257M    /mnt/storage/filecoins/3/sealed
16K    /mnt/storage/filecoins/3/snapshots
64K    /mnt/storage/filecoins/3/deals
257M    /mnt/storage/filecoins/3/staging
455M    /mnt/storage/filecoins/3/badger
20K    /mnt/storage/filecoins/3/wallet
8.0K    /mnt/storage/filecoins/3/keystore
995M    /mnt/storage/filecoins/3

28M    /mnt/storage/filecoins/4/chain
257M    /mnt/storage/filecoins/4/sealed
16K    /mnt/storage/filecoins/4/snapshots
64K    /mnt/storage/filecoins/4/deals
257M    /mnt/storage/filecoins/4/staging
453M    /mnt/storage/filecoins/4/badger
20K    /mnt/storage/filecoins/4/wallet
8.0K    /mnt/storage/filecoins/4/keystore
994M    /mnt/storage/filecoins/4

The amount of data stored in badger is between 46x (test node 0) and 16x (other test nodes) that of the data stored in chain

We don't know what this data is and why it is dominating our repo size. It appears that the badger directory wraps the blockstore/cborstore of the filecoin node (need to confirm this).

We need to determine where this data comes from and sketch out a plan for using our storage resources more efficiently.

Acceptance criteria

We have a document describing in vivid detail why we see this large resource use and understand some of our options for reducing it. The explanation should also give us a better understanding of datastore usage patterns generally (who is storing what). This document might live in the design docs "other" category.

Risks + pitfalls

Where to begin

One hypothesis is that the bloat comes from storing lots of actor state over and over. This really only makes sense if there are lots of actors with state that changes a lot because the content addressed nature of the blockstore should be doing heavy duty deduplication automatically.

[ZenGround0]

We now know that the biggest source of disk usage is flushing state storage to the cborstore HAMT.
See this writeup for an explanation and steps to reproduce:

[whyrusleeping]

@ZenGround0 your analysis is missing a few things.

• Blocks have all messages embedded directly in them. So we are storing every message once, plus an additional time for each block they are contained in (tipsets with multiple blocks will have duplicate messages)
• sector commitments are stored as a flat array in the miner actors state storage. This means that every time a miner adds a sector, it rewrites the entire list as a new object, leading to sector storing taking up (N^2)/2 space on disk per miner.
• The HAMT is probably too wide, its current width is 256, which leads to a lower deduplication factor than we might like to have. We should investigate dropping this down to 128, 64, or even 32.
• All objects are way bigger than they need to be, since we're storing things as plain cbor with all the field names embedded. FCS will help here

cc @frrist who I know was looking into this as well.

[whyrusleeping]

Also, really think we should avoid having writeups like this in google docs. Would be much better to have these as issues.

[anorth]

Thanks @ZenGround0. Please summarise the findings and suggested actions here (tho IMO the full analysis and repro steps are ok in a doc). We need a few follow-up issues too outlining the various opportunities for improvement, which we can then prioritise independently.

[ZenGround0]

[Meta note: My strong opinion is that the analysis and repro steps should only live in a doc. A one sentence summary and link is WAI. The question at hand is "what kind of doc should they live in?" I have a similar (but weaker) opinion about design docs living as markdown on github but let's talk about that [here](https://github.com/filecoin-project/pm/issues/147).]

Suggested Actions Related to State Storage

Garbage Collecting State Storage

Our HAMT/CHAMP should be documented

HAMT Usage optimizations

Suggested Actions Related to the go-filecoin Disk Use in General

Don't copy chain state unnecessarily

Garbage Collecting Chain Data Storage

Chain Datastore Increases on Daemon Restart

I didn't look into this deeply, just observed the problem. First step is to make sure it's not just badger trying to be clever and allocating in advance because there are a lot of writes during load

Stop JSON Encoding Chain

this happens here and here

Merkelize Messages for Deduplication

see spec issue for latest work

Correctly Encode CBOR Data

Miner Actor State Improvements

Note this is still spec work not tracked in go-filecoin issues