Lighthouse: Project directory restructure

[paulhauner]

Description

Our current directory structure is not clear when it comes to the eth2/utils directory. For example:

• serde_hex is not ethereum specific.
• Why is ssz in eth2/utils, not in eth2? (and vice-versa for things in eth2/, why aren't they in eth2/utils?)

I propose the following change.

Proposal

Here are the directories that should live in the root:

• beacon_node: binary + any crates that are specific to the binary.
• validator_client: binary + any crates that are specific to the binary.
• account_manager: binary + any crates that are specific to the binary.
• utils: any non-ethereum-specific utility crates (e.g., logging, serde utils, etc).
• eth: ethereum-specific libraries (may be related to eth1 or eth2).
• testing: binaries or libraries that are exclusively for testing (e.g., ef tests, test harness).
• docs: documentation

Here's an example tree:

- beacon_node/
  |- src/
  |- client/
  |- network/
  |- Cargo.toml
- validator_client/
  |- src/
  |- Cargo.toml
- account_manager/
  |- src/
  |- Cargo.toml
- utils/
  |- logging/
  |- serde_hex/
- eth/
  |- beacon_state_processing/
  |- beacon_types
  |- lmd_ghost
  |- shard_types
  |- ssz
  |- tree_hash
- testing/
  |- ef_tests
  |- test_harness
- docs/
  |- contributing.md

Additionally, in order to have a clear name space between beacon chain things and shard chain things, I propose:

• Prefix the existing state_processing and types crates with beacon_. When we add shard chain specific things, prefix with shard_. This is going to become important as the Quilt team contributes and we need to keep that code separate from the Phase 0 security review.

Summary

The changes are mostly around flattening the current eth2/utils/ structure and making a distinction between general utils and ethereum-specific utils.

Keen to hear people's thoughts.

[michaelsproul]

I quite like this restructure!

I was thinking we should maybe use subdirectories for beacon and shard code, but actually I think the prefix is better because it trivially propagates to the crate name (you don't end up with two crates called state_processing).

One crate that's missing in the current plan is the operation pool, it could live under either eth/beacon_op_pool (similar to the current situation) or beacon_node/op_pool (possibly cleaner?)

Also, eth2/validator_change? I'd never noticed this crate, it seems to be dead code. We could delete it as part of the restructure, if not before.

[paulhauner]

As mentioned in #1089, it would be nice to have a separation of "consensus critical" code from other code. As such, here's an updated proposal:

Proposal

Here are the directories that should live in the root:

• lighthouse: binary that encapsulates beacon_node, validator_client and account_manager.
• beacon_node: library + any crates that are specific to the binary.
• validator_client: library + any crates that are specific to the binary.
• account_manager: library + any crates that are specific to the binary.
• common: any crate that is used by more than one other top-level crate, but is not consensus critical.
• consensus: "consensus critical" eth2 libraries.
• testing: binaries or libraries that are exclusively for testing (e.g., ef tests, test harness).
• lcli: the testing CLI tool, separate from lighthouse since it isn't user-facing.
• book: documentation

Of course, "consensus critical" is a bit blurry so I don't think we can make a firm rule about it, but I think we could define it roughly as "pure state transition logic". I think we'll have to deal with this on a case-by-case.

Here's an example tree:

- beacon_node/
  |- src/
  |- client/
  |- network/
  |- Cargo.toml
- validator_client/
  |- src/
  |- Cargo.toml
- account_manager/
  |- src/
  |- Cargo.toml
- utils/
  |- logging/
  |- serde_hex/
- consensus/
  |- state_processing/
  |- types
  |- proto_array_fork_choice
  |- eth2_ssz
  |- eth2_hashing
- testing/
  |- ef_tests
  |- test_harness
- lcli/
  |- src/
- docs/
  |- contributing.md

[paulhauner]

I'm also in favour of adding a root-level crypto/ dir which can contain things like:

• bls
• merkle_hashing
• eth2_keystore
• ..etc