Eth2.0-specs: Make SSZ code accessible

[JustinDrake]

Currently the SSZ code lives in test_libs/pyspec/eth2spec/utils/ssz/. The long directory path is unfriendly. It is also semantically wrong to put ssz implementation code in test_libs since SSZ is core to the spec, and not just used for testing.

[protolambda]

The whole pyspec is in test_libs, not just ssz. Its only purpose was providing tests, being used for testing changes spec design, and being fuzzed. We could move out the pyspec directory to the root of the repository, and put ssz next to the utils package. If you like to put the pyspec more forward.

More fundamentally: I am hesitant doing that, the pyspec is not designed as external tooling. It has hard performance limits (slow BLS, slow-ish SSZ, readability over optimal implementation), and I don't think we should be relying on it for more and more, as eventually we will hit mainnet with higher performance requirements, e.g. to go quickly through dumped chain data. Hence also the effort put into the go-spec. Ideally I like to have the pyspec in a place where it bridges the spec formality to something people can test and validate their code with, but not run as a primary program.

[dankrad]

Also another comment on this: I believe the SSZ core spec is still in simple-serialize.md. At least the current code in test_libs/pyspec/eth2spec/utils/ssz/ is not suitable as a replacement for the spec document in my opinion.

We could move out the pyspec directory to the root of the repository, and put ssz next to the utils package. If you like to put the pyspec more forward.

I would be happy with that, I find the current place in test_libs a bit awkward to use.

[mratsim]

For a python implementation, it's probably better to just refer people to trinity https://github.com/ethereum/trinity/blob/master/eth2/_utils/ssz.py

[ralexstokes]

where i think you want to point to this: https://github.com/ethereum/py-ssz/

[JustinDrake]

The whole pyspec is in test_libs, not just ssz.

I'm confused. The majority of the spec, e.g. the phase 0 state transition function, is in specs/, not in test_libs/.

[protolambda]

It builds to test_libs, which adds all the things needed for testing, test generation, etc.

The future organization is looking like:

• Update ssz repo draft, rm signing-root from there too, review it, and then decide on standardization stages for the different features described there, and enshrine it. Then remove SSZ inner workings from the eth2 spec here, similar to BLS being described in terms of the IETF work.
• Choose between two approaches (or somehow make them interchangeable) that are long-term more sustainable for mainnet chain test performance, in place of the minimal but python-magic backed pyspec-ssz:
  
  
  
  • py-ssz, from trinity: most production ready, but opinionated for trinity and not built for merkle proofs
  
  
  • something like my pymerkles experiment, that completes the idea of representing state as a typed merkle tree, and does away with the caching/multi-proof complexity by such representation. But it would cost more time.
  
  

For now I don't see much reason to stick to this pyspec-ssz code as a third party (i.e. outside of spec), when there are better much faster alternatives available. Such as py-ssz for python, zssz for uncached cases in Go (outperforms every client cold start), or fast and cached, but opinionated choices such as go-ssz (Prysm) and lighthouse-ssz. And plenty more to choose from (see ssz repo draft for up to date ssz implementation table).