Metamask-extension: libp2p mesh testing

Created on 10 Oct 2016 · 38Comments · Source: MetaMask/metamask-extension

In the name of facilitating libp2p-based Ethereum light client development, we're discussing running some libp2p mesh testing in the metamask extension.

Some Design Goals:

light: minimal overhead, gentle testing
safe: runs on a different domain than the extension (via iframe)
big: thousands of nodes

Progress Tracker:

[ ] Ethereum IPLD
- [x] RLP
- [x] eth-block-list
  - [x] basic
  - [x] resolve leaves (blocks)
- [x] dump ommers
- [ ] Simple Objects
- [x] eth-tx
- [ ] eth-tx-receipt
- [ ] Eth Trie
- [x] eth-trie rewrite
- [x] eth-state-trie rewrite
- [x] eth-storage-trie
- [x] eth-tx-trie
- [ ] eth-tx-receipt-trie
- [x] dump transactions
- [ ] dump transactionReceipts
[ ] Networking
- [x] ipfs over webrtc
- [ ] determine the current blockchain HEAD (infura or p2p)
- [ ] new tx, new block global pubsub
[ ] libp2p test mesh
- [x] add option to disable ipfs discovery
- [ ] manually setup discovery
- [x] update eth-ipld resolvers PR
- [ ] transfer eth-ipld resolvers to ipld org
[ ] Other
- [x] blockchain - separate blockDb + detailsDb
- [ ] block - update fromRpc
- [ ] web3 ipfs-provider (RPC methods via IPFS lookups)
- [ ] explore merkelization of external indexes
- [ ] CHT (canonical hash tree)
- [ ] index:txToBlock
- [ ] index:logToTx
- [ ] eth-*-trie correctly resolve keys not in trie
- [ ] experiment with identity hash fn https://github.com/multiformats/multihash/issues/13#issuecomment-270024879

T13-blocked

Source

kumavis

🎉5

Most helpful comment

here is the current custom rolled parity->ipfs server https://github.com/kumavis/ipfs-eth-bridge
it works if you can connect to it

kumavis on 9 May 2017

🎉4

All 38 comments

node.js (non-browser) intensive test suite: https://github.com/gavinmcdermott/js-libp2p-pstn

this is a great start but has been built with a few different design goals (non-browser, short and intensive, fully connected mesh)

metamask mesh tests would need to be:

browser based, running on webrtc
setup for a topology with many nodes (1-3k)
tests more lightweight, intermittent
focus on long-running tests and long-term health of the mesh

kumavis on 10 Oct 2016

cross-team interface/deployment will most likely happen in this way:
the extension runs a hidden iframe pointed at a domain (untrusted by the extension) controlled by the ipfs/libp2p team with some mesh test code that they can update at their will

kumavis on 10 Oct 2016

ping @diasdavid @gavinmcdermott

kumavis on 10 Oct 2016

marking as blocked until I have some concrete action items ( happy to work on some libp2p-side stuff if needed )

kumavis on 10 Oct 2016

This is blocked by this big crypto PR https://github.com/libp2p/js-libp2p-crypto/pull/12

kumavis on 20 Oct 2016

next blockers:

kumavis on 7 Nov 2016

next blockers:

[x] https://github.com/ipfs/js-libp2p-ipfs-browser/issues/122
[x] multiformats/js-multihash#20
[x] https://github.com/ipld/js-ipld-eth-block/issues/3

kumavis on 16 Nov 2016

next blockers:

[x] [js-ipld-resolver - add support for eth-block](https://github.com/ipld/js-ipld-resolver/pull/69)
- [x] ipfs/js-cid#7
- [x] multiformats/js-multicodec#7
- [x] https://github.com/ipfs/js-cid/pull/8
- [x] https://github.com/multiformats/js-multicodec/pull/8
- [x] https://github.com/ipfs/js-cid/pull/10
- [x] https://github.com/ipfs/js-cid/pull/11
[x] try on js-ipfs#feat/dag-api

kumavis on 5 Dec 2016

heres a new sketch of the explicit ethereum obj types ( 8 types in total )

[0x90] eth-block 
  ommers: eth-block-list
  txs: eth-tx-trie
  receipts: eth-tx-receipt-trie
  state: eth-state-trie

[0x91] eth-tx (local data only)
[0x92] eth-tx-receipt (local data only)
[0x93] eth-account-snapshot (local data only)

[0x94] eth-block-list (rlp array)
  [leaves]: eth-block

[0x95] eth-tx-trie (merkle trie)
  [leaves]: eth-tx

[0x96] eth-tx-receipt-trie (merkle trie)
  [leaves]: links to eth-tx-receipt

[0x97] eth-state-trie (secure merkle trie)
  [leaves]: links to eth-account-snapshot

[0x98] eth-storage-trie (secure merkle trie)
  [leaves]: links to raw binary

kumavis on 15 Dec 2016

here is an attempt at a concise-yet-hacky set ( 6 types )

[0x90] eth-block  (rlp)
  ommers: eth-rlp:eth-block
  txs: eth-trie:eth-tx
  receipts: eth-trie:eth-tx-receipt
  state: eth-trie:eth-account-snapshot

[0x91] eth-tx  (rlp) (local data only)
[0x92] eth-tx-receipt  (rlp) (local data only)
[0x93] eth-account-snapshot  (rlp)
  storage: eth-trie

[0x94] eth-trie[:type] (merkle trie)
  [leaves]: buffer or <type>

[0x95] eth-rlp[:type] (rlp)
  [leaves]: buffer or <type>

The hack works like this:
[ethBlock cid, 'state/<address>/balance']
would first resolve to
[<ethBlock.stateRoot cid>, '/<hash of address>:eth-account/balance']

since ethereum objects determine their leaf types from context, it doesnt fit neatly into a couple resolvers -- but if we cleverly use remainingPath we can specify the leaf type

kumavis on 16 Dec 2016

here is an ultra-concise ( with only 2 types! ), by maximizing path fanciness

[0x90] eth-rlp[:type]
  [leaves]: buffer or <type>

[0x91] eth-trie[:type]
  [leaves]: buffer or <type>

how to lookup:

eth-block: `<cid for block as eth-rlp>/:block/`
  ommers(index): `<cid for ommersRoot as eth-rlp>/<index>/:block/`
  txs(index): `<cid for transactionsTrie as eth-trie>/<rlp.encode(index)>/:tx/`
  receipts(index): `<cid for receiptTrie as eth-trie>/<rlp.encode(index)>/:txr/`
  state(address): `<cid for stateRoot as eth-trie>/<hash(address)>/:account/`

eth-tx: `<cid for tx as eth-rlp>/:tx/`
  (local data only)

eth-tx-receipt: `<cid for txr as eth-rlp>/:txr/`
  (local data only)

eth-account-snapshot: `<cid for accountSnapshot as eth-rlp>/:account/`
  storage(key): `<cid for storageRoot as eth-trie>/<hash(key)>/`

kumavis on 16 Dec 2016

theres one weird thing but i realize it would be true for the 6-type and 2-type version

resolving <cid for eth-block>/state/<address>
would not be the same as
resolving <cid for eth-block>/state/
then resolving <address>
bc <cid for eth-block>/state/<address> will hash the address after the first step, resolving to <cid for stateRoot as eth-trie>/<hash of address>

kumavis on 16 Dec 2016

@wanderer what do you think of the 2-type solution?

kumavis on 16 Dec 2016

zomg unified ethereum ipld proposal ( 1 type 😎 )

[0x90] eth-star[:type] ( basically rlp but we look for `trie@` prefix and `/:<type>` postfixes
  {leaves}: buffer
  {leaves}/:type: <type>

how to lookup:

eth-block: `<cid for block as eth-star>/:block/`
  ommers(index): `<cid for ommersRoot as eth-star>/<index>/:block/`
  txs(index): `<cid for transactionsTrie as eth-star>/trie@<rlp.encode(index)>/:tx/`
  receipts(index): `<cid for receiptTrie as eth-star>/trie@<rlp.encode(index)>/:txr/`
  state(address): `<cid for stateRoot as eth-star>/<hash(address)>/:account/`

eth-tx: `<cid for tx as eth-star>/:tx/`
  (local data only)

eth-tx-receipt: `<cid for txr as eth-star>/:txr/`
  (local data only)

eth-account-snapshot: `<cid for accountSnapshot as eth-star>/:account/`
  storage(key): `<cid for storageRoot as eth-star>/trie@<hash(key)>/`

note: special chars can be changed to maintain compliance with ipld-query spec. paths are ether pre-defined by the type or specified as hex, so avoiding collision is not a hard problem.

kumavis on 16 Dec 2016

that said, we have room for 16384 codes before we hit the 3 byte range, which is currently reserved for apps
so maybe we dont need to be so concise
so maybe taking up 8 types and having the prettiest paths is reasonable 😸

kumavis on 16 Dec 2016

tho im sure in the future ill be interested in adding this fancy type casting for some other reason, like converting eth contract storage values into cids or other formats

kumavis on 16 Dec 2016

@wanderer btw jbenet and daviddias said dont hack the paths, better to have more formats

kumavis on 16 Dec 2016

👍1

Major TODOS:
( moved to top )

kumavis on 23 Dec 2016

who likes todo lists? you do!
( moved to top )

kumavis on 2 Jan 2017

😄1

TIL "identity hash fn" https://github.com/multiformats/multihash/issues/13#issuecomment-270024879

kumavis on 2 Jan 2017

consider using rlpx over libp2p transport

kumavis on 13 Jan 2017

proposal: blocks via ipfs-http-api on parity https://github.com/paritytech/parity/issues/4172
@wanderer @cdetrio this would be a nice way to skip syncing all together:

run parity client
wrap parity ipfs-http-api as block service
run public ipfs gateway on top of parity-backed block service

kumavis on 13 Jan 2017

identifying RPC method data/index requirements

  eth_protocolVersion
  eth_syncing
  eth_coinbase
  eth_mining
  eth_hashrate
  eth_gasPrice
  eth_accounts
  eth_sign
  eth_sendTransaction
  eth_sendRawTransaction
  eth_newFilter
  eth_newBlockFilter
  eth_newPendingTransactionFilter
  eth_uninstallFilter
  eth_getFilterChanges
  eth_getFilterLogs

ipfs
  eth_getBlockByHash
  eth_getUncleByBlockHashAndIndex

ipfs:selectors (selectors/transforms)
  eth_getBlockTransactionCountByHash
  eth_getUncleCountByBlockHash

index:txToBlock
  eth_getTransactionReceipt
  eth_getTransactionByHash
  eth_getTransactionByBlockHashAndIndex

blockTracker (head tracking)
  eth_blockNumber

index:CHT
  eth_call
  eth_estimateGas
  eth_getBalance
  eth_getStorageAt
  eth_getTransactionCount
  eth_getCode
  eth_getBlockByNumber
  eth_getTransactionByBlockNumberAndIndex (+index:txToBlock)
  eth_getBlockTransactionCountByNumber (+ipfs:selectors)
  eth_getUncleCountByBlockNumber (+ipfs:selectors)
  eth_getUncleByBlockNumberAndIndex
  eth_getLogs (+index:logToTx)


HEX String - an integer block number
String "earliest" for the earliest/genesis block
String "latest" - for the latest mined block
String "pending" - for the pending state/transactions

transaction
├─ blockHash { txToBlock }
├─ blockNumber { txToBlock }
├─ creates { txToBlock, compute, ... }
├─ from { ecrecover }
├─ gas {}
├─ gasPrice {}
├─ hash {}
├─ input {}
├─ nonce {}
├─ publicKey { ecrecover }
├─ r {}
├─ raw {}
├─ s {}
├─ to {}
├─ transactionIndex { txToBlock, blockToTxs }
├─ v {}
└─ value {}

kumavis on 16 Feb 2017

for those playing along at home, i made a thing: https://kumavis.github.io/metamask-mesh

kumavis on 16 Feb 2017

maybe relevant: https://github.com/bitcoin/bips/blob/master/bip-0122.mediawiki

kumavis on 28 Feb 2017

migrated js-ipld-eth-* modules to ipld org

kumavis on 22 Mar 2017

(a few weeks back) friends at Parity added an http server that serves parity data from CIDs. it got merged into master : )

kumavis on 22 Mar 2017

tracking parity integration here https://github.com/ipfs/js-ipfs/issues/763

kumavis on 6 May 2017

ipfs routing improvements https://github.com/ipfs/js-ipfs/issues/833

kumavis on 9 May 2017

here is the current custom rolled parity->ipfs server https://github.com/kumavis/ipfs-eth-bridge
it works if you can connect to it

kumavis on 9 May 2017

🎉4

currently eth-ipfs bridge only serves blocks, transactions (no tries), state and storage tries, no tx receipts or no tx receipt tries. These are missing b/c parity does not index them by hash (tx receipt) or doesn't store them at all (tx trie, tx receipt tree).

the bridge is available over http as https://ipfs.lab.metamask.io

kumavis on 3 Jul 2017

what if: generic ipfs merkle tree heat map could show us what is happening on ethereum

kumavis on 4 Jul 2017

👍1

we will need to: create a special index for { tx tries, tx rxs, tx rx tries }

kumavis on 4 Jul 2017

what if: (in the long run): a box that users can just "_download and run_". Will be like this https://github.com/kumavis/ipfs-eth-bridge

With the idea "Contribute to the light clients mesh, making it stronger and resilient by downloading this application".