Eos: DAWN-496 ⁃ Initialize Genesis from ERC-20 Snapshot

[bytemaster]

1. Add snapshot script to /eos/programs/snapshot
2. Provide instructions on using this script in /eos/progams/snapshot
3. Create a program to convert snapshot.csv into snapshot.json file that can be used with eosd to start a properly configured blockchain
4. Check in an snapshot.json file that is ready to go for the test network

┆Attachments: snapshot (period 118).json | snapshot 118_csv.zip

[dskvr]

1. Snapshot is converted from JSON to CSV in the script, could it just export both? Or are two steps necessary
2. We will need to determine the final ethereum block for the testnet snapshot.

[nathanhourt]

The EOS ERC-20 freezes all balances at the end of the sale, does it not? So knowing the exact ethereum block wouldn't matter, you just need some block after the sale ends?

[dskvr]

@nathanhourt Yes and no. The _Token_ is frozen at end of crowdsale, but registration transactions to the _Crowdsale_ contract can still be confirmed. This would cause potential mismatch between different snapshots, because the entire snapshot is based off the registry. _No longer true_

_Hypothetically_

Alice wants to verify the snapshot and runs generator. Meanwhile, Roger sends a Registration TX. Bob wants to verify the snapshot and runs generator. Alice and Bob's versions are different.

While consistency is important for any snapshot, this request was directed towards the testnet screenshot. Ending block is important here because tokens are not frozen, if multiple people generated a snapshot, they would all have different versions. _Important Note: This is true regardless of end block, but end block adds more consistency_

[rangershi]

grate

[dskvr]

• [x] Add snapshot script to /eos/programs/snapshot
• [x] Provide instructions on using this script in /eos/progams/snapshot
• [x] Create a program to convert snapshot.csv into snapshot.json* file that can be used with eosd to start a properly configured blockchain
• [x] Check in an snapshot.json* file that is ready to go for the test network

_* I think you meant genesis.json, yes?_

[dskvr]

Snapshot specifications have changed since this issue was initially opened. Unregistered addresses will now be included in snapshot through a fallback strategy which is currently being tested. This change of strategy has a completely different approach from initial requirements, as it requires full supply monitoring. The first pivot (supply monitoring and registration initiative) set the stage for the second pivot, so little time was lost. However, still requires refactoring.

High Priority

• [x] Token Balances (contract state and calculated) Removed balanceOf from Contract State, it can only be used reliably in Mainnet Snapshot.
• [x] Unclaimed Balances (calculated)
• [x] Reclaim transactions sent to Crowdsale/Token contract by mistake (calculated)
• [x] Key Validation
• [x] Registration Validation
• [x] Database Implementation (mysql, sequelize)
• [x] Period/Block Maps and Period Change Detection
• [x] Network-Wide EOS Token Supply Monitoring (was complete, depends on wallet state sync refactor)
• [x] Crowdsale Utilities
• [x] Contract State Synchronization (transfers, buys, claims, registrations)
• [x] Wallet State DB Synchronization/Queueing [was complete, now needs to be refactored]
• [x] State Polling/Queuing [was complete, now needs to be refactored]
• [x] Test various cases for fallback registration written by @jcalfee
• [x] Implement Fallback into Snapshot
• [x] Snapshot Generator based on DB State (trivial after all above tasks are completed)
• [x] Snapshot Validation
• [x] Better error-handling and edge case detection
• [x] Configuration
• [x] Wide-scope public key discovery

Lower Priority

• [ ] Dockerization
• [x] Implement Snapshot Logging
• [ ] Daily Snapshots (end of period)
• [ ] API interface for frontend
• [ ] Frontend so users can easily check their status with relatively short delay (30 minutes)

Deprecated
Modes (testnet)
Modes (mainnet, polling)

[dskvr]

Utility logic has been complete for a while and the majority of business logic is complete but still working through some bugs, numbers aren't lining up since refactoring supply monitor -> snapshot. Problems I'm having are familiar to ones I've already worked through in previous iteration so not worried. Aiming to have high priority tasks listed complete by Tuesday.

[dskvr]

Almost all the numbers are aligned now. Once everything is aligned like it was in previous iteration, I'll be able to complete the remaining tasks.

[dskvr]

Need to refactor instantiation to accommodate various modes.

[jcalfee]

Refactoring is good.. I'm happy to review your work when it is ready. Thanks for the updates..

[dskvr]

Reminder: There is almost nothing in common to pre-release code in 'genesis'. Completely different strategy due to spec pivots and web3 1.0 beta stability.

[dskvr]

Made use of the space I had from this for a week and took a turn to simplify as much as possible. Put polling functionality in a separate local branch and will address at a later time. Did even more testing on EOS registration fallback, and still looks promising. Starting on snapshot generation and verifying the output, looking to push the most recent iteration within 1-2 days.

[dskvr]

Missed an update yesterday. Taking a break from aligning numbers and working on collecting as many public keys as possible and optimizing registration fallback method. Have an issue with not aggregating all buys for the last period of any given snapshot, have a feeling it's related to block number vs transaction timestamp. I will close the gap on the 0.014% discrepancy when snapshotting up to period 109 on Monday. Stripped out some error detection and logging, it was giving me bad information. Will rewrite on Monday as well.

[dskvr]

Got snapshot within acceptable margin of error on Sunday (worked through the weekend). Was going to write up docs and push yesterday, but ended up feeling some burnout so took a break. Tested fallback and worked with @jcalfee to correct some issues. Spent some time yesterday thinking up a way to more efficient way to discover public keys, but came up empty.

[dskvr]

Moved the process for ethereum public key scanning (web3) into it's own step so testing supply validation can take 20 minutes vs 4-5 hours. This needs some work to ensure script doesn't break. Made the snapshot script configurable so there's a map for docker config and so it can be configured without docker. Added a tools directory for some web tools, started on tool to check snapshot balance associated an ETH address/EOS public key. Started testing the tests (programmatic validation of database numbers vs contract state, for example). Organized the directory a bit and updated references.

[dskvr]

Been moving logic around for configurability, public key discovery is a wider scope now and wrapping up the various tests.

[dskvr]

Sped up public key discovery by ~20x, working on refactoring logging and making the script more portable. Spent a little time on some frontend interfaces. Starting on Readme.

[dskvr]

Margin of error is now consistently -0.00000001%. Modified some existing tests, added some new ones, removed others. Finishing readme. Started modifying existing genesis block generator. Considering adding a prompt so no file configuration is necessary.

[dskvr]

This was pushed to EOSIO/genesis repo yesterday. Unfinished tasks above are now issues in genesis repo.

[dskvr]

Accidentally closed.

Sidenote: Committing a genesis.json file requires knowing up to which period the snapshot should be generated for. I would assume a couple days before testnet is launched?

@bytemaster @heifner @thomasbcox

[blockone-syncclient]

➤ Corey Lederer commented:

Thomas Added:

EOS community is asking what Easter Eggs the genesis block will contain. Satoshi included some in the Bitcoin genesis block, apparently (e.g. some headlines from newspapers). Should we? And if so, what?

[blockone-syncclient]

➤ Corey Lederer commented:

[~Sean.Mitchell] Is this still open?

[gleehokie]

OBE