Prysm: Add Skylark Rule For Abigen

[prestonvanloon]

go_abi_library(
  name = "my_contract_go_library",
  srcs = [
     "my_contract.sol",
   ],
   # other options
)

# then this can be used as a go_library 
go_library(
  name = "go_default_library",
  srcs = ["library.go"],
  deps = [":my_contract_go_library"],
)

This involves writing the skylark rule which leverages the go-ethereum abigen and solidity compiler in the bazel toolchain to generate the go bindings and expose them as a go_library.

https://docs.bazel.build/versions/master/skylark/spec.html

The benefit here is that users will no longer need to regenerate the go-bindings when updating contracts and we can conform to the same version of solidity (solc) for this project.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

__This issue now has a funding of 0.4 ETH (203.45 USD @ $508.62/ETH) attached to it.__

• If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $30,213.32 more funded OSS Work available on the Gitcoin Issue Explorer

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

__Work has been started__.

These users each claimed they can complete the work by 9 months ago.
Please review their action plans below:

1) hubris has started work.

Hi, I would like to try this one if you don't mind. On which branch sould I work? master? Thank you

Learn more on the Gitcoin Issue Details page.

[terencechain]

@hubris
Fork, create a new branch, and open a PR. Check out these guidelines

[hubris]

@terenc3t Since I'm new to this project I want to be sure that I'm heading toward the right direction:

1. Add the solidity compiler as an external dep
2. Add a rule which:
   
   
   
   • Use the solc compiler to generate the abi
   
   
   • Use abigen (which is already an external) to generate the go library
   
   

Also it seems abigen can directly generate the lib from the sol file if it has a compiler.
Let me now if I missed something
Thank you.

[prestonvanloon]

@hubris We're looking to build something that is along the same lines as the go-proto-library.

Ideally, this would be the same user workflow. However, we use a solidity contract rather than a protobuf definition.

[hubris]

@prestonvanloon awesome, it will be of great help to me!

[hubris]

@prestonvanloon Hi, just to let you know what I'm doing.

So I took the code from go-proto-library and I'm making the needed change for solidity.
For now, like the go_proto_library, I have an additional abi_library rule which generates the abi files. I don't know if it is of any use to you to have this additional step or if it should be merged with the go_abi_library rule in the final version. Here is an example:

abi_library(
  name = "my_contract_library",
  srcs = [
     "my_contract.sol",
   ],
   # other options
)

go_abi_library(
  name = "my_contract_go_library",
  abi = [":my_contract_library"]
)

# then this can be used as a go_library 
go_library(
  name = "go_default_library",
  srcs = ["library.go"],
  deps = [":my_contract_go_library"],
)

As soon as my rules are working I will create a repo with only the rules so you can check them before I start the integration in your code base.

[prestonvanloon]

@hubris that is great! With that pattern we can extend it for other languages.

[hubris]

@prestonvanloon Hi,
I'm using abigen from https://github.com/prysmaticlabs/bazel-go-ethereum to compile .abi to .go.
The problem is that the generated go file don't have all the needed import. If I use abigen generated form go-ethereum Makefile AND I launch it from go-ethereum source directory the go file is correct.
Do you have any clues about that?

// Code generated - DO NOT EDIT.
// This file is a generated binding and any manual changes will be lost.

package testpkg

import (
        "math/big"
        "strings"
)

instead of

package main

import (
        "math/big"
        "strings"

        "github.com/ethereum/go-ethereum/accounts/abi"
        "github.com/ethereum/go-ethereum/accounts/abi/bind"
        "github.com/ethereum/go-ethereum/common"
        "github.com/ethereum/go-ethereum/core/types"
        "github.com/ethereum/go-ethereum/event"
)

[hubris]

Ok abigen needs go-ethereum in its GOPATH to work properly.

[prestonvanloon]

@hubris Everything must be part of the toolchain. Bazel should provide everything so you should be able to build and run these targets on a new machine with only Bazel installed (no go, no GOPATH, or anything).

If you have go-ethereum as go-repository dependency, you can use the abigen target:

@com_github_ethereum_go_ethereum//cmd/abigen:abigen

We have a bazel-friendly fork of go-ethereum here: https://github.com/prysmaticlabs/bazel-go-ethereum

[hubris]

@prestonvanloon Yes that's what I'm using.

[hubris]

According to this https://github.com/ethereum/go-ethereum/issues/15827, abigen indeed expects go-ethereum in its GOPATH.

@prestonvanloon To generate the go file from the abi file, I'm using an action which executes @com_github_ethereum_go_ethereum//cmd/abigen:abigen. I'm trying to set GOPATH in the action's env.

    go.actions.run(
        inputs = abi,
        outputs = go_srcs,
        progress_message = "Generating into %s" % go_srcs[0].dirname,
        mnemonic = "GoAbiGen",
        # This is set to "@com_github_ethereum_go_ethereum/cmd/abigen:abigen"
        executable = compiler.abigen,
        arguments = [args],
        #env = {"GOPATH": "should be go-ethereum path"}
    )

For now, my main problem is I don't know which path in the project match the go-ethereum.

Did abigen has already been used in the project successfully?

My WIP is here.
bzl bazel build //soltest:contract-go-abi will fail because of _bind_ is undefined (missing import).

Maybe I should try the rules directly in prysm in case I missed something when integrating prysmaticlabs/bazel-go-ethereum in my test project.

Thank you.

[prestonvanloon]

@hubris this will not work if it depends on go-ethereum being installed in the GOPATH.

We have used abigen successfully in this project.

# /tmp is not the GOPATH
cd /tmp
git clone https://github.com/prysmaticlabs/prysm.git
cd prysm

# go-ethereum is available as a dependency so we can build and run the abigen tool.
bazel build @com_github_ethereum_go_ethereum//cmd/abigen:abigen

# The executable is available now
bazel-bin/external/com_github_ethereum_go_ethereum/cmd/abigen/linux_amd64_stripped/abigen

The above tool is available to run in the same way as the tutorial, for example

# Build the tool as an executable
bazel build @com_github_ethereum_go_ethereum//cmd/abigen:abigen 

# Run the tool executable
bazel-bin/external/com_github_ethereum_go_ethereum/cmd/abigen/linux_amd64_stripped/abigen \
   --pkg=contracts \
   --sol=contracts/validator_registration.sol

To confirm that abigen doesn't expect anything from the gopath, I've run the above command in a cloud shell that does not have go installed or any gopath

However, this does depend on the solidity compiler in the $PATH, so we'd need to make that available via the bazel toolchain as well.

You could try the rules within prysm, but the should work for any project ideally.

[hubris]

@prestonvanloon Thank you for the answer. I tried the exact same commands as you. At the end my generated file still miss some imports:

// Code generated - DO NOT EDIT.
// This file is a generated binding and any manual changes will be lost.

package contracts

import (
        "math/big"
        "strings"
)

I checked that my GOPATH was empty. I'm working on a Fedora 26. solc is 0.4.23+commit.124ca40d.Linux.g++.

[hubris]

@prestonvanloon Sorry to disturb you again but can you confirm that the generated output of your abigen has all the go-ethereum import?
Also can you give me the exact Linux distribution of the node so I can try to reproduce your result in a VM?
Thank you.

[prestonvanloon]

Ah, thanks for the context @hubris. I apologize, I did not understand the constraint.

Here we see that the abigen uses golang's imports.Process

And it looks like there is a proposal to support goimports from bazel WORKSPACE. https://github.com/golang/go/issues/18441

However, it looks like atlassian has a tool that you might be able to use to generate the goimports: https://github.com/atlassian/bazel-tools/tree/master/goimports

[gitcoinbot]

@hubris Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] warning (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[hubris]

@gitcoinbot yes

[hubris]

@prestonvanloon Hi,
I'm a little stuck with the goimport. Here is the result of my investigation.

First, goimports expects a tree structure like this _src/github.com/ethereum/go-ethereum_. This structure is generated in the atlassian script here.

The atlassian tool allows to do a goimport for package generated by the current source tree only, not for external dependecies.

My plan was to write a similar script but instead of using $BUILD_WORKSPACE_DIRECTORY, I wanted to use the path to @com_github_ethereum_go_ethereum so I can create the symlinked src directory structure. Unfortunately, I can't figure how to get the path to the absolute path of the bazel external/ directory.

Do you know how I could get the absolute path to external/com_github_ethereum_go_ethereum ?
Thank you.

[prestonvanloon]

Hey @hubris!

Sorry that this is turning out to be so challenging! I don't know how to do that without digging deep into Skylark/Bazel rules/macros. I believe you should be able to use make variables provided by bazel. Something like $(location @com_github_ethereum_go_ethereum//:some_target) which would provide you the location of that target in a genrule.

Another idea is to provide those targets a data attribute. I haven't explored this much though.

[gitcoinbot]

@hubris Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] warning (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[hubris]

@gitcoinbot yes
@prestonvanloon Sorry I didn't have so much time to work on it, plus the unexpected difficulties. If you are not in a hurry I will continue to work on it.

[vs77bb]

Thanks for the updates @hubris. Appreciate your persistence here!

[gitcoinbot]

@hubris Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] warning (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[hubris]

@gitcoinbot yes

[hubris]

@prestonvanloon
Hi,
I tried to focus on generating a go file from a sol file.
I made a wip branch here. The rule looks like this:

sol2go(
  name = "contract-go",
  src = "Ballot.sol",
  pkg = "ballot",
  )

I use a script similar to the goimports atlassian's rules. I only managed to find the "external" directory I had to do an ugly sed on the 'sandbox' directory which is probably not a good thing. Also I didn't manage to use make variables in my own attributes, they don't seem to be expanded.

You can try the rule on my repo with:

bazel build    //soltest:contract-go

The generated Ballot.go will have the correct import at least.

Let me know if I should continue with this base and try to generate a library. Thank you.

[hubris]

@prestonvanloon Hi, I submitted a PR for you to review. I put the rules in contracts/rules. The 2 contracts validator-registration-contract and sharding-manager-contract are now generated with the new rule.

Note that in order to work solc must be installed since it doesn't seem to exist a bazel repo for this.

In the end, I drifted away from the original design. As explained in the PR, the rule sol2go will generate a go file from a given sol. Then a library can be built with the standard go_library rule. For example:

sol2go(
  name = "contract_go",
  src = "sharding_manager.sol",
  pkg = "smc",
)

This generate sharding_manager.go and the library can be created wilth:

go_library(
    name = "go_default_library",
     srcs = ["sharding_manager.go"],
     importpath = "github.com/prysmaticlabs/prysm/contracts/sharding-manager-contract",
     visibility = ["//client:__subpackages__"],
     deps = [
         "@com_github_ethereum_go_ethereum//:go_default_library",
         "@com_github_ethereum_go_ethereum//accounts/abi:go_default_library",
         "@com_github_ethereum_go_ethereum//accounts/abi/bind:go_default_library",
         "@com_github_ethereum_go_ethereum//common:go_default_library",
         "@com_github_ethereum_go_ethereum//core/types:go_default_library",
         "@com_github_ethereum_go_ethereum//event:go_default_library",
     ],
)

Let me know what you think. Thank you.

[prestonvanloon]

Hey @hubris, I left some feedback on #411

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

__Work for 0.4 ETH (114.53 USD @ $286.32/ETH) has been submitted by__:

1. @hubris
2. @hubris

@vs77bb please take a look at the submitted work:

• PR by @hubris
• PR by @hubris

---

• Learn more on the Gitcoin Issue Details page
• Want to chip in? Add your own contribution here.
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $22,380.49 more funded OSS Work available on the Gitcoin Issue Explorer

[rmshea]

Hey @hubris, what's the status on this bounty? Wanted to check in :)

[rauljordan]

Hey @hubris apologies for keeping this open for so long. Given the final contract will most likely be deployed in Vyper, we have decided to close this issue, but we would still like to award you with the bounty for your work. @ceresstation can you help close this and award @hubris?

[rmshea]

@ceresstation a ping on this one. thanks @rauljordan !

[rauljordan]

Stale PR

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

__The funding of 0.4 ETH (97.57 USD @ $243.93/ETH) attached to this issue has been approved & issued to @hubris.__

• Learn more on the Gitcoin Issue Details page
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $85,673.22 more funded OSS Work available on the Gitcoin Issue Explorer