Go: cmd/go: error when enable cgo and static link with build flag '-mod=vendor'

[sanguohot]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11 linux/amd64.

Does this issue reproduce with the latest release?

of course it is.

What operating system and processor architecture are you using (go env)?

NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

What did you do?

this is my simple test code

package main

import (
    "fmt"
    "github.com/ethereum/go-ethereum/common/hexutil"
    "github.com/ethereum/go-ethereum/crypto"
    "github.com/ethereum/go-ethereum/crypto/secp256k1"
    "log"
)

func main()  {
    bytes, err := hexutil.Decode("0x71b9732fa43b52d8c9085c053064b55dccc8785da7f12b3812787ee3def367ba0a8796dafb76fd5a146c226036092b7f5dd59feecb92776f7bb2cf2a48594ae201")
    if err != nil {
        log.Fatal(err)
    }
    pub, err := secp256k1.RecoverPubkey(crypto.Keccak256Hash([]byte("hello")).Bytes(), bytes)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(hexutil.Encode(pub))
}

then I ran the command go build --ldflags "-linkmode external -extldflags -static" -a -o main -v -mod=vendor secp256k1.test.go

What did you expect to see?

well, not error and output with the executable file main

What did you see instead?

error and it still go on but with no main file output at the end.

> runtime/debug
> github.com/ethereum/go-ethereum/common
> github.com/ethereum/go-ethereum/common/math
> runtime/cgo
> github.com/ethereum/go-ethereum/crypto/secp256k1
> # github.com/ethereum/go-ethereum/crypto/secp256k1
> ../vendor/github.com/ethereum/go-ethereum/crypto/secp256k1/curve.go:42:44: fatal error: libsecp256k1/include/secp256k1.h: No such file or directory
> compilation terminated.
> github.com/ethereum/go-ethereum/rlp
> log

by the way, if I do not add the flag -mod=vendor, it seems ok.

[sanguohot]

no body take a look at this issue?

[dmitshur]

You're saying that go build --ldflags "-linkmode external -extldflags -static" -a -o main -v -mod=vendor secp256k1.test.go produces an error above, but there's no error if you take out -mod=vendor.

What happens if you take out all the other flags, and run go build -mod=vendor secp256k1.test.go?

As is, it's hard to tell if the issue is within Go, or if there's a problem with the go-ethereum source code (it's also not clear which commit was being used). To make progress, we'll need to narrow it down into a smaller, reproducible test case (ideally, one that doesn't depend on an external project).

If you're unsure how to do that, I'd suggest first reporting this issue in the go-ethereum project's issue tracker. If they can confirm it's not an issue in that codebase, we can investigate further.

/cc @bcmills

[sanguohot]

@dmitshur
My test result is the same with the latest go-ethereum commit.
It is very easy to replay with the following commands:

> root@sanguohot:/opt# rm -rf test
> root@sanguohot:/opt# mkdir test
> root@sanguohot:/opt# cp ./medichain/test/secp256k1.test.go test/
> root@sanguohot:/opt# cd test/
> root@sanguohot:/opt/test# go mod init github.com/sanguohot/test
> go: creating new go.mod: module github.com/sanguohot/test
> root@sanguohot:/opt/test# go mod tidy && go mod vendor
> go: finding github.com/ethereum/go-ethereum/common/hexutil latest
> go: finding github.com/ethereum/go-ethereum/crypto latest
> go: finding github.com/ethereum/go-ethereum/crypto/secp256k1 latest
> go: finding github.com/ethereum/go-ethereum/common latest
> go: finding github.com/btcsuite/btcd/btcec latest
> go: finding github.com/btcsuite/btcd latest
> go: finding gopkg.in/check.v1 latest
> go: finding github.com/davecgh/go-spew/spew latest
> go: finding github.com/kr/text v0.1.0
> go: finding github.com/kr/pty v1.1.1
> go: downloading github.com/kr/text v0.1.0
> root@sanguohot:/opt/test# go build --ldflags "-linkmode external -extldflags -static" -a -o main -mod=vendor
> # github.com/ethereum/go-ethereum/crypto/secp256k1
> vendor/github.com/ethereum/go-ethereum/crypto/secp256k1/curve.go:42:44: fatal error: libsecp256k1/include/secp256k1.h: No such file or directory
> compilation terminated.
> root@sanguohot:/opt/test# ll
> total 24
> drwxr-xr-x  3 root root 4096 Sep 18 09:45 ./
> drwxr-xr-x 11 root root 4096 Sep 18 09:44 ../
> -rw-r--r--  1 root root  314 Sep 18 09:45 go.mod
> -rw-r--r--  1 root root 1180 Sep 18 09:45 go.sum
> -rw-r--r--  1 root root  567 Sep 18 09:44 secp256k1.test.go
> drwxr-xr-x  3 root root 4096 Sep 18 09:45 vendor/
> root@sanguohot:/opt/test# go build --ldflags "-linkmode external -extldflags -static" -a -o main
> root@sanguohot:/opt/test# ls
> go.mod  go.sum  main  secp256k1.test.go  vendor

But your suggestion is good, it may be the go-ethereum's problem.
I will report an issue to their project.

[dmitshur]

But your suggestion is good, it may be the go-ethereum's problem.
I will report an issue to their project.

Sounds good. Feel free to link it to this issue so we can keep track of progress.

Also, did you see this question:

What happens if you take out all the other flags, and run go build -mod=vendor secp256k1.test.go?

[sanguohot]

@dmitshur There is no difference with go build -mod=vendor secp256k1.test.go.
I test it in the issue ethereum/go-ethereum:17700

[myitcv]

This is almost certainly a dup of https://github.com/golang/go/issues/26366. The thread gives some good background on what's happening here, especially https://github.com/golang/go/issues/26366#issuecomment-405683150.

If you want to "vendor" the entire module then https://github.com/golang/go/issues/26366#issuecomment-424075008 gives links to how to do that.

[sanguohot]

This is almost certainly a dup of #26366. The thread gives some good background on what's happening here, especially #26366 (comment).

If you want to "vendor" the entire module then #26366 (comment) gives links to how to do that.

@myitcv Thank you very much, it works to me.
I hope there will be go mod modvendor as you say.