Go: plugin: panic: plugin was built with a different version of package

[juhwany]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN="/Users/juhwany/go_workspace/bin"
GOCACHE="/Users/juhwany/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/juhwany/go_workspace"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/juhwany/gos/go_server/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/mk/2tyywdsn0jq6l0vl8094tss80000gp/T/go-build380269681=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

1. Clone the blow repo in each directory. (not under GOPATH)
   server (main program) : https://github.com/juhwany/go_server
   plugin : https://github.com/juhwany/go_plugin

2. On same machine, build server & plugin in each directory
   Build server : go build -o main
   Build plugin : go build -buildmode=plugin -o plugin.so

3. Copy plugin.so to server directory and run
   ./main

4. See error message when main program try to open plugin.

panic: plugin.Open("plugin"): plugin was built with a different version of package github.com/juhwany/go_server/v2/apis

What did you expect to see?

Main program should load plugin.so successfully.
I could not understand why it fails even though the referenced package version is same between main program and plugin.
Am I using plugin in wrong way?

What did you see instead?

plugin was built with a different version of package error message

[randall77]

Related (dup?) #26759 .

[juhwany]

@randall77

No. I think building environment is slightly different from #26759.
I didn't build main program and plugin under GOPATH and the above sample project uses Go module for specifying dependency package. Go version is also different.

[BryceDFisher]

I do think they are all related though, including https://github.com/golang/go/issues/27062. I think it all boils down to changing (absolute) paths at build time between the plugin build and the app build.

[bcmills]

CC @ianlancetaylor @cherrymui

[tamasfe]

I'm not sure if anyone's already mentioned it, but putting the common(apis) package to a separate module outside the server one will work. It's definitely cumbersome, but can be used as a workaround until this issue is fixed.

[oszika]

The command "go list -m -json all" shows version of go_server on go_plugin, but it's not defined on go_server itself. I have a similar problem using multiples modules on the same repository using "replace" directive. I don't know how to have the same version of the module inside and outside the repository.

[oszika]

In #28983, @bcmills suggests to build plugins from within the main module.

// server/go.mod
module github.com/juhwany/go_server/v2
require github.com/juhwany/plugin v0.0.0
replace github.com/juhwany/plugin => ../go_plugin

$ go build -buildmode=plugin github.com/juhwany/plugin
$ go run main.go 
DoSomething called!

[jbrukh]

Just throwing in my two cents here. Both the @tamasf97 and @oszika are workarounds.

However, @tamasf97's workaround only works if you can factor the shared code out without creating a circular dependency (not always possible).

@oszika's workaround using the "replace" directive is pretty inconvenient for clients of the shared code, as you need that local repo.

[itsmontoya]

We are unable to run our plugin-based service with gomod due to this issue.

[dhalman]

We are experiencing this issue after implementing go.mod in both the server and plugin

[bcmills]

@itsmontoya, @dhalman: what specific commands did you run to build the plugin?

(Did you follow the approach in https://github.com/golang/go/issues/27751#issuecomment-443452815? Did you build with -trimpath? Both of those may be necessary.)

[itsmontoya]

@bcmills we will try -trimpath and the other approach and get back to you ASAP! 👍

[dhalman]

added -trimpath and the error moved from our microservice dependency lib to now internal/cpu:

plugin.Open("plugins/thePlugin"): plugin was built with a different version of package internal/cpu

[novabyte]

@dhaimeen We've used the Go plugin API for quite a while now (over a year) and the error "different version of package internal/cpu" occurs when you've not used the -trimpath flag on both the shared object you build (plugin) and the main binary. Both bundle a copy of the Go runtime (although there's an open issue on the tracker about not packaging a copy of the Go runtime with plugin code). Hope this helps.

[dhalman]

The main bin solved it for us in a manual build process. Thank you!!

We are now running into an issue where it appears -trimpath isnt working as expected when running from a command in the .Exec(...) env

The .so files produced by the exec command differ from the files produced by go build in the cli

[dhalman]

Ok, I think we have determined the difference:

Successful:

cd <path-to-plugin>
go build -trimpath -buildmode=plugin -o <filename> 

Unsuccessful:

go build -trimpath -buildmode=plugin -o <filename> <path-to-plugin>

Are there any ways to determine why the latter fails to produce the same output as the former?

Thanks!

[cherrymui]

@dhalman I think you can try using go build's -x flag, e.g.

go build -x -trimpath -buildmode=plugin -o <filename>

to print out how the go command invokes the compiler and the linker, and how it differs.
Maybe you want to do a go clean -cache first.

[bcmills]

@dhalman, is the plugin in the same module as the main binary? If not, the module metadata could be what's throwing it off.

Do you have any cgo dependencies? That could also do it (#36072).

[mytototo]

I have a collection of Go plugins I need to package and distribute. When building and running on my machine everything works as expected. However plugins don't work on other machines.

Here is the command I execute:

$ GO111MODULE=on go build -trimpath -buildmode=plugin -o ./my-plugin.so ./my-plugin-directory

I tried running the parent application calling the plugins both with and without -trimpath. Nothing works.

How can I fix this behavior?

[paralin]

@bcmills I got caught up on this one with modules - the problem can occur when you import a package from the same module in the "loader," and you import the same package but from a different module in the "plugin."

[dhalman]

We have developed an open source solution for syncing go.mod repositories across large projects with internal dependencies. We find it especially useful for managing large collections of plugins along with their private imports.

Happy to share it if requested, but would not want to advertise in this forum

[mytototo]

@paralin Do you have a working and a non-working example so I can try to reproduce your solution? I am not sure to fully understand.