Protobuf: Protobuf with go modules

[andreaaizza]

Hi,

as a aggravation I tried to read here and there about the topic, but could not get the solution.

I would like to use a this go package https://github.com/andreaaizza/sniffer, which is generating via its makefile 3 different protobuf files, inside a outer package, let's call the latter 'Outer'.

Outer is also using modules and its go.mod looks like:

module andreaaizza/outer

go 1.15

require (
        github.com/andreaaizza/sniffer v0.1.0
...
        google.golang.org/protobuf v1.25.0
)

When I try to go build it, I get the following error:

# github.com/andreaaizza/sniffer/logger
~/go/pkg/mod/github.com/andreaaizza/[email protected]/logger/logger.go:36:2: undefined: LoggerBuffer
~/go/pkg/mod/github.com/andreaaizza/[email protected]/logger/logger.go:37:19: undefined: DataUnit
...

obviously because the .proto files of github.com/andreaaizza/sniffer are not compiled.

Now, I tried multiple combinations including adding the following to Makefile in order to build .pb.go files inside vendor/... directory:

clean_vendor: 
        -rm -rf vendor

vendor: clean_vendor
        $(eval SNIFFER_PKG_DIR=$(shell go list -f "{{ .Dir }}" -m github.com/andreaaizza/sniffer))
        echo $(SNIFFER_PKG_DIR)
        go mod vendor
        make -f $(SNIFFER_PKG_DIR)/Makefile -C vendor/github.com/andreaaizza/sniffer proto

but build still fails with:

$ go build -mod vendor
vendor/github.com/andreaaizza/sniffer/logger/logger.pb.go:10:2: cannot find package "." in:
        ./vendor/github.com/golang/protobuf/proto

The only way I could make this working is by building .proto files inside readonly pkg directory:

pushd `go list -f "{{ .Dir }}" -m github.com/andreaaizza/sniffer`/..
chmod -R u+w [email protected] && make -C [email protected] proto
ake: Entering directory '~/go/pkg/mod/github.com/andreaaizza/[email protected]'
protoc --go_out=./ --go_opt=module=github.com/andreaaizza/sniffer logger/logger.proto
protoc --go_out=./ --go_opt=module=github.com/andreaaizza/sniffer dissector/dissector.proto
protoc --go_out=./ --go_opt=module=github.com/andreaaizza/sniffer sniffer.proto
make: Leaving directory '~/go/pkg/mod/github.com/andreaaizza/[email protected]'
popd
go build 

I belive this is a evident blunder, but I cannot find the right solution.

Can anybody help in getting me to the right path to handle protobuf with go modules?

Thanks in advance,
Andrea

[dsnet]

I suspect that this has nothing to do with modules, but rather a missing or malformed go_package option in the .proto files. What do you have it specified to?

[andreaaizza]

I suspect that this has nothing to do with modules, but rather a missing or malformed go_package option in the .proto files. What do you have it specified to?

Hi,

the inner proto files (e.g. https://github.com/andreaaizza/sniffer/blob/master/sniffer.proto) have

syntax = "proto3";
package sniffer;

option go_package = "github.com/andreaaizza/sniffer";

import "dissector/dissector.proto";
...

or https://github.com/andreaaizza/sniffer/blob/master/dissector/dissector.proto:

syntax = "proto3";
package dissector;

option go_package = "github.com/andreaaizza/sniffer/dissector";
...

Outer package has not yet any .proto file.

[dsnet]

What about logger.proto? That's the one reporting the build failure with: cannot find package "."

[dsnet]

I just ran the following:

$ git clone https://github.com/andreaaizza/sniffer
$ cd sniffer/
$ protoc --go_out=./ --go_opt=module=github.com/andreaaizza/sniffer logger/logger.proto
$ protoc --go_out=./ --go_opt=module=github.com/andreaaizza/sniffer dissector/dissector.proto
$ protoc --go_out=./ --go_opt=module=github.com/andreaaizza/sniffer sniffer.proto
$ go build ./...
$ go mod vendor
$ go build -mod vendor ./...
$ go version
go version go1.15.2 linux/amd64

Everything worked fine. I don't seem to be able to replicate any problems.

[andreaaizza]

The issue I would like to solve is not with this one: https://github.com/andreaaizza/sniffer. The issue is pops when I utilize it within another package (which uses modules).

[dsnet]

Without a reproduction, I don't know how else to help.

[andreaaizza]

$ cat main.go
package main

import (
        "github.com/andreaaizza/sniffer"
)

func main() {
        s := sniffer.Sniffer{}
        s.Close()
}
$ cat go.mod
module test/test

go 1.15

require github.com/andreaaizza/sniffer v0.0.0-20201015143716-39fab437e220
$ go build main.go
# github.com/andreaaizza/sniffer/logger
~/go/pkg/mod/github.com/andreaaizza/[email protected]/logger/logger.go:36:2: undefined: LoggerBuffer
~/go/pkg/mod/github.com/andreaaizza/[email protected]/logger/logger.go:37:19: undefined: DataUnit
~/go/pkg/mod/github.com/andreaaizza/[email protected]/logger/logger.go:57:27: undefined: DataUnit
~/go/pkg/mod/github.com/andreaaizza/[email protected]/logger/logger.go:114:35: undefined: DataUnit
~/go/pkg/mod/github.com/andreaaizza/[email protected]/logger/logger.go:140:11: undefined: LoggerBuffer
~/go/pkg/mod/github.com/andreaaizza/[email protected]/logger/logger.go:144:11: undefined: LoggerBuffer
~/go/pkg/mod/github.com/andreaaizza/[email protected]/logger/logger.go:151:11: undefined: DataUnit

[dsnet]

The build failure here is entirely expected because the .pb.go files have not be generated. The go build command does not automatically invoke protoc.

[andreaaizza]

Sure, like a mantioned in the initial question. What is the right way to invoke protoc, when using a package including .proto files?

[neild]

There is documentation on using protoc here:

https://developers.google.com/protocol-buffers/docs/gotutorial#compiling-your-protocol-buffers
https://developers.google.com/protocol-buffers/docs/reference/go-generated#invocation

There is no way to use the go tool to build from .proto files directly. You need to compile them into .pb.go files and make those accessible to the go tool, usually by checking them in to source control.

[andreaaizza]

Hi Niel,

thanks.

So the "right way" would be to update https://github.com/andreaaizza/sniffer: make proto and commit also .pb.go files?

Ciao,
Andrea

[dsnet]

So the "right way" would be to update https://github.com/andreaaizza/sniffer: make proto and commit also .pb.go files?

Correct.

[dsnet]

Closing as working as intended. For questions regarding usage, it is better to ask on Slack or someone other means where you can more readily get immediate help. Consider joining the #protobuf channel or the Gophers slack workspace: https://gophers.slack.com/messages/general/