Beats: Fix broken external dependencies

[Pungyeon]

Describe the enhancement:
It is currently not possible to import github.com/elastic/beats/cmd.GenRootWithSettings using go modules. This will break (several things):

1. This will retrieve a really old version of the docker-client, which is incompatible
2. The same is true for k8s.io client
3. The github.com/dop251/goja dependencies which are included (vendored) in the github repository, is completely broken. The function RegisterSimpleMaptype() does not exist anymore, and even worse, there are no releases and no branches for this library. The only way to fix this would be to fork the project and revert the changes back to before the deletion of said function.

I therefore propose to fix these dependencies and include a go.mod, to make working with the libbeat library much easier.

I am willing to help with this project and will happily contribute to fixing this.

Describe a specific use case for the enhancement or feature:
Make writing custom beats much easier (when using go modules).

[kvch]

Beats has just adopted go modules as a dependency manager. Does this fix your problems?

[Pungyeon]

Hi @kvch I though this would fix it, but it seems that there are still some external dependencies which aren't being imported correctly, when using the libbeat library from a new project.

Perhaps I am doing something wrong though.... This is the error message that I receive, when I try to build my project:

# github.com/elastic/beats/libbeat/common/docker
../../go/pkg/mod/github.com/elastic/[email protected]+incompatible/libbeat/common/docker/client.go:37:12: undefined: client.Opt
../../go/pkg/mod/github.com/elastic/[email protected]+incompatible/libbeat/common/docker/client.go:46:23: undefined: client.WithVersion
../../go/pkg/mod/github.com/elastic/[email protected]+incompatible/libbeat/common/docker/client.go:49:23: undefined: client.WithAPIVersionNegotiation
../../go/pkg/mod/github.com/elastic/[email protected]+incompatible/libbeat/common/docker/client.go:52:9: undefined: client.NewClientWithOpts
# k8s.io/client-go/rest
../../go/pkg/mod/k8s.io/[email protected]+incompatible/rest/request.go:598:31: not enough arguments in call to watch.NewStreamWatcher
    have (*versioned.Decoder)
    want (watch.Decoder, watch.Reporter)
# github.com/elastic/beats/libbeat/processors/script/javascript
../../go/pkg/mod/github.com/elastic/[email protected]+incompatible/libbeat/processors/script/javascript/session.go:260:14: s.Runtime().RegisterSimpleMapType undefined (type *goja.Runtime has no field or method RegisterSimpleMapType)
# github.com/corticph/benchmark-model/beater
beater/beater.go:84:5: cannot use "github.com/Pungyeon/beats/libbeat/common".MapStr literal (type "github.com/Pungyeon/beats/libbeat/common".MapStr) as type "github.com/elastic/beats/libbeat/common".MapStr in field value

[kvch]

Could you share your go.mod?

We are using a forks of a few dependencies. You need to add those to the replace section of your Beats because the go version we are using is a bit limited in that regard. I think adding those directives will fix your issue.

Also, I am working on moving Beat generators to go modules as well. It is in progress, but I expect it to be done soon. You might be interested in it: https://github.com/elastic/beats/pull/16288

[Pungyeon]

@kvch , so having tried quite a few things. I started over... in the end I was still receiving the following error:

../../go/pkg/mod/k8s.io/[email protected]+incompatible/rest/request.go:598:31: not enough arguments in call to watch.NewStreamWatcher
        have (*versioned.Decoder)
        want (watch.Decoder, watch.Reporter)

But updating the version to the following:

go get k8s.io/[email protected]

Seemed to work a charm. So, now everything is indeed working as intended.

Thank you so much for helping with addressing this 🙏 You are awesome !! 👏

[richard-mauri]

Is it true that beats has adopted gomodules?
Was this publicly announced and if so where and are there samples to adopting it?
EllieMae has several golang projects that need to build beats from source and so far have ben struggling with elastic beat go modules.
I want to upgrade our important custom beats to test if it helps several production issues.

[richard-mauri]

I got it to work using instructions https://github.com/elastic/go-elasticsearch
The key for me was to has this in go.mod: require github.com/elastic/beats/v7 v7.8.0 and to change import statements to refer to /v7 like import "github.com/elastic/beats/v7/libbeat/cmd/instance"

I think v7.8.0 is the latest released beats...

Thanks eats team for getting go modules.