Confluent-kafka-go: Bazel Integration

[zanes2016]

Has anyone tried to integrate the confluent-kafka-go package with their existing Bazel project using go rules?

[zanes2016]

If anyone else is interested, I can create a PR that builds this with Bazel (using genrules).

Here's a snippet of my code for anyone who may be interested:

BUILD  WORKSPACE  kafka-go.BUILD  librdkafka.BUILD  main.go  zlib.BUILD

# tail -vn +1 *
==> BUILD <==
package(default_visibility = ["//visibility:public"])

load("@io_bazel_rules_go//go:def.bzl", "go_binary")

go_binary(
    name = "main",
    srcs = ['main.go'],
    deps = [
      "@com_github_confluent_kafka_go//:go_library",
    ],
#    static = 'on',
#    clinkopts = ['-pthread', '-Wl,--no-as-needed', '-ldl'],
#    cgo = True,
)
==> WORKSPACE <==

http_archive(
    name = "io_bazel_rules_go",
    url = "https://github.com/bazelbuild/rules_go/releases/download/0.10.1/rules_go-0.10.1.tar.gz",
    sha256 = "4b14d8dd31c6dbaf3ff871adcd03f28c3274e42abc855cb8fb4d01233c0154dc",
)

load("@io_bazel_rules_go//go:def.bzl", "go_repository", "go_rules_dependencies", "go_register_toolchains")
go_register_toolchains()
go_rules_dependencies()

new_git_repository(
    name = "com_github_confluent_kafka_go",
    remote = "https://github.com/confluentinc/confluent-kafka-go.git",
    tag = "v0.11.4",
    build_file = "kafka-go.BUILD",
)

new_git_repository(
  name = 'zlib',
  remote = "https://github.com/madler/zlib.git",
  tag = "v1.2.11",
  build_file = "zlib.BUILD"
)

new_git_repository(
    name = "com_github_edenhill_librdkafka",
    remote = "https://github.com/edenhill/librdkafka.git",
    tag = "v0.11.4",
    build_file = "librdkafka.BUILD",
)

==> kafka-go.BUILD <==
package(default_visibility = ["//visibility:public"])
load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")

go_library(
    name = 'go_library',
    srcs = glob(['kafka/**/*.go'], exclude=['**/go_rdkafka_generr.go']),
    importpath = 'github.com/confluentinc/confluent-kafka-go/kafka',
    cgo = True,
    cdeps = [
      ":localglue",
      "@com_github_edenhill_librdkafka//:kafka",
      "@zlib//:z",
    ],
    clinkopts = [
    '-Wl,--no-as-needed',
      '-ldl',
    ],
)

cc_library(
    name = 'localglue',
    hdrs = ['kafka/glue_rdkafka.h'],
    strip_include_prefix = 'kafka',
)

==> librdkafka.BUILD <==

genrule(
    name = "librdkafka-a",
    cmd = "cd external/com_github_edenhill_librdkafka && ./configure && make && mv src/librdkafka.a ../../$@",
    srcs = glob(["**"]),
    outs = ["src/librdkafka.a"],
)

cc_import(
    name = "librdkafka",
    static_library = ":librdkafka-a",
)

cc_library(
    name = "kafka",
    hdrs = ["src/rdkafka.h"],
    strip_include_prefix = 'src',
    include_prefix = 'librdkafka',
    deps = [
      ":librdkafka",
    ],
    visibility = ["//visibility:public"],
)
==> main.go <==
package main

import (
    _ "github.com/confluentinc/confluent-kafka-go/kafka"
)

func main() { }

==> zlib.BUILD <==
cc_library(
    name = "z",
    srcs = [
        "adler32.c",
        "compress.c",
        "crc32.c",
        "deflate.c",
        "infback.c",
        "inffast.c",
        "inflate.c",
        "inftrees.c",
        "trees.c",
        "uncompr.c",
        "zutil.c",
    ],
    hdrs = [
        "crc32.h",
        "deflate.h",
        "gzguts.h",
        "inffast.h",
        "inffixed.h",
        "inflate.h",
        "inftrees.h",
        "trees.h",
        "zconf.h",
        "zlib.h",
        "zutil.h",
    ],
    includes = [
        ".",
    ],
    linkstatic = 1,
    visibility = [
        "//visibility:public",
    ],
)

_note that i haven't been able to get static linking to work yet - any suggestions are welcome_

[wkj89]

did it work?

[mancini0]

I created this question on stackoverflow: https://stackoverflow.com/questions/55698979/bazel-rules-go-linking-go-binary-against-a-static-c-library-a-file-produce

I'm able to produce the required librdkafka libraries inside bazel through rules_foreign_cc's cmake_external rule, but have not had any luck linking my go_binary against it (I've only tried statically linking however), perhaps due to a lack of know-how. Other c++ targets in my bazel project link fine against my librdkafka library, however.

[s-garg]

Anyone got it working for the v1.0.0 release?

[prestonvanloon]

@s-garg yes it works with bazel v1.0.0.

# WORKSPACE
all_content = """filegroup(name = "all", srcs = glob(["**"]), visibility = ["//visibility:public"])"""

http_archive(
    name = "rules_foreign_cc",
    strip_prefix = "rules_foreign_cc-master",
    url = "https://github.com/bazelbuild/rules_foreign_cc/archive/master.zip",
)

load("@rules_foreign_cc//:workspace_definitions.bzl", "rules_foreign_cc_dependencies")

rules_foreign_cc_dependencies()

http_archive(
    name = "librdkafka",
    build_file_content = all_content,
    strip_prefix = "librdkafka-1.2.1",
    urls = ["https://github.com/edenhill/librdkafka/archive/v1.2.1.tar.gz"],
)

# third_party/kafka/BUILD.bazel

load("@rules_foreign_cc//tools/build_defs:cmake.bzl", "cmake_external")

cmake_external(
    name = "librdkafka",
    cache_entries = {
        "RDKAFKA_BUILD_STATIC": "ON",
        "WITH_ZSTD": "OFF",
        "WITH_SSL": "OFF",
        "WITH_SASL": "OFF",
        "ENABLE_LZ4_EXT": "OFF",
        "WITH_LIBDL": "OFF",
    },
    lib_source = "@librdkafka//:all",
    static_libraries = [
        "librdkafka++.a",
        "librdkafka.a",
    ],
    visibility = ["//visibility:public"],
)

[priyendra]

@prestonvanloon Can you confirm that you got the combination of rules_go + librdkafka + confluent-kafka-go working?

Can you provide more details about your solution:

• The WORKSPACE file entry for the confluent-kafka-go repository, and
• The go_library/go_binary entry for your code that invokes confluent-kafka-go library

Thanks!

[prestonvanloon]

hey! yeah it is working, but I haven't deployed to prod.

Here's the PR in our project: https://github.com/prysmaticlabs/prysm/pull/3840

I think the missing piece you need is a rules_go patch for confluent-kafka-go.

WORKSPACE

go_repository(
    name = "in_gopkg_confluentinc_confluent_kafka_go_v1",
    importpath = "gopkg.in/confluentinc/confluent-kafka-go.v1",
    patch_args = ["-p1"],
    patches = ["//third_party:in_gopkg_confluentinc_confluent_kafka_go_v1.patch"],
    sum = "h1:roy97m/3wj9/o8OuU3sZ5wildk30ep38k2x8nhNbKrI=",
    version = "v1.1.0",
)

third_party/in_gopkg_confluentinc_confluent_kafka_go_v1.patch

--- a/kafka/BUILD.bazel
+++ b/kafka/BUILD.bazel
@@ -23,6 +23,7 @@
         "producer.go",
         "testhelpers.go",
     ],
+    cdeps = ["@prysm//third_party/kafka:librdkafka"],
     cgo = True,
     importpath = "gopkg.in/confluentinc/confluent-kafka-go.v1/kafka",
     visibility = ["//visibility:public"],

I hope that helps.

[mikekamornikov]

@prestonvanloon As I see you support MacOS as bazel host system. Do you have cross-compilation issues (darwin -> linux)? If there are any how do you handle that?

[mikekamornikov]

BTW, to build an image with the code above I do:

# WORKSPACE
http_archive(
    name = "io_bazel_rules_docker",
    sha256 = "e513c0ac6534810eb7a14bf025a0f159726753f97f74ab7863c650d26e01d677",
    strip_prefix = "rules_docker-0.9.0",
    urls = ["https://github.com/bazelbuild/rules_docker/archive/v0.9.0.tar.gz"],
)

# Golang images
# This is using gcr.io/distroless/base
load(
    "@io_bazel_rules_docker//go:image.bzl",
    _go_image_repos = "repositories",
)

_go_image_repos()

# CC images
# This is using gcr.io/distroless/base
load(
    "@io_bazel_rules_docker//cc:image.bzl",
    _cc_image_repos = "repositories",
)

_cc_image_repos()

# BUILD.bazel
go_image(
    name = "docker",
    embed = [":go_default_library"],
    # CGO_ENABLED=0 for pure builds
    pure = "off",
    # to avoid "error while loading shared libraries: libstdc++.so.6"
    base = "@cc_image_base//image",
    visibility = ["//visibility:public"],
)

This was I can get a usable image from linux host at least ...

[prestonvanloon]

@mikekamornikov yeah, we have darwin -> linux cross compile issues. CI builds our docker images so it's not much of an issue, but i wish I had a resolution for the problem.

I don't recall the specific error, something about needing to define a cc toolchain. I've gone down that path a few times but never was able to resolve it.