Homebrew-core: Needs to rebuild gRPC package after abseil version bump

[rizsotto]

Bug report

After abseil version bump got merged, my project fails to link against gRPC.

I presume grpc needs to re-build.

• [x] ran brew update and can still reproduce the problem?
• [ ] ran brew doctor, fixed all issues and can still reproduce the problem?
• [ ] ran brew gist-logs <formula> (where <formula> is the name of the formula that failed) and included the output link?
• [ ] if brew gist-logs didn't work: ran brew config and brew doctor and included their output with your issue?

What you were trying to do (and why)

My project depends on gRPC (not directly on abseil). And it fails to link the executables.

Actually the project I got, had a commit which was only update the documentation. Before the build step success, after failure. The difference is what brew installed.

success:

==> Pouring abseil-20200225.2.catalina.bottle.tar.gz
==> Pouring gflags-2.2.2.catalina.bottle.1.tar.gz
==> Pouring protobuf-3.13.0.catalina.bottle.tar.gz
==> Pouring re2-20200801_1.catalina.bottle.tar.gz
==> Pouring grpc-1.32.0_1.catalina.bottle.1.tar.gz

fail:

==> Pouring abseil-20200923.catalina.bottle.tar.gz
==> Pouring gflags-2.2.2.catalina.bottle.1.tar.gz
==> Pouring protobuf-3.13.0.catalina.bottle.tar.gz
==> Pouring re2-20200801_1.catalina.bottle.tar.gz
==> Pouring grpc-1.32.0_1.catalina.bottle.1.tar.gz

What happened (include command output)

ld: library not found for -labsl_dynamic_annotations

What you expected to happen

Link successful as it did before.

Step-by-step reproduction instructions (by running brew install commands)

brew install grpc

[SMillerDev]

I'm not sure I understand what failed in brew here. Both installs seem to be succeeding. Please provide the information in the issue regardless of any CI or external factors.

[rizsotto]

Hi @SMillerDev , thanks for looking into this.

I'm not sure I'm reporting this issue in the right place. It's not a problem with the command brew itself. It's the grpc package needs to be recompiled after the abseil package has a new version.

I don't own a macOS machine, so I can't give you more details than the CI outputs. What I presume is happening here:

• abseil-20200225.2.catalina.bottle.tar.gz is the old version of abseil, which contains a library libabsl_dynamic_annotations.dylib file.
• grpc-1.32.0_1.catalina.bottle.1.tar.gz was created when this version of abseil was released, and the pkg-config was capturing that library as link dependency.
• abseil-20200923.catalina.bottle.tar.gz is a new version of abseil, which did change something with the library (renamed, removed, moved to different place).
• my project is depend on grpc and when it builds it asks for compiler and linker flags from pkg-config which returns -labsl_dynamic_annotations amongst the other flags. But this library is not available on the machine, and the linking step of my project fails.

[SMillerDev]

Okay, so how do I reproduce this without your project? Because if it's really only your project that has this issue I doubt it's in the homebrew packaging.

[rizsotto]

Build one of the gRPC examples, which uses pkg-config in the build?

https://github.com/grpc/grpc/tree/master/examples/cpp/route_guide

[rizsotto]

Also, I could not find a web ui, which shows the content of a "bottle". Is there a way to see what files are in a brew package, without installing it?

[jonchang]

Can you open a pull request that adds this example test to the grpc formula? It's not really clear what is (or isn't) happening based on your description so far. In particular, your reproduction example is just installing the grpc formula, which, based on my testing, works correctly.

[rizsotto]

@jonchang thanks for your comment.

I think it's good idea to add the examples to enrich the test coverage. But I think that can be only useful if you run the test when abseil version changed, not when you are building grpc. (Can create a PR if you tell me how can I validate it? I don't own a mac.)

I think this would be the way to reproduce the error:

brew install grpc pkg-config
git clone [email protected]:grpc/grpc.git
cd grpc/examples/cpp/route_guide
make

[jonchang]

I think it's good idea to add the examples to enrich the test coverage. But I think that can be only useful if you run the test when abseil version changed, not when you are building grpc.

We test all reverse dependencies on our infrastructure. So if anything that grpc depends on gets rebuilt, grpc also gets tested.

Can create a PR if you tell me how can I validate it? I don't own a mac.

Our CI can verify the test. Alternatively you can try building it on Linux, though I don't know if we have grpc working on Linux yet.

[rizsotto]

@jonchang Can I ask you to run the commands I've posted in my last comment? I would like to know if it's reproduce the error I've seen. If I create a PR, that will recompile grpc package, which will solve the problem itself. (I open this ticket to ask a rebuild of grpc package :))

About the PR... I saw the current test can be modified to capture this error:

test do
    (testpath/"test.cpp").write <<~EOS
      #include <grpc/grpc.h>
      int main() {
        grpc_init();
        grpc_shutdown();
        return GRPC_STATUS_OK;
      }
    EOS
    system ENV.cc, "test.cpp", "-I#{include}", "-L#{lib}", "-lgrpc", "-o", "test"
    system "./test"
  end

I don't know rubby, but I know what would I do: instead of passing the CFLAGS and LDFLAGS directly, can we use pkg-config --cflags protobuf grpc and pkg-config --libs protobuf grpc++? Or it won't work because the directory path is not valid, because the package is not yet installed at that stage?

[SMillerDev]

It's installed. Tests are ran independent of the install so changing the parameters there seems fine.

[rizsotto]

Can you guys run these commands?

brew install grpc pkg-config
git clone [email protected]:grpc/grpc.git
cd grpc/examples/cpp/route_guide
make

• I can set up a git repo for it, but seems a bit big effort. (I still have no machine with macOS.)
• If we put this test into the grpc package test section, it will not catch this kind of errors. (And will always pass the test.) Because it creates the right pkg-config with the new flags. But that does not mean the old package won't break. The content of the pkg-config file is different in the re-built package and the old package. (It might sound weird, but from the same source it will build different pakcages, because the dependency influences what is in the pkg-config file.)

We test all reverse dependencies on our infrastructure. So if anything that grpc depends on gets rebuilt, grpc also gets tested.

How is that tested? Do you run the test section of the package description with the existing package? Or you do rebuild the package?

[SMillerDev]

How is that tested? Do you run the test section of the package description with the existing package? Or you do rebuild the package?

We just run the test section without rebuilding, unless someone specifies that it should rebuild.

[Mizux]

FYI I just bump abseil to build it against c++17 (#62803) (needed for incoming formula of Google or-tools #62790).
I'll try to test gRPC locally and see if I broke it ASAP

see: https://discourse.brew.sh/t/abseil-and-c-revision/8641

[Mizux]

grpc is broken by my PR working on a patch to fix it asap !
Fixed by #62916

[Mizux]

The problem is in the cmake config (or .pc) installed by grpc, you could add a CMakeLists.txt test project to reproduce the problem BUT cmake is only a build dependency of grpc since you can use grpc without using the cmake build system or the pkg-config file.

I don't know if homebrew support optional dependency for an optional test...

[SMillerDev]

You can just do depends_on "cmake" => :test to make it a dependency only for tests. It isn't really optional as much as it is very specific.

[rizsotto]

Thanks @Mizux . Yes, it was smell that the pkg-config file is wrong. (Those are the .pc files.) Are you installing the .cmake files file as well?

But I don't think you can use the grpc libraries without querying the flags. (Hardcode it into a project would make that code not portable.) Providing pkg-config files helps dependent packages to be portable between distributions or OSes.

IMO if the pkg-config file is installed with the package, developers will expect that to be correct. If other packaging systems do it right, then Homebrew should do it too.

[SMillerDev]

Homebrew doesn't patch anything unless upstream agrees or it's specific to Homebrew. This might be where the difference comes from. To be blunt, the gRPC cmake and pkg-config files are awful and break other formulae too. The best course of action here is for someone who knows those systems well to help them fix it.

---

Example that I recently encountered: https://github.com/Homebrew/homebrew-core/pull/62893
You can't load re2 in CMake if you also have a gRPC dependency before, because gRPCs files will happily preload all the re2 setting.

[Mizux]

https://github.com/grpc/grpc/blob/4645da201ae2c7d0b15fe56d86b41354fa4af0ca/CMakeLists.txt#L123 must be remove IF you bump abseil from 20200225 to 20200923

note: there is still not patch nor issue in grpc about it...
note2: While I didn't use grpc, I maintain Google OR-Tools (open sourcing) at Google which must rely on abseil-cpp

Breaking Changes:
Abseil no longer contains a dynamic_annotations library. Users
using a supported build system (Bazel or CMake) are unaffected by
this, but users manually specifying link libraries may get an error
about a missing linker input.

ref: https://github.com/abseil/abseil-cpp/releases/tag/20200923.1

[SMillerDev]

Could you make an issue in the gRPC repo for it?