Google-cloud-dotnet: Failed to load gRPC lib in Docker

[deeptowncitizen]

I'm running my app in Docker (linux x64 environment).
And the service fails to start because of the error:

System.IO.IOException: Error loading native library \"/app/runtimes/linux/native/libgrpc_csharp_ext.x64.so\". Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /app/runtimes/linux/native/libgrpc_csharp_ext.x64.so)

So it is trying to load gRPC library from runtimes/.... folder. But whenever I publish the project locally (with Linux target) I see this library in the project output root. So it looks like it is trying to load the library from the wrong location.

Does anyone have the same issue?

The build command is nothing specific (tried with and without -r flag.):

RUN dotnet publish -c Release [-r linux-x64]...

[jskeet]

Transferred to google-cloud-dotnet as that's rather likely to be more appropriate (google-api-dotnet-client doesn't use gRPC) but we still really need more information. At the moment we don't have any idea what your library or the rest of your Dockerfile looks like. Please could you provide more information, ideally a complete example we can test with?

[deeptowncitizen]

This is a custom base image based on Alpine 3.10
I'm using --copy-from to get binaries to the destination image

Build image:

RUN apk update && apk upgrade && apk --no-cache add && apk add libc6-compat // -> tried with and without this lib

RUN dotnet restore ${projectFilter} -r linux-musl-x64 --configfile nuget.config
RUN dotnet publish -c Release -o ${buildDir} --self-contained=false -r linux-musl-x64 --no-restore ${projectDirectory}

tried with/without -r (both linux and musl)

[jskeet]

Alpine certainly has caused problems before - but we're really going to need a way of reproducing the problem if we're going to help you. That means having a complete Dockerfile and a complete project. (The source code doesn't need to do much - just a Main method that tries to create a channel would almost certainly be enough.)

[jskeet]

Any more information? I'll close this issue at the end of the week unless I hear anything more. (If you have information after I've closed it, please just add it in a comment and I'll reopen - it's not like closure is permanent.)

[LStuyck]

@jskeet : I'm facing the same issue here.

When running in GKE i get the following exception :
System.IO.IOException: Error loading native library "/app/runtimes/linux/native/libgrpc_csharp_ext.x64.so". Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /app/runtimes/linux/native/libgrpc_csharp_ext.x64.so)

1. Checking dotnet publish output
   When running the "dotnet publish" command locally that i'm using in my dockerfile, I find the required files in the output "\out" directory. (details below)

In this output directory i can find the files that seem to be causing the issues :

• \out\runtimes\linux\native\libgrpc_csharp_ext.x64.so
• \out\runtimes\linux\nativelibgrpc_csharp_ext.x86.so

1. Checking local docker build:
   Everything from "\out" gets copied to the "/app" directory shown in the path from exception above.
   To verify that, i added following line in my dockerfile : "RUN ls -R" and both files appear to be there when running docker build locally.

2. Checking ci/cd docker build:
   I also verified that in my azure devops pipeline those files get copied over when running docker build. (with RUN ls -R)
   These files are also present on docker build step of the project.

I'm quiet new to this, maybe i'm doing something wrong.

Any suggestions?

[jskeet]

@LStuyck: Again, what we really need is a complete example so we can reproduce the problem, although it seems very likely that the issue would be better in the gRPC repo than this one.

[LStuyck]

I'll provide a sample project beginning of next week.

[LStuyck]

@jskeet: I made an example project available here : repo

[jskeet]

@LStuyck: Thanks, I'll have a look tomorrow. (It's a public holiday today.)

I'm expecting my first step after reproducing the problem to be removing all trace of PubSub, and just changing Main to "create new Channel" as that's been enough to reproduce similar problems.

[jskeet]

Okay, some progress:

• I've reproduced the issue
• I've removed everything that's really PubSub related, including the PubSub library (so when we've got to the end of what I can do, the issue should definitely be raised somewhere else - it's not caused by anything directly in google-cloud-dotnet)
• I've removed the MS-specific bits of the project file

So I've ended up with a project file like this:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Grpc.Core" Version="2.31.0" />
  </ItemGroup>
</Project>

and C# code of:

using Grpc.Core;
using System;

namespace PubSubExceptionRepro
{
    class Program
    {
        static void Main()
        {
            try
            {
                Console.WriteLine("Main entry");

                var channel = new Channel("pubsub.googleapis.com", ChannelCredentials.Insecure);

                Console.WriteLine("Reached end");
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
                throw;
            }
        }
    }
}

This still gives the same error. Looking into why now...

[jskeet]

Okay, it looks like this is basically a dupe of #4780.

You just need to add this into Dockerfile (in the image that you'll be running, not build-env):

RUN echo 'http://dl-cdn.alpinelinux.org/alpine/v3.8/main' >> /etc/apk/repositories && \
    apk update --no-cache && \
    apk add --no-cache bash libc6-compat=1.1.19-r11

With that in place, the repro runs correctly.

See also: https://github.com/grpc/grpc/issues/21446

[LStuyck]

@jskeet: giving that a try immediately
Thank you so much for having a look at this.

[StasPerekrestov]

There is a little issue with the fix, it makes the container exposed to CVE-2019-14697 with 7.5 score.

[jskeet]

@StasPerekrestov: There may be a newer version of libc6-compat which avoids this - it's far from my area of expertise.

I'd recommend commenting on https://github.com/grpc/grpc/issues/21446 - there really isn't anything that the API libraries can do about this; it's in the gRPC library's court.

[StasPerekrestov]

@jskeet thank you for the update. I'll leave a comment under that thread.