Tonic: Hope tonic to be running in docker container

[kevingzhang]

Feature Request

Crates

This is not only for tonic, the tower-gRPC, tower-web has the same issue when I tried all of them in docker container.

This could be related to how docker handle network conflict with how rust handle.

I do not know if I did something or not. the tonic server cannot get accessed from other docker containers within the same bridge network.

Here is what I did:

1. Have docker-compose setup a bridge network and several docker containers.
2. One of the docker containers run tonic hello world server
3. another docker container runs tonic hello world client.
4. The client container has no problem ping the server (that means they are connected via bridge network), but the gRPC won't work. Error is connect refused.
5. I tried to make sure other server won't have same trouble, so I run a node.js server and a node.js client in other two containers. The test result is all of the works except the tonic server

So I kind of make sure it is related to rust implementation on gRPC.

I tried the tower-web too. I found tower-web also has the same problem. So I guess it is related to rust.

Motivation

Running microservices is getting more and more popular. I think there must be some work around I just do not know. Can someone help to figure it out?

Proposal

Alternatives

[alce]

Hi @kevingzhang

Just to make sure I understand, can the node client talk to the tonic server?
Is the tonic server listening on 0.0.0.0 ?

[kevingzhang]

The server is listening on [::0]:50051.
If I run a client in the same container (I meant the server and client on the same container), the client can get response without any problem.
but if the client is running on another containers. The situations is

1. the client can ping the server.
2. the client cannot connect to the server at port 50051 . The result is connect refused.

I did some future experiments.

1. Replace the gRPC rust server to any rust http server. ( eg. tower-web, hyper, jsonrpc). the result is the same. Other client from other container cannot access.

1. Peplace the rust server to a Node.js server. The result is OK. Other clients can access the node server even on different containers.

So my current conclusion

1. The problem could be the rust docker container. This is something I am going to try today. I will try to build an rust container from scratch. My current test was using an official rust container.
   Or
2. The container is fine. but rust server http binding need some special config to be working inside a container which I do not know.
   Or
3. May it be related to IPV6?

[kevingzhang]

I discovered some new evidence.

I use a node container as a base image. This container is tested to work when running a node server inside. I have no trouble access this node server from other container.

I try to run the rust gRPC hello world demo server in this node container, I found it won't run because of missing library: Raspbian lite: GLIBC_2.28 not found.
Then I googled to find a solution:
add deb http://ftp.de.debian.org/debian sid main to /etc/apt/sources.list
and execute apt-get update && apt-get upgrade libc6

I did the solution and install the lib. Now the rust gprc hello world demo can run in this container without any complain.

Now. I test a client inside the same node container, it works very well. client can get server response.
However, when I test again from another container, the connect got refused as before.

In this case, the conclusion:
Before I run apt-get upgrade libc6, the containers can be accessed by other client from other container, bgut it cannot run rust due to missing library.
After I run apt-get upgrade libc6, I can run rust program inside this container, BUT, other container cannot access to this container.

So this is what i find today.

[kevingzhang]

BTW, Missing one piece. The container after installed libc6 will become unaccessable from other container, not only the rust hello world server, but also to original working node server. So that is to say, the installing libc6 make the container not accessable.

[polarathene]

Probably not the best way to investigate this for you, but when I built the examples locally it seemed to pull in some 1.39 beta version of Rust, and needed rustfmt as well. I've not used the Rust docker container(which presumably would not build atm if it needs 1.39 of rust?) so I just went with using the binaries I built locally(linux x86_64) and adding those into the container to run(which afaik is fine unless it's musl based like Alpine).

Without knowing much about networking(especially with Docker involved), I noticed the following:

• example runs fine locally with the IPV6 / 127.0.0.1 localhost/loopback(not sure which one this is called) address.
• errors in docker containers, have to use 0.0.0.0 instead(which afaik bridges all network interfaces by default?(eg if exposing a webservice on the local network, if it's bound to 127.0.0.1 but you're connected to the network over wireless, other networked devices won't be able to access that service, instead you listen/bind to 0.0.0.0)
• the server/client can now run on the same container as you mention, you can map the port to the host and also have a host tonic client reach the docker container tonic server
• cross container the 0.0.0.0 address isn't able to reach outside of itself, the containers are like their own systems, each have their own IP, if you want to route between them this way, I think you need a reverse proxy(Traefik v2 supports this for TCP iirc)
• containers(at least when spun up via docker-compose) can be reached by their defined service or container name. This requires the code in client.rs to specify the service name instead of "::0" that you've been trying. Works communicating between containers now.
• Note that on the host there is no hostname mapping to each containers actual IP(which will change each time you spin them up afaik), but it seems to work if you add .localhost as a suffix, this is not available within the containers however, so a client cannot connect from both host/container with the same approach(unless you have a fixed IP you can rely on).

Here's an example docker-compose.yaml for reproducing:

version: '3'
services:
  tonic_server:
    image: rust
    container_name: test-server
    # only required if you want to reach from host via tonic_server.localhost
    ports:
      - "50051:50051"
    volumes:
      - ./services/:/services
    command: "./services/helloworld-server"
  tonic_client:
    image: rust
    container_name: test-client
    volumes:
      - ./services/:/services
    # This is just a way to keep the container running that worked for me, so I could attach a shell to it after.
    command: "tail -f /dev/null"

Change the client.rs line to let mut client = GreeterClient::connect("http://tonic_server:50051")?;(or for client running on host use "http://tonic_server.localhost:50051", note that you can use either tonic_server service name, or the test-server container name). Compile both server and client binaries and put them into the ./services directory(or adjust that path to one that suits you).

For server.rs, I changed it to listen on [::0]:50051, so it'd be 0.0.0.0 or ipv6 equivalent instead of 127.0.0.1(or ipv6 equivalent).

You can then docker-compose up(optionally with -d if you don't want logs from the containers being picked up there). The server should be running and you can access the client with docker exec -it test-client bash or docker exec -it test-client ./services/helloworld-client if you just want to run the command and not have a shell to the test_client container up.

I noticed no difference between node and rust images. Perhaps you can provide more info on reproducing if the above is not able to resolve your issue.

[kevingzhang]

Thank you @polarathene for your such detailed instruction, even code money can follow :)

Well, I did exactly the same. To prevent any stupid mistake I made yesterday. i did it again, using the same docker compose yml. Bad luck, I still got

Error from run function Status {
    code: Unknown,
    message: "Client: buffered service failed: error trying to connect: Connection refused (os error 111)",
}

There is a big differences between mine and your environment. I am using Raspberry Pi. So that the image I was using is not the "standard" rust image, it is

docker image inspect rust
[
    {
        "Id": "sha256:f2a89aa8b31c1dde5822a63182bf0ba14301c52f4b9dea94db3aab004a5b6719",
        "RepoTags": [
            "arm32v7/rust:latest",
            "rust:latest"
        ],
        "RepoDigests": [
            "arm32v7/rust@sha256:e59477fdeee0e4a92efe60a2db89c6c35d7d3a543ba8d717afaa05307fbb1795",
            "rust@sha256:f0b507e322c99fe2b6015aa6cdfce05c5acaaf519a4628d83913189dfca438f7"
        ],
......

And, of course, my helloworld-server and helloworld-client need to be compiled into arm32v7 binary too.

I guess this could be where the problem lays.

I am just starting to try again on my Mac. Try repeat everything I did. I will post the result below after few minutes when I am done.

[kevingzhang]

Good news, I do the same thing in Mac's docker. it works.
I found onething I did wrong when I was working on raspberry pi, that was I forgot to change the [::1] to [::0] as you mentioned. I am going to do this test now. and post the result here later.

[kevingzhang]

After I tried again to set the server ip to [::0] in raspberrypi's docker. The client and server actually works !!!
Well, thank you very much. but still not fully get the differences between [::0] and [::1].

[polarathene]

0.0.0.0(IPv4) / ::0(IPv6) makes available to listen on all network interfaces on the system, so localhost(127.0.0.1), the LAN IP or WLAN IP(wifi):
https://serverfault.com/questions/876698/whats-the-difference-in-localhost-vs-0-0-0-0

https://pythonspeed.com/articles/docker-connection-refused/

Other than information there, I don't know enough about networking and docker to explain why listening on 127.0.0.1 / ::1 in the container fails for the helloworld server. The 0.0.0.0 / ::0 address will have it listen on the unique IP assigned to the container(which the service/container name maps to) that others will interact with though.

[kevingzhang]

I am not an expert on docker networking, but afaik the docker engine will assign each docker a different IP address, in my case, 172.17.0.2 for the hello world server, and 172.17.0.3 for the hello world client.
The bridge network allow them to connect to each other on 172.17.0.x. But due to some unknown reason, if the server listen to 127.0.0.1, it won't get request from 172.17.0.2. Since 0.0.0.0 listen to all of them, it works after I modify the IP to 0.0.0.0.

Although not fully understand the reason deeply, I am good enough to continue my project now as long as I set the listening IP to 0.

I think I can close this issue if no further discussion on this.

I really appreciate everyone kindly help on this issue, even it may not directly related to tonic framework.

Thank you again.

[kevingzhang]

Yeah, this https://pythonspeed.com/articles/docker-connection-refused/ (thinks polarathene) explains very well.

So I would suggest we update our example code to use [::0] instead of [::1], thoughts?

[polarathene]

Probably should get confirmation that 0.0.0.0 works on Windows. A quick google shows quite a few results of users having issues with that working.

That said, considering this is more likely to be run on Linux or within docker linux containers, it'd probably still be a better default to switch to?