tikv-ctl compact run infinitely

[Connor1996]

Bug Report

When tikv-ctl --host is set with url with a schema like 172.0.0.1:20161, compact runs successfully and returns soon. But once the url has a schema like http://172.0.0.1:20161, compact will run infinitely and actually it does nothing.

[poonai]

I would like to take this

 pub fn secure_connect(mut self, addr: &str, mut creds: ChannelCredentials) -> Channel {
            let args = self.prepare_connect_args();
           // does the http prefix has to be removed?
            let addr = CString::new(addr).unwrap();
            let addr_ptr = addr.as_ptr();
            let channel = unsafe {
                grpc_sys::grpc_secure_channel_create(
                    creds.as_mut_ptr(),
                    addr_ptr,
                    args.args,
                    ptr::null_mut(),
                )
            };

            Channel::new(self.env.pick_cq(), self.env, channel)
        }

does the http prefix has to be removed on connect?

[zhangjinpeng1987]

@Connor1996 We need to find out why it can not work when has http:// prefix.

[Connor1996]

@sch00lb0y only remove http prefix is not elegant, any invalid url will cause it run infinitely. IMO, it is better to return some error info.

[overvenus]

Looks like grpc dns resolution failed.

GRPC_VERBOSITY=debug RUST_BACKTRACE=1 tikv-ctl --host http://127.0.0.1:2379 compact
I1012 18:30:00.103591814   19552 ev_epoll1_linux.c:93]       grpc epoll fd: 3
D1012 18:30:00.103678277   19552 ev_posix.c:107]             Using polling engine: epoll1
D1012 18:30:00.103723979   19552 dns_resolver.c:301]         Using native dns resolver
I1012 18:30:00.157116868   19554 dns_resolver.c:170]         dns resolution failed (will retry): {"created":"@1539340200.157073771","description":"OS Error","errno":-2,"file":"/home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.2.3/grpc/src/core/lib/iomgr/resolve_address_posix.c","file_line":104,"os_error":"Name or service not known","syscall":"getaddrinfo","target_address":"http://127.0.0.1:2379"}
D1012 18:30:00.157149170   19554 dns_resolver.c:176]         retrying in 1.000000000 seconds
I1012 18:30:01.206784191   19554 dns_resolver.c:170]         dns resolution failed (will retry): {"created":"@1539340201.206747007","description":"OS Error","errno":-2,"file":"/home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.2.3/grpc/src/core/lib/iomgr/resolve_address_posix.c","file_line":104,"os_error":"Name or service not known","syscall":"getaddrinfo","target_address":"http://127.0.0.1:2379"}
D1012 18:30:01.206818784   19554 dns_resolver.c:176]         retrying in 1.000000000 seconds
I1012 18:30:02.258244236   19554 dns_resolver.c:170]         dns resolution failed (will retry): {"created":"@1539340202.258199163","description":"OS Error","errno":-2,"file":"/home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.2.3/grpc/src/core/lib/iomgr/resolve_address_posix.c","file_line":104,"os_error":"Name or service not known","syscall":"getaddrinfo","target_address":"http://127.0.0.1:2379"}
D1012 18:30:02.258287685   19554 dns_resolver.c:176]         retrying in 1.000000000 seconds
I1012 18:30:03.312975073   19554 dns_resolver.c:170]         dns resolution failed (will retry): {"created":"@1539340203.312935103","description":"OS Error","errno":-2,"file":"/home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.2.3/grpc/src/core/lib/iomgr/resolve_address_posix.c","file_line":104,"os_error":"Name or service not known","syscall":"getaddrinfo","target_address":"http://127.0.0.1:2379"}
D1012 18:30:03.313031626   19554 dns_resolver.c:176]         retrying in 1.000000000 seconds

[poonai]

@Connor1996 @zhangjinpeng1987 @overvenus I did some investigation on this with help of log.
Error is happening here https://github.com/grpc/grpc/blob/master/src/core/lib/iomgr/resolve_address_posix.cc#L87
cause for the bug is not parsing the hostname and port properly.(https://github.com/grpc/grpc/blob/master/src/core/lib/gpr/host_port.cc#L41)
They passing http://localhost:80 in getaddrinfo instead of localhost:80

I'm not sure whether it is not convention to use http.

But the problem is not in tikv, It's in grpc core library.

[poonai]

And I've tried this with sample server. It was panicked and exited.
When I tried to compile tikv with latest grpc-rs,I was not able to compile.
I think, This issue will be solved with the new grpc-rs

[Connor1996]

@overvenus /cc

[overvenus]

Yes, it is solved with grpc-rs 0.4.

GRPC_VERBOSITY=debug RUST_BACKTRACE=1 ./tikv-ctl --host http://127.0.0.1:2379 metrics
D1017 00:01:47.164783511   66453 ev_posix.cc:145]            Using polling engine: epollex
D1017 00:01:47.164902734   66453 dns_resolver.cc:339]        Using native dns resolver
D1017 00:01:47.165200193   66453 dns_resolver.cc:280]        Start resolving.
I1017 00:01:47.217677672   66454 dns_resolver.cc:210]        dns resolution failed (will retry): {"created":"@1539705707.217636686","description":"OS Error","errno":-2,"file":"/home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.4.0/grpc/src/core/lib/iomgr/resolve_address_posix.cc","file_line":108,"os_error":"Name or service not known","syscall":"getaddrinfo","target_address":"http://127.0.0.1:2379"}
D1017 00:01:47.217708128   66454 dns_resolver.cc:221]        retrying in 1000 milliseconds
DebugClient::metrics: RpcFailure(RpcStatus { status: Unavailable, details: Some("Name resolution failure") })

[Connor1996]

@sch00lb0y Thanks a lot😄