Tikv: TiKV try to connect HTTP/1.x server

Created on 31 Mar 2020 · 8Comments · Source: tikv/tikv

Bug Report

It seems the TLS does not work on Mac

What version of TiKV are you using?

Release Version:   3.1.0-beta.2
Git Commit Hash:   acd585d8aba82343f094d094a63f16c841ecfe29
Git Commit Branch: HEAD
UTC Build Time:    2020-03-31 06:43:47
Rust Version:      rustc 1.42.0-nightly (0de96d37f 2019-12-19)

What operating system and CPU are you using?

> $ sw_vers                                                                                            
ProductName:    Mac OS X
ProductVersion: 10.14.6
BuildVersion:   18G1012

Steps to reproduce

Start pd-server

start pd-server with the following config

name = "pd"
client-urls = "https://127.0.0.1:2379"
peer-urls = "https://127.0.0.1:2380"
advertise-client-urls = "https://127.0.0.1:2379"
advertise-peer-urls = "https://127.0.0.1:2380"

[security]
cacert-path = "/Users/shirly/Downloads/0cluster/tls/myCA.pem"
cert-path = "/Users/shirly/Downloads/0cluster/tls/pd-server-mtls.pem"
key-path = "/Users/shirly/Downloads/0cluster/tls/pd-server-mtls.key"
client-cert-auth = false

start shell

./bin/pd-server -data-dir=data.pd -initial-cluster="pd=https://127.0.0.1:2380" -config pd-config.toml

And PD is working with the following log

[2020/03/31 15:56:45.875 +08:00] [INFO] [util.go:84] ["load cluster version"] [cluster-version=0.0.0]
[2020/03/31 15:56:45.875 +08:00] [INFO] [server.go:1069] ["PD cluster leader is ready to serve"] [leader-name=pd]

start tikv-server

the configuration

[security]
ca-path = "/Users/shirly/Downloads/0cluster/tls/myCA.pem"
cert-path = "/Users/shirly/Downloads/0cluster/tls/tikv-server-mtls.pem"
key-path = "/Users/shirly/Downloads/0cluster/tls/tikv-server-mtls.key"

the start shell

./bin/tikv-server --pd-endpoints="127.0.0.1:2379" --addr="127.0.0.1:20160" --data-dir=data.tikv -C "tikv-config.toml"

tikv-server start failed with the following log

[2020/03/31 15:59:04.740 +08:00] [INFO] [subchannel.cc:841] ["New connected subchannel at 0x7fa3f3015020 for subchannel 0x7fa3f0d20380"]
[2020/03/31 15:59:04.740 +08:00] [INFO] [util.rs:357] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { status: Unavailable, details: Some(\"Trying to connect an http1.x server\") }))"] [endpoints=127.0.0.1:2379]
[2020/03/31 15:59:05.040 +08:00] [INFO] [util.rs:397] ["connecting to PD endpoint"] [endpoints=127.0.0.1:2379]
[2020/03/31 15:59:05.047 +08:00] [INFO] [subchannel.cc:841] ["New connected subchannel at 0x7fa3f0d1adf0 for subchannel 0x7fa3f0e07140"]
[2020/03/31 15:59:05.047 +08:00] [INFO] [util.rs:357] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { status: Unavailable, details: Some(\"Trying to connect an http1.x server\") }))"] [endpoints=127.0.0.1:2379]

What did you expect?

tikv-server started without any error

What did happened?

It failed with PD failed to respond

componengRPC typbug

Source

AndreMouche

Most helpful comment

After investigating PD, TiKV, gRPC, Golang and negotiation protocols, I found the root reason. The problem is divided into two aspects: tikv and pd.

TiKV uses grpc-rs as its rpc framework. grpc-rs disabled ALPN support after PR merged. So since then, the negotiation of tls depends on the support of NPN which has been replaced by ALPN if you use grpc-rs with openssl. This means that the server must also provide NPN support. BTW grpc also provides security powered by boringssl where ALPN is always enabled.

PD is a go-based project, using the crypto/tls library. But in the newer go release, they has removed support for NPN here. So the pd compiled with the new version of go only provides ALPN negotiation option.

Given that NPN is old, I think this problem can be solved by enabling ALPN support by default in grpc-rs or switching back to boringssl. Related grpc-rs PR

If you want to understand the relationship between ALPN, http, tls, check here https://tools.ietf.org/html/rfc7301 https://imququ.com/post/protocol-negotiation-in-http2.html
If you want to test more details, wireshark may be a good friend.

hunterlxt on 4 Apr 2020

👍4

All 8 comments

For the same test method, I have tested that it works fine on Linux systems. I will investigate the issue after finishing the matter at hand.

hunterlxt on 31 Mar 2020

Hm, I think I was able to reproduce this on Linux, also. I just did builds of pd and tikv on Ubuntu 18.04 and I got these same tikv-server errors when trying to connect to pd-server.

$ tikv-server -V
TiKV
Release Version:   4.1.0-alpha
Git Commit Hash:   50397a6e9f9e4f80ac640ec2b4bf6b1315ebb9d7
Git Commit Branch: master
UTC Build Time:    2020-04-02 07:33:17
Rust Version:      rustc 1.42.0-nightly (0de96d37f 2019-12-19)
Enable Features:   jemalloc portable sse protobuf-codec
Profile:           release

$ pd-server -V
Release Version: v4.0.0-beta.2-36-ga22224df-dirty
Git Commit Hash: a22224df124e8a8f9261e4129751200255db6f2c
Git Branch: master
UTC Build Time:  2020-04-02 07:55:19

kolbe on 2 Apr 2020

According to tests, PD compiled on multiple platforms(ubuntu 18.04, 20.04, MacOs) will cause tikv to try to connect http/1.x server.

hunterlxt on 3 Apr 2020

After investigating PD, TiKV, gRPC, Golang and negotiation protocols, I found the root reason. The problem is divided into two aspects: tikv and pd.

Given that NPN is old, I think this problem can be solved by enabling ALPN support by default in grpc-rs or switching back to boringssl. Related grpc-rs PR

hunterlxt on 4 Apr 2020

👍4

@hunterlxt so the problem is not only with Mac, can you confirm? can you also confirm what's the Go version NPN being drop?