Serving: Documentation for Using Secure Channel

[yuanw]

I am interested to follow https://tensorflow.github.io/serving/serving_basic with secure grpc channel between client and server. (preferably SSL/TLS )

Client side seems pretty straightforward

import grpc
creds = grpc.ssl_channel_credentials(open('roots.pem').read())
channel = implementations.secure_channel(host, int(port), creds)

But I am not know how to using secure channel on server side
https://github.com/yuanwang-wf/serving/blob/master/tensorflow_serving/model_servers/main.cc

It would be great that documentations can add section on this subject

[bknl]

here is code that (mostly) does enable SSL on the server:
https://github.com/bknl/serving/commit/9bf59e625e4547be6d1927c30c4f1f4db90c2264

I am currently chasing down a bug somewhere in the build chain though that keeps this from working out of the box with the current TensorFlow Serving build rules.

The change to the BUILD file from
"@grpc//:grpc++unsecure"

to

"@grpc//:grpc++"

should do the right thing, but somehow Bazel manages to include both library elements from the unsecure and secure grpc portions, which essentially disables it.

Manually fixing the libraries does indeed give a server with secure channel with this code.

Once I figure out why grpc linking is broken in this way, I will update the BUILD instructions

[bknl]

Here is a small script to generate the necessary certificates.

#!/bin/bash

# You need to set these to something that makes sense for your client/server environment:

THECLIENTNAME=myclient.mydomain.com
THESERVERNAME=myserver.mydomain.com

# Generate valid CA
openssl genrsa -passout pass:1234 -des3 -out ca.key 4096
openssl req -passin pass:1234 -new -x509 -days 365 -key ca.key -out ca.crt -subj  "/C=DE/ST=BW/L=Walldorf/O=MYCOMPANY/OU=MYDEPARTMENT/CN=SAMPLE CA"

# Generate valid Server Key/Cert
openssl genrsa -passout pass:1234 -des3 -out server.key 4096
openssl req -passin pass:1234 -new -key server.key -out server.csr -subj  "/C=DE/ST=BW/L=Walldorf/O=MYCOMPANY/OU=MYDEPARTMENT/CN=$THESERVERNAME"
openssl x509 -req -passin pass:1234 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt

# Remove passphrase from the Server Key
openssl rsa -passin pass:1234 -in server.key -out server.key

# Generate valid Client Key/Cert
openssl genrsa -passout pass:1234 -des3 -out client.key 4096
openssl req -passin pass:1234 -new -key client.key -out client.csr -subj  "/C=DE/ST=BW/L=Walldorf/O=MYCOMPANY/OU=MYDEPARTMENT/CN=$THECLIENTNAME"
openssl x509 -passin pass:1234 -req -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out client.crt

# Remove passphrase from Client Key
openssl rsa -passin pass:1234 -in client.key -out client.key

[jkinkead]

gRPC has an odd requirement in bazel that you need to specify in a bind whether you want it compiled with SSL support, or if you want the SSL layer stubbed out. TensorFlow stubs it out insecurely, which gets pulled in to the TensorFlow Serving build.

Fortunately, you can fix this by appending the following to the workspace file:

  # Use the secure (SSL-enabled) version of gRPC.
  native.bind(
      name = "grpc_lib",
      actual = "@grpc//:grpc++",
  )

You still need to update the dependencies in tensorflow_serving/model_servers/BUILD, but it should link in the correct libraries.

FYI @bknl .

[bknl]

Thanks @jkinkead that suggestion finally worked!
I have updated the relevant files in my repository

[nicolasalt]

Unfortunately this bind() is not enough if you need to build //tensorflow:tensorflow_py (like I do).

I had to fork tensorflow to cut the hard dependencies on the unsecure grpc. It could be implemented via
bind() and merged to the main repo:
https://github.com/nicolasalt/tensorflow/tree/r1.6

[AswathKiruba]

@nicolasalt I also cut the hard dependencies on the unsecure grpc. But still, I am not able to establish a secure call. I think this because we are not installing the SSL certificate on the model server. Could you please let me know how to install the SSL certificate on the model server.

[AswathKiruba]

@jkinkead @bknl Could you please provide any insight on how to install the SSL certificate on the tensorflow model server. Thanks in advance.

[bknl]

@AswathKiruba : Well, if you are using my patches https://github.com/bknl/serving/commit/b886f28d4b49c022451a9904f2a2cc9ced444998, then the model-server just has command line arguments to take the certificate and (optionally) a custom certification authority:

  tensorflow::Flag("server_key", &server_key_path,
                       "path to private server key for SSL, "
                       "implies using SSL to connect"),
      tensorflow::Flag("server_cert", &server_cert_path,
                       "path to public server certificate for SSL, "
                       "implies using SSL to connect"),
      tensorflow::Flag("custom_ca", &custom_root_path,
                       "path to custom certificate authorities for SSL, "
                       "implies using SSL to connect"),

So you would just put the key and cert that you generate into files and the specify these files to the appropriate flags. As an example what that looks like, here is a command line using the files generated with the method outlined a few replies above.

tensorflow_model_server --port=50053 --model_config_file=sslserving.conf --server_key=server.key --server_cert=server.crt --custom_ca=ca.crt

[AswathKiruba]

@bknl Thank you for your response. I tried implementing your patches. On building tensorflow_serving using bazel build -c opt tensorflow_serving/... I am getting an error stating,
ERROR: /root/.cache/bazel/_bazel_root/f8d1071c69ea316497c31e40fe01608c/external/org_tensorflow/tensorflow/python/BUILD:3274:1: Linking of rule '@org_tensorflow//tensorflow/python:_pywrap_tensorflow
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/grpc/_objs/grpc++_unsecure/external/grpc/src/cpp/client/insecure_credentials.pic.o: multiple definition of 'grpc::InsecureChannelCredentials()

This looks like an internal compiler error. Also, I tried building it with Bazel version 0.5.4 and 0.12.0. Both resulted in the same error. Could you please let me know how to handle this.

ca.crt follow by server.crt

[bknl]

The repository was badly out of date. I synced the "sps03" branch (which contains the relevant patches) with "master" this morning and needed to resolve a few merge conflicts in the BUILD file, which may explain why you are getting the linkage error. It builds and runs clean for me now.

[AswathKiruba]

@chrisolston @bknl Could you please let me know what version of TLS protocol is being used in TF serving. Thanks in advance!!

[bknl]

Current versions of gRPC use TLS1.2

[elvys-zhang]

@bknl So what do you need when using a python client to call an SSL tensorflow serving?
The same key as server or some other files?
And what is custom_ca used for?
Thx.

[bknl]

@elvys-zhang : The client either uses "nothing" (i.e all you want is the encryption piece fom SSL, then only the server keys are needed) or a client certificate (this is used when you also check the validity of the caller. this is triggered in the code through the --client_verify flag). In either case, client and server need to agree on the validity of their respective certificates by trusting the certification authority (CA) that signed the respective keys.

gRPC has multiple ways of specifying what certificate authorities are being trusted. You can specify the trusted CA directly in your code (this is what "custom_ca" is for in my code). Alternatively gRPC will look either for a file specified in the environment variable GRPC_DEFAULT_SSL_ROOTS_FILE_PATH_ENV_VAR and if that isn't set it will look for a default list in ${INSTALLPREFIX}/share/grpc/roots.pem which is placed there when you install the whole gRPC base package.

File based solutions often don't work for my environment (the CA is local to the database my code is a part of), so my code relies on explicitly specifying the trusted CA.

For the python call see

https://grpc.io/grpc/python/grpc.html#grpc.ssl_channel_credentials

[elvys-zhang]

@bknl Thanks for your reply!
I modified my model_server/main.cc referring to your branch sps03. I started it with arguements like this

bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server --port=9000 --enable_batching=true --model_config_file=../models.conf --server_key=./backend.key --server_cert=./backend.cert --client_verify=false

In my python client

  ...
from grpc.beta import implementations
  ...
    with open('./backend.cert') as f:
      root = f.read()
      creds = implementations.ssl_channel_credentials(root, None, None)
      channel = implementations.secure_channel(host, int(port), creds)
      stub = prediction_service_pb2.beta_create_PredictionService_stub(channel) 
  ...

But it returned errors like

ssl_transport_security.cc:1228] Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
AbortionError(code=StatusCode.UNAVAILABLE, details="Connect Failed")

But if I start a python grpc server, or a golang grpc server, my python client would work fine.
Do you have any ideas about this problem?
Thx.

[AswathKiruba]

@elvys-zhang If you are using @bknl sps3 branch, start the server using the command
tensorflow_model_server --port=50053 --model_config_file=sslserving.conf --server_key=server.key --server_cert=server.crt --custom_ca=ca.crt

At your client create a roots.pem file by copying ca.crt followed by server.crt. Now use this roots.pem in your client code as mentioned below.
creds = grpc.ssl_channel_credentials(open('roots.pem').read())
channel = implementations.secure_channel(host, int(port),creds)
stub = prediction_service_pb2.beta_create_PredictionService_stub(channel)
Hope this helps!

[AswathKiruba]

@bknl Thanks for your efforts on SSL/TLS feature in TFS contribution!!

[elvys-zhang]

@AswathKiruba I am a little confused about your custom_ca. Actually, I started a python server without ca, like this

    with open('backend.key', 'rb') as f:
        private_key = f.read()
    with open('backend.cert', 'rb') as f:
        certificate_chain = f.read()

    server_credentials = grpc.ssl_server_credentials(
      ((private_key, certificate_chain,),))

    server = grpc.server(futures.ThreadPoolExecutor(max_workers=1))
    service_pb2_grpc.add_ServerServicer_to_server(ServerServicer(), server)

And called it with a Python client like this, it worked fine.

    with open('backend.cert', 'rb') as f:
        trusted_certs = f.read()

    credentials = grpc.ssl_channel_credentials(root_certificates=trusted_certs)
    channel = grpc.secure_channel('{}:{}'.format(host, port), credentials)

    stub = service_pb2_grpc.ServerStub(channel)

As you mentioned, "your client create a roots.pem file by copying ca.crt followed by server.crt". It seems that your ca.crt server.crt and root.pem is the same thing? Or did I misunderstand it? So is the ca.crt neccessary?

[AswathKiruba]

@elvys-zhang ca.crt (Certificate authority) is necessary for my approach. Your keys may already be in PEM format but just named with .crt or .key.

If the file's content begins with -----BEGIN and you can read it in a text editor:

The file uses base64, which is readable in ASCII, not a binary format. The certificate is already in PEM format.

If your keys are not in PEM format, please follow the small script to generate the necessary certificates in the above thread provided by @bknl. So once you have your keys in PEM format, on a text editor copy the contents of ca.crt be followed by server.crt and name this as roots.pem. Now follow the steps which I mentioned above for creating the server and usage of roots.pem in the client. Hope this helps!

[elvys-zhang]

@AswathKiruba Thanks for your suggestions. I will try it, thank you.

[wt-huang]

Thanks @bknl for the contribution! @AswathKiruba is correct that ca is required.
@elvys-zhang feel free to reopen this issue if the problem still persists.

[bhushanbrb]

I tried with ssl_config_file option in tensorflow model server
but got issues with it.
https://stackoverflow.com/questions/58701081/issue-with-tensorflow-model-server-with-ssl-confog-file
Since you all tried using TFS on SSL , can you please help me to know what worked for you ?thanks a lot
@bknl @wt-huang @elvys-zhang