Dgraph: Trouble setting up TLS authentication

[ospaarmann]

I'm trying to get TLS authentication up and running locally with Docker compose. I'm currently writing an Elixir client and am stuck. This is not a bug but any help is greatly appreciated since the documentation is not that clear on how to proceed. So thank you for any hint in the right direction.

Create a self-signed CA & client certificate with OpenSSL

(Source)

Generate CA key & certificate

$ openssl genrsa -out MyRootCA.key 2048
$ openssl req -x509 -new -nodes -key MyRootCA.key -sha256 -days 1024 -out MyRootCA.pem

Generate client key & certificate signing request
I use 'localhost' as Common Name

$ openssl genrsa -out MyClient1.key 2048
$ openssl req -new -key MyClient1.key -out MyClient1.csr

Generate client certificate based on our own CA certificate

$ openssl x509 -req -in MyClient1.csr -CA MyRootCA.pem -CAkey MyRootCA.key -CAcreateserial -out MyClient1.pem -days 1024 -sha256

We end up with:

MyClient1.csr
MyClient1.key
MyClient1.pem
MyRootCA.key
MyRootCA.pem
MyRootCA.srl

Adapt docker-compose.yml

_(Please note: I create the certificates with openssl in ~/exdgraph/tls and then mount this as a volume into the Dgraph server container.)_

version: "3.2"
services:
  zero:
    image: dgraph/dgraph:latest
    volumes:
      - type: volume
        source: dgraph
        target: /dgraph
        volume:
          nocopy: true
    ports:
      - 5080:5080
      - 6080:6080
    restart: on-failure
    command: dgraph zero --my=zero:5080
  server:
    image: dgraph/dgraph:latest
    volumes:
      - type: volume
        source: dgraph
        target: /dgraph
        volume:
          nocopy: true
      - ~/exdgraph/tls:/var/dgraph/tls
    ports:
      - 8080:8080
      - 9080:9080
    restart: on-failure
    command: dgraph server --my=server:7080 --memory_mb=2048 --zero=zero:5080 --tls_on --tls_ca_certs=/var/dgraph/tls/MyRootCA.pem --tls_cert=/var/dgraph/tls/MyClient1.pem --tls_cert_key=/var/dgraph/tls/MyClient1.key --tls_client_auth=REQUIREANDVERIFY
  ratel:
    image: dgraph/dgraph:latest
    volumes:
      - type: volume
        source: dgraph
        target: /dgraph
        volume:
          nocopy: true
    ports:
      - 8000:8000
    command: dgraph-ratel

volumes:
  dgraph:

Elixir Code to connect to Dgraph via gRPC

def connect(_opts) do
  ca_path = "~/exdgraph/tls/MyClient1.pem"
  cred = GRPC.Credential.new(ssl: [cacertfile: ca_path])
  opts = [cred: cred]
  case GRPC.Stub.connect("localhost:9080", opts) do
    {:ok, channel} ->
      {:ok, channel}

    _ ->
      Logger.error("ExDgraph: Connection Failed")
      {:error, ExDgraph.Error}
      # TODO: Proper error handling
  end
end

Error message when connecting / running tests:

23:01:21.188 [error] ExDgraph: Connection Failed

23:01:21.189 [error] ExDgraph.Protocol (#PID<0.300.0>) failed to connect: ** (ErlangError) Erlang error: ExDgraph.Error

23:01:21.188 [info]  ['TLS', 32, 'client', 58, 32, 73, 110, 32, 115, 116, 97, 116, 101, 32, 'cipher', 32, 'received SERVER ALERT: Fatal - Bad Certificate', 10]


 11) test mutation/2 returns {:error, error} for incorrect mutation (MutationTest)
     test/mutation_test.exs:27
     ** (DBConnection.ConnectionError) connection not available because of disconnection
     stacktrace:
       (db_connection) lib/db_connection.ex:860: DBConnection.run/3
       (ex_dgraph) lib/exdgraph/operation.ex:19: ExDgraph.Operation.operation/2
       test/mutation_test.exs:10: MutationTest.__ex_unit_setup_0/1
       test/mutation_test.exs:1: MutationTest.__ex_unit__/2

[pawanrawal]

I agree that the documentation is bad with regards to TLS. We intend to publish a tool which can generate the required certs in a folder. Then this folder can be given as input to Dgraph server while it boots up and the server automatically infers all the information from it. That might take some time.

In the meanwhile, I will look at what is wrong here and try to suggest a solution.

[pawanrawal]

Had a look at this and tried a similar example using Go. There are a couple of reasons because of which this doesn't work.

ca_path = "~/exdgraph/tls/MyClient1.pem"
cred = GRPC.Credential.new(ssl: [cacertfile: ca_path])

The ca_path should point to MyRootCA.pem.

The other reason is --tls_client_auth=REQUIREANDVERIFY while starting the Dgraph server. When you specify that option, the client must also have a certificate signed by the CA and should provide it as part of the request. Looks like you don't want that, so you should remove that and try.

[ospaarmann]

Thanks for the reply

The ca_path should point to MyRootCA.pem

But that is the certificate of the certificate authority. Shouldn't I use the client certificate in the request from the rpc client?

The other reason is --tls_client_auth=REQUIREANDVERIFY while starting the Dgraph server. When you specify that option, the client must also have a certificate signed by the CA and should provide it as part of the request. Looks like you don't want that, so you should remove that and try.

That is actually exactly what I want. I think. I want to authenticate the user via TLS to make sure not everyone can connect to the dgraph server. As I understand this is achieved via TLS client authentication with a certificate.

[pawanrawal]

I want to authenticate the user via TLS to make sure not everyone can connect to the dgraph server. As I understand this is achieved via TLS client authentication with a certificate.

Ok, so in that case, the certificates need to be loaded by the client. Right now, Dgraph server assumes that if --tls_on is set, then the server has a cert and key loaded. This would be used to authenticate that the server we are talking to is correct. Use the same procedure that you used to generate MyClient1.key and MyClient1.pem to generate MyServer1.key and MyServer1.pem. Then start the server like this:

dgraph server --my=server:7080 --memory_mb=2048 --zero=zero:5080 --tls_on --tls_ca_certs=/var/dgraph/tls/MyRootCA.pem --tls_cert=/var/dgraph/tls/MyServer1.pem --tls_cert_key=/var/dgraph/tls/MyServer1.key --tls_client_auth=REQUIREANDVERIFY

On the client, provide something like the following.

cred = GRPC.Credential.new(ssl: [certfile: 'MyClient1.pem', 
                keyfile: 'MyClient1.key`,
                cacertfile: 'MyRootCA.pem'])

[ospaarmann]

Thank you for your support! I was able to implement SSL and TLS client authentication.

https://github.com/ospaarmann/exdgraph/wiki/TLS-client-authentication

[manishrjain]

I'm not too deep into this area, but you're better off using self-signed certs.

[ondrowan]

I've tried this and generated keys by docs written by @ospaarmann. They work with Go client, however for some reason fail in both Python and Javascript with:

E0610 16:27:18.681164000 140735656280960 ssl_transport_security.cc:1063] Handshake failed with fatal error SSL_ERROR_SSL: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed.

The clients I'm using + docker-compose.yml are here: https://gist.github.com/ondrowan/e638526753d37b90ad3ca79cfd1ee9a7 The host dgraph-test is set in /etc/hosts as 127.0.0.1 dgraph-test.

[emhagman]

@ondrowan Any luck? Also running into the same issue with the JS client. I am getting the same error with the same setup.

ssl_transport_security.cc:989] Handshake failed with fatal error SSL_ERROR_SSL: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed.

[ondrowan]

@emhagman No luck unfortunatelly. Colleague set up Nginx in front of Dgraph as a workaround.

[manishrjain]

@paulftw -- Can you take a look for JS client?

[emhagman]

Separately, the GRPC NodeJS client still does not support a passphrase for keys. Just a heads up. https://github.com/grpc/grpc-node/issues/413 for anyone who was going to try to go down that path.

[emhagman]

@ondrowan I used nginx as well for now, thanks for the tip. They recently added builtin GRPC proxying so I have mutual TLS auth working with my self signed certs now

[manishrjain]

@srfrog : Can you have a look at this?

[srfrog]

@emhagman yes, I'm preparing a PR for self-signed certs and I had to get rid of key pass phrases. The other issue I have is that the pass is held in memory, which kind of defeats the point. Thanks for letting us know though.

[Vliro]

Also, using the library https://github.com/AlexandreDaSilva/DgraphNet does not work either, as it simply gives a 'Connecting Failed' error message. Using the method by @ospaarmann , and have all the proper certificates. Removing the certificates allows the connection to work properly.

[srfrog]

Also, using the library https://github.com/AlexandreDaSilva/DgraphNet does not work either, as it simply gives a 'Connecting Failed' error message. Using the method by @ospaarmann , and have all the proper certificates. Removing the certificates allows the connection to work properly.

This is a bug related to gRPC with TLS, it has been fixed in PR https://github.com/dgraph-io/dgraph/pull/2580