Envoy: : Failed to load private key from /etc/example-com.key

Created on 9 Feb 2019 · 12Comments · Source: envoyproxy/envoy

Envoy cant load the key file with passphrase.

I am following https://www.learnenvoy.io/articles/ssl.html for my project purpose. I want to enable tls security in envoy. According to the page which I am following, when rebuilding docker after modifying the yaml file, it should take the key and certificate file.

Logs when runnning coomand docker-compose up

/home/docker/envoy/examples/front-proxy> docker-compose up
Recreating front-proxy_front-envoy_1 ... done
Starting front-proxy_service2_1 ... done
Starting front-proxy_service1_1 ... done
Attaching to front-proxy_front-envoy_1, front-proxy_service2_1, front-proxy_service1_1
front-envoy_1 | [2019-02-08 10:57:59.283][7][info][main] [source/server/server.cc:201] initializing epoch 0 (hot restart version=10.200.16384.127.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=2654312)
front-envoy_1 | [2019-02-08 10:57:59.284][7][info][main] [source/server/server.cc:203] statically linked extensions:
front-envoy_1 | [2019-02-08 10:57:59.284][7][info][main] [source/server/server.cc:205] access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
front-envoy_1 | [2019-02-08 10:57:59.284][7][info][main] [source/server/server.cc:208] filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.grpc_http1_reverse_bridge,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.filters.http.tap,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash
front-envoy_1 | [2019-02-08 10:57:59.284][7][info][main] [source/server/server.cc:211] filters.listener: envoy.listener.original_dst,envoy.listener.original_src,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
front-envoy_1 | [2019-02-08 10:57:59.284][7][info][main] [source/server/server.cc:214] filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.dubbo_proxy,envoy.filters.network.mysql_proxy,envoy.filters.network.rbac,envoy.filters.network.sni_cluster,envoy.filters.network.thrift_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy
front-envoy_1 | [2019-02-08 10:57:59.284][7][info][main] [source/server/server.cc:216] stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd
front-envoy_1 | [2019-02-08 10:57:59.284][7][info][main] [source/server/server.cc:218] tracers: envoy.dynamic.ot,envoy.lightstep,envoy.tracers.datadog,envoy.zipkin
front-envoy_1 | [2019-02-08 10:57:59.285][7][info][main] [source/server/server.cc:221] transport_sockets.downstream: envoy.transport_sockets.alts,envoy.transport_sockets.tap,raw_buffer,tls
front-envoy_1 | [2019-02-08 10:57:59.285][7][info][main] [source/server/server.cc:224] transport_sockets.upstream: envoy.transport_sockets.alts,envoy.transport_sockets.tap,raw_buffer,tls
front-envoy_1 | [2019-02-08 10:57:59.288][7][warning][misc] [source/common/protobuf/utility.cc:129] Using deprecated option 'envoy.api.v2.listener.Filter.config'. This configuration will be removed from Envoy soon. Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md for details.
front-envoy_1 | [2019-02-08 10:57:59.288][7][warning][misc] [source/common/protobuf/utility.cc:129] Using deprecated option 'envoy.api.v2.Cluster.hosts'. This configuration will be removed from Envoy soon. Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md for details.
front-envoy_1 | [2019-02-08 10:57:59.288][7][warning][misc] [source/common/protobuf/utility.cc:129] Using deprecated option 'envoy.api.v2.Cluster.hosts'. This configuration will be removed from Envoy soon. Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md for details.
front-envoy_1 | [2019-02-08 10:57:59.289][7][info][main] [source/server/server.cc:271] admin address: 0.0.0.0:8001
front-envoy_1 | [2019-02-08 10:57:59.290][7][info][config] [source/server/configuration_impl.cc:50] loading 0 static secret(s)
front-envoy_1 | [2019-02-08 10:57:59.290][7][info][config] [source/server/configuration_impl.cc:56] loading 2 cluster(s)
front-envoy_1 | [2019-02-08 10:57:59.293][7][info][config] [source/server/configuration_impl.cc:65] loading 1 listener(s)
front-envoy_1 | [2019-02-08 10:57:59.294][7][warning][misc] [source/common/protobuf/utility.cc:129] Using deprecated option 'envoy.config.filter.network.http_connection_manager.v2.HttpFilter.config'. This configuration will be removed from Envoy soon. Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md for details.
front-envoy_1 | [2019-02-08 10:57:59.299][7][critical][main] [source/server/server.cc:86] error initializing configuration '/etc/front-envoy.yaml': Failed to load private key from /etc/example-com.key

I am using docker version with minikube
Client:
Version: 17.09.0-ce
API version: 1.32
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:39:28 2017
OS/Arch: linux/amd64

Server:
Version: 17.09.0-ce
API version: 1.32 (minimum version 1.12)
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:45:38 2017
OS/Arch: linux/amd64
Experimental: false

question stale

Source

justancoder

Most helpful comment

@venilnoronha @subhan-nadeem Here is the solution which i found after so much research.
There is no error in dockerfile. The issue is when we generate .key and .crt file then we give passphrase. But we have to provide .key and .crt without passphrase or remove passphrase after creation. I am writing down the steps how to do that.

openssl genrsa -des3 -out server.key 2048
openssl req -new -key server.key -out server.csr
cp server.key server.key.org
openssl rsa -in server.key.org -out server.key //This will remove passphrase from key
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

Now use these server.key and server.crt files.

The reason behind this is envoy don't suppport passphrase in keys. So we have to remove it.

justancoder on 13 Feb 2019

👍4

All 12 comments

Can you paste your Dockerfile here?

venilnoronha on 12 Feb 2019

@venilnoronha I'm having the same issue.

Dockerfile:

FROM envoyproxy/envoy:latest

ADD ./example-com.crt /etc/example-com.crt
ADD ./example-com.key /etc/example-com.key

COPY ./envoy.yaml /etc/envoy/envoy.yaml

CMD /usr/local/bin/envoy -c /etc/envoy/envoy.yaml -l trace

envoy.yaml my straight from the example:

static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 80
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          codec_type: auto
          stat_prefix: ingress_http
          route_config:
            virtual_hosts:
            - name: backend
              domains:
              - "example.com"
              routes:
              - match:
                  prefix: "/"
                redirect:
                  path_redirect: "/"
                  https_redirect: true
          http_filters:
          - name: envoy.router
            config: {}
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 443
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          codec_type: auto
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains:
              - "example.com"
              routes:
              - match:
                  prefix: "/service/1"
                route:
                  cluster: service1
              - match:
                  prefix: "/service/2"
                route:
                  cluster: service2
          http_filters:
          - name: envoy.router
            config: {}
      tls_context:
        common_tls_context:
          tls_certificates:
            - certificate_chain:
                filename: "/etc/example-com.crt"
              private_key:
                filename: "/etc/example-com.key"
  clusters:
  - name: service1
    connect_timeout: 0.25s
    type: strict_dns
    lb_policy: round_robin
    http2_protocol_options: {}
    hosts:
    - socket_address:
        address: service1
        port_value: 80
  - name: service2
    connect_timeout: 0.25s
    type: strict_dns
    lb_policy: round_robin
    http2_protocol_options: {}
    hosts:
    - socket_address:
        address: service2
        port_value: 80
admin:
  access_log_path: "/dev/null"
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8001

subhan-nadeem on 13 Feb 2019

openssl genrsa -des3 -out server.key 2048
openssl req -new -key server.key -out server.csr
cp server.key server.key.org
openssl rsa -in server.key.org -out server.key //This will remove passphrase from key
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

Now use these server.key and server.crt files.

The reason behind this is envoy don't suppport passphrase in keys. So we have to remove it.

justancoder on 13 Feb 2019

👍4

I had added support for password encrypted certificates a few months ago. See
https://www.envoyproxy.io/docs/envoy/latest/api-v2/api/v2/auth/cert.proto#envoy-api-msg-auth-tlscertificate.

Essentially, you can supply the password (as file or text) via the password field similar to the certificate_chain and private_key parameters..

venilnoronha on 13 Feb 2019

yeah thanks. I should close this issue now.

justancoder on 13 Feb 2019

@venilnoronha I'm still facing the same issue despite inlining a password. Am I missing something?

I generate a certificate + private key using the following command, with PEM passphrase as "1234":

openssl req -x509 -newkey rsa:4096 -keyout example-com.key -out example-com.crt -days 365

My Dockerfile is as follows (note the added "password" field:

static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 80
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          codec_type: auto
          stat_prefix: ingress_http
          route_config:
            virtual_hosts:
            - name: backend
              domains:
              - "example.com"
              routes:
              - match:
                  prefix: "/"
                redirect:
                  path_redirect: "/"
                  https_redirect: true
          http_filters:
          - name: envoy.router
            config: {}
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 443
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          codec_type: auto
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains:
              - "example.com"
              routes:
              - match:
                  prefix: "/service/1"
                route:
                  cluster: service1
              - match:
                  prefix: "/service/2"
                route:
                  cluster: service2
          http_filters:
          - name: envoy.router
            config: {}
      tls_context:
        common_tls_context:
          tls_certificates:
            - certificate_chain:
                filename: "/etc/example-com.crt"
              private_key:
                filename: "/etc/example-com.key"
              password:
                inline_string: "1234"
  clusters:
  - name: service1
    connect_timeout: 0.25s
    type: strict_dns
    lb_policy: round_robin
    http2_protocol_options: {}
    hosts:
    - socket_address:
        address: service1
        port_value: 80
  - name: service2
    connect_timeout: 0.25s
    type: strict_dns
    lb_policy: round_robin
    http2_protocol_options: {}
    hosts:
    - socket_address:
        address: service2
        port_value: 80
admin:
  access_log_path: "/dev/null"
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8001

subhan-nadeem on 13 Feb 2019

@subhan-nadeem can you try generating it as shown in this diff: https://github.com/envoyproxy/envoy/pull/5175/files#diff-fb9b963bd49322dfcbfaf892ae4d45c6.

venilnoronha on 13 Feb 2019

@subhan-nadeem I think bit encryption should be 2048 instead of 4096

justancoder on 13 Feb 2019

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

stale[bot] on 15 Mar 2019

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.

stale[bot] on 22 Mar 2019

@venilnoronha what is san_uri_cert.cfg exactly in https://github.com/envoyproxy/envoy/pull/5175/files#diff-fb9b963bd49322dfcbfaf892ae4d45c6 ?