Envoy: Cannot connect to gRPC service from envoy when using TLS, gets a response of 503

Created on 26 Jul 2019  路  24Comments  路  Source: envoyproxy/envoy

Title: Cannot connect to gRPC service from envoy when using TLS, get a response of 503

Description:
I have a gRPC service that is called from envoy on behalf of a web-browser client. When I have a non-TLS based website everything works. However, when I switch the website and the proxy to use TLS via a self-signed certificate, the envoy proxy returns with a 503.
Thanks for your help

Config:
```static_resources:
listeners:

  • address:
    socket_address:
    address: 0.0.0.0
    port_value: 8080
    filter_chains:

    • filters:



      • name: envoy.http_connection_manager


        config:


        codec_type: auto


        stat_prefix: ingress_http


        route_config:


        name: local_route


        virtual_hosts:





        • name: local_service



          domains:







          • "*"




            routes:




          • match:




            prefix: "/"




            route:




            cluster: AuctioneerService




            max_grpc_timeout: 0s




            cors:




            allow_origin:




          • "yams.com"




            allow_methods: GET, PUT, DELETE, POST, OPTIONS




            allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc




            max_age: "1728000"




            expose_headers: custom-header-1,grpc-status,grpc-message




            http_filters:







        • name: envoy.router



        • name: envoy.grpc_web



        • name: envoy.cors



          config: {}



          tls_context:



          common_tls_context:



          alpn_protocols: "h2"



          tls_certificates:



        • certificate_chain:



          filename: "/etc/crt"



          private_key:



          filename: "/etc/key"



          clusters:






  • name: AuctioneerService
    connect_timeout: 0.25s
    type: strict_dns
    http2_protocol_options: {}
    lb_policy: round_robin
    hosts: [{ socket_address: { address: 0.0.0.0, port_value: 9090 }}]
    admin:
    access_log_path: "/dev/null"
    address:
    socket_address:
    address: 0.0.0.0
*Logs*:
```[2019-07-26 20:35:40.223][13][debug][http] [source/common/http/conn_manager_impl.cc:600] [C0][S12928722490925780738] request headers complete (end_stream=true):
':method', 'OPTIONS'
':authority', 'yams.com:8080'
':scheme', 'https'
':path', '/auctioneer.Auctioneer/Sell'
'access-control-request-method', 'POST'
'origin', 'https://yams.com:4200'
'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36'
'access-control-request-headers', 'content-type,x-grpc-web,x-user-agent'
'accept', '*/*'
'referer', 'https://yams.com:4200/yams'
'accept-encoding', 'gzip, deflate, br'
'accept-language', 'en-US,en;q=0.9'

[2019-07-26 20:35:40.223][13][debug][http] [source/common/http/conn_manager_impl.cc:1092] [C0][S12928722490925780738] request end stream
[2019-07-26 20:35:40.224][13][debug][router] [source/common/router/router.cc:401] [C0][S12928722490925780738] cluster 'AuctioneerService' match for URL '/auctioneer.Auctioneer/Sell'
[2019-07-26 20:35:40.224][13][debug][router] [source/common/router/router.cc:514] [C0][S12928722490925780738] router decoding headers:
':method', 'OPTIONS'
':authority', 'yams.com:8080'
':scheme', 'http'
':path', '/auctioneer.Auctioneer/Sell'
'access-control-request-method', 'POST'
'origin', 'https://yams.com:4200'
'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36'
'access-control-request-headers', 'content-type,x-grpc-web,x-user-agent'
'accept', '*/*'
'referer', 'https://yams.com:4200/yams'
'accept-encoding', 'gzip, deflate, br'
'accept-language', 'en-US,en;q=0.9'
'x-forwarded-proto', 'https'
'x-request-id', 'aedda97a-27df-4d5d-9983-84eb95e24fe5'
'x-envoy-expected-rq-timeout-ms', '15000'

[2019-07-26 20:35:40.224][13][debug][pool] [source/common/http/http2/conn_pool.cc:96] [C1] creating stream
[2019-07-26 20:35:40.224][13][debug][router] [source/common/router/router.cc:1503] [C0][S12928722490925780738] pool ready
[2019-07-26 20:35:40.224][13][trace][http2] [source/common/http/http2/codec_impl.cc:559] [C1] send data: bytes=72
[2019-07-26 20:35:40.224][13][trace][connection] [source/common/network/connection_impl.cc:392] [C1] writing 72 bytes, end_stream false
[2019-07-26 20:35:40.224][13][trace][http2] [source/common/http/http2/codec_impl.cc:511] [C1] sent frame type=1
[2019-07-26 20:35:40.224][13][trace][http] [source/common/http/conn_manager_impl.cc:857] [C0][S12928722490925780738] decode headers called: filter=0x39fd450 status=1
[2019-07-26 20:35:40.224][13][trace][http2] [source/common/http/http2/codec_impl.cc:401] [C0] recv frame type=6
[2019-07-26 20:35:40.224][13][trace][http2] [source/common/http/http2/codec_impl.cc:358] [C0] dispatched 82 bytes
[2019-07-26 20:35:40.224][13][trace][http2] [source/common/http/http2/codec_impl.cc:559] [C0] send data: bytes=17
[2019-07-26 20:35:40.224][13][trace][connection] [source/common/network/connection_impl.cc:392] [C0] writing 17 bytes, end_stream false
[2019-07-26 20:35:40.224][13][trace][http2] [source/common/http/http2/codec_impl.cc:511] [C0] sent frame type=6
[2019-07-26 20:35:40.224][13][trace][connection] [source/common/network/connection_impl.cc:456] [C1] socket event: 2
[2019-07-26 20:35:40.224][13][trace][connection] [source/common/network/connection_impl.cc:541] [C1] write ready
[2019-07-26 20:35:40.224][13][trace][connection] [source/common/network/raw_buffer_socket.cc:66] [C1] write returns: 72
[2019-07-26 20:35:40.224][13][trace][connection] [source/common/network/connection_impl.cc:456] [C0] socket event: 2
[2019-07-26 20:35:40.224][13][trace][connection] [source/common/network/connection_impl.cc:541] [C0] write ready
[2019-07-26 20:35:40.224][13][trace][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:234] [C0] ssl write returns: 17
[2019-07-26 20:35:40.224][13][trace][connection] [source/common/network/connection_impl.cc:456] [C0] socket event: 2
[2019-07-26 20:35:40.224][13][trace][connection] [source/common/network/connection_impl.cc:541] [C0] write ready
[2019-07-26 20:35:40.224][13][trace][connection] [source/common/network/connection_impl.cc:456] [C1] socket event: 3
[2019-07-26 20:35:40.224][13][trace][connection] [source/common/network/connection_impl.cc:541] [C1] write ready
[2019-07-26 20:35:40.224][13][trace][connection] [source/common/network/connection_impl.cc:494] [C1] read ready
[2019-07-26 20:35:40.224][13][trace][connection] [source/common/network/raw_buffer_socket.cc:23] [C1] read returns: 30
[2019-07-26 20:35:40.224][13][trace][connection] [source/common/network/raw_buffer_socket.cc:37] [C1] read error: Resource temporarily unavailable
[2019-07-26 20:35:40.224][13][trace][http2] [source/common/http/http2/codec_impl.cc:343] [C1] dispatching 30 bytes
[2019-07-26 20:35:40.224][13][trace][http2] [source/common/http/http2/codec_impl.cc:401] [C1] recv frame type=3
[2019-07-26 20:35:40.224][13][trace][http2] [source/common/http/http2/codec_impl.cc:497] [C1] remote reset: 2
[2019-07-26 20:35:40.224][13][debug][http2] [source/common/http/http2/codec_impl.cc:568] [C1] stream closed: 2
[2019-07-26 20:35:40.224][13][debug][client] [source/common/http/codec_client.cc:105] [C1] request reset
[2019-07-26 20:35:40.224][13][trace][main] [source/common/event/dispatcher_impl.cc:158] item added to deferred deletion list (size=1)
[2019-07-26 20:35:40.224][13][debug][pool] [source/common/http/http2/conn_pool.cc:233] [C1] destroying stream: 0 remaining
[2019-07-26 20:35:40.224][13][debug][router] [source/common/router/router.cc:868] [C0][S12928722490925780738] upstream reset: reset reason remote reset
[2019-07-26 20:35:40.224][13][debug][http] [source/common/http/conn_manager_impl.cc:1167] [C0][S12928722490925780738] Sending local reply with details upstream_reset_before_response_started{remote reset}
[2019-07-26 20:35:40.224][13][trace][http] [source/common/http/conn_manager_impl.cc:1254] [C0][S12928722490925780738] encode headers called: filter=0x39f92c0 status=0
[2019-07-26 20:35:40.224][13][trace][http] [source/common/http/conn_manager_impl.cc:1254] [C0][S12928722490925780738] encode headers called: filter=0x39f9600 status=0
[2019-07-26 20:35:40.224][13][debug][http] [source/common/http/conn_manager_impl.cc:1359] [C0][S12928722490925780738] encoding headers via codec (end_stream=false):
':status', '503'
'content-length', '85'
'content-type', 'text/plain'
'date', 'Fri, 26 Jul 2019 20:35:39 GMT'
'server', 'envoy'

[2019-07-26 20:35:40.224][13][trace][http2] [source/common/http/http2/codec_impl.cc:559] [C0] send data: bytes=41
[2019-07-26 20:35:40.224][13][trace][connection] [source/common/network/connection_impl.cc:392] [C0] writing 41 bytes, end_stream false
[2019-07-26 20:35:40.224][13][trace][http2] [source/common/http/http2/codec_impl.cc:511] [C0] sent frame type=1
[2019-07-26 20:35:40.224][13][trace][http] [source/common/http/conn_manager_impl.cc:1484] [C0][S12928722490925780738] encode data called: filter=0x39f92c0 status=0
[2019-07-26 20:35:40.224][13][trace][http] [source/common/http/conn_manager_impl.cc:1484] [C0][S12928722490925780738] encode data called: filter=0x39f9600 status=0
[2019-07-26 20:35:40.224][13][trace][http] [source/common/http/conn_manager_impl.cc:1497] [C0][S12928722490925780738] encoding data via codec (size=85 end_stream=true)
[2019-07-26 20:35:40.224][13][trace][connection] [source/common/network/connection_impl.cc:392] [C0] writing 94 bytes, end_stream false
[2019-07-26 20:35:40.224][13][trace][http2] [source/common/http/http2/codec_impl.cc:511] [C0] sent frame type=0
[2019-07-26 20:35:40.224][13][debug][http2] [source/common/http/http2/codec_impl.cc:568] [C0] stream closed: 0
[2019-07-26 20:35:40.224][13][trace][main] [source/common/event/dispatcher_impl.cc:158] item added to deferred deletion list (size=2)
[2019-07-26 20:35:40.224][13][trace][main] [source/common/event/dispatcher_impl.cc:158] item added to deferred deletion list (size=3)
[2019-07-26 20:35:40.224][13][trace][main] [source/common/event/dispatcher_impl.cc:158] item added to deferred deletion list (size=4)
[2019-07-26 20:35:40.224][13][trace][http2] [source/common/http/http2/codec_impl.cc:401] [C1] recv frame type=6
[2019-07-26 20:35:40.224][13][trace][http2] [source/common/http/http2/codec_impl.cc:358] [C1] dispatched 30 bytes
[2019-07-26 20:35:40.224][13][trace][http2] [source/common/http/http2/codec_impl.cc:559] [C1] send data: bytes=17
[2019-07-26 20:35:40.224][13][trace][connection] [source/common/network/connection_impl.cc:392] [C1] writing 17 bytes, end_stream false
question
Source
picture bitsmaker

Most helpful comment

@bitsmaker Tried the above No success.
On checking envoy logs i am getting following error.
TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
but the same certificate is working when I am creating grpc client and using it without grpc-web.

picture kumaranshu72 on 11 Jun 2020
👍2

All 24 comments

curl response

curl -v --cacert ~/Documents/yams.com.cer https://yams.com:8080/auctioneer.Auctioneer/Sell
*   Trying 192.168.0.17...
* TCP_NODELAY set
* Connected to yams.com (192.168.0.17) port 8080 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: yams.com
> GET /auctioneer.Auctioneer/Sell HTTP/1.1
> Host: yams.com:8080
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< content-length: 85
< content-type: text/plain
< date: Tue, 30 Jul 2019 03:56:35 GMT
< server: envoy
<
* Curl_http_done: called premature == 0
* Connection #0 to host yams.com left intact
upstream connect error or disconnect/reset before headers. reset reason: remote reset
bitsmaker picture bitsmaker on 30 Jul 2019

I was able to print the grpc logs and its complaining (not 100% sure) about the missing header TE
I0730 10:10:13.369444613 15315 parsing.cc:656] parsing initial_metadata I0730 10:10:13.369512486 15315 hpack_parser.cc:636] Decode: ':method: OPTIONS', elem_interned=1 [1], k_interned=1, v_interned=1 I0730 10:10:13.369530691 15315 parsing.cc:407] HTTP:3:HDR:SVR: :method: 4f 50 54 49 4f 4e 53 'OPTIONS' I0730 10:10:13.369545205 15315 hpack_parser.cc:636] Decode: ':authority: yams.com:8080', elem_interned=1 [1], k_interned=1, v_interned=1 I0730 10:10:13.369558031 15315 parsing.cc:407] HTTP:3:HDR:SVR: :authority: 79 61 6d 73 2e 63 6f 6d 3a 38 30 38 30 'yams.com:8080' I0730 10:10:13.369570016 15315 hpack_parser.cc:636] Decode: ':scheme: http', elem_interned=1 [3], k_interned=1, v_interned=1 I0730 10:10:13.369582195 15315 parsing.cc:407] HTTP:3:HDR:SVR: :scheme: 68 74 74 70 'http' I0730 10:10:13.369600878 15315 hpack_parser.cc:636] Decode: ':path: /auctioneer.Auctioneer/Sell', elem_interned=0 [2], k_interned=1, v_interned=0 I0730 10:10:13.369617580 15315 parsing.cc:407] HTTP:3:HDR:SVR: :path: 2f 61 75 63 74 69 6f 6e 65 65 72 2e 41 75 63 74 69 6f 6e 65 65 72 2f 53 65 6c 6c '/auctioneer.Auctioneer/Sell' I0730 10:10:13.369630817 15315 hpack_parser.cc:636] Decode: 'access-control-request-method: POST', elem_interned=1 [1], k_interned=1, v_interned=1 I0730 10:10:13.369643143 15315 parsing.cc:407] HTTP:3:HDR:SVR: access-control-request-method: 50 4f 53 54 'POST' I0730 10:10:13.369655095 15315 hpack_parser.cc:636] Decode: 'origin: https://yams.com:4200', elem_interned=1 [1], k_interned=1, v_interned=1 I0730 10:10:13.369669192 15315 parsing.cc:407] HTTP:3:HDR:SVR: origin: 68 74 74 70 73 3a 2f 2f 79 61 6d 73 2e 63 6f 6d 3a 34 32 30 30 'https://yams.com:4200' I0730 10:10:13.369682710 15315 hpack_parser.cc:636] Decode: 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36', elem_interned=1 [1], k_interned=1, v_interned=1 I0730 10:10:13.369701761 15315 parsing.cc:407] HTTP:3:HDR:SVR: user-agent: 4d 6f 7a 69 6c 6c 61 2f 35 2e 30 20 28 4d 61 63 69 6e 74 6f 73 68 3b 20 49 6e 74 65 6c 20 4d 61 63 20 4f 53 20 58 20 31 30 5f 31 32 5f 32 29 20 41 70 70 6c 65 57 65 62 4b 69 74 2f 35 33 37 2e 33 36 20 28 4b 48 54 4d 4c 2c 20 6c 69 6b 65 20 47 65 63 6b 6f 29 20 43 68 72 6f 6d 65 2f 37 35 2e 30 2e 33 37 37 30 2e 31 30 30 20 53 61 66 61 72 69 2f 35 33 37 2e 33 36 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36' I0730 10:10:13.369717007 15315 hpack_parser.cc:636] Decode: 'access-control-request-headers: content-type,x-grpc-web,x-user-agent', elem_interned=1 [1], k_interned=1, v_interned=1 I0730 10:10:13.369731866 15315 parsing.cc:407] HTTP:3:HDR:SVR: access-control-request-headers: 63 6f 6e 74 65 6e 74 2d 74 79 70 65 2c 78 2d 67 72 70 63 2d 77 65 62 2c 78 2d 75 73 65 72 2d 61 67 65 6e 74 'content-type,x-grpc-web,x-user-agent' I0730 10:10:13.369765157 15315 hpack_parser.cc:636] Decode: 'accept: */*', elem_interned=1 [1], k_interned=1, v_interned=1 I0730 10:10:13.369777742 15315 parsing.cc:407] HTTP:3:HDR:SVR: accept: 2a 2f 2a '*/*' I0730 10:10:13.369789382 15315 hpack_parser.cc:636] Decode: 'referer: https://yams.com:4200/yams', elem_interned=1 [1], k_interned=1, v_interned=1 I0730 10:10:13.369803135 15315 parsing.cc:407] HTTP:3:HDR:SVR: referer: 68 74 74 70 73 3a 2f 2f 79 61 6d 73 2e 63 6f 6d 3a 34 32 30 30 2f 79 61 6d 73 'https://yams.com:4200/yams' I0730 10:10:13.369815076 15315 hpack_parser.cc:636] Decode: 'accept-encoding: gzip, deflate, br', elem_interned=1 [1], k_interned=1, v_interned=1 I0730 10:10:13.369828409 15315 parsing.cc:407] HTTP:3:HDR:SVR: accept-encoding: 67 7a 69 70 2c 20 64 65 66 6c 61 74 65 2c 20 62 72 'gzip, deflate, br' I0730 10:10:13.369840457 15315 hpack_parser.cc:636] Decode: 'accept-language: en-US,en;q=0.9', elem_interned=1 [1], k_interned=1, v_interned=1 I0730 10:10:13.369852252 15315 parsing.cc:407] HTTP:3:HDR:SVR: accept-language: 65 6e 2d 55 53 2c 65 6e 3b 71 3d 30 2e 39 'en-US,en;q=0.9' I0730 10:10:13.369863687 15315 hpack_parser.cc:636] Decode: 'x-forwarded-proto: https', elem_interned=1 [1], k_interned=1, v_interned=1 I0730 10:10:13.369875178 15315 parsing.cc:407] HTTP:3:HDR:SVR: x-forwarded-proto: 68 74 74 70 73 'https' I0730 10:10:13.369895124 15315 hpack_parser.cc:636] Decode: 'x-request-id: 9dcfa137-4ef6-447d-9acf-3ea294711193', elem_interned=1 [1], k_interned=1, v_interned=1 I0730 10:10:13.369910401 15315 parsing.cc:407] HTTP:3:HDR:SVR: x-request-id: 39 64 63 66 61 31 33 37 2d 34 65 66 36 2d 34 34 37 64 2d 39 61 63 66 2d 33 65 61 32 39 34 37 31 31 31 39 33 '9dcfa137-4ef6-447d-9acf-3ea294711193' I0730 10:10:13.369923519 15315 hpack_parser.cc:636] Decode: 'x-envoy-expected-rq-timeout-ms: 15000', elem_interned=1 [1], k_interned=1, v_interned=1 I0730 10:10:13.369935235 15315 parsing.cc:407] HTTP:3:HDR:SVR: x-envoy-expected-rq-timeout-ms: 31 35 30 30 30 '15000' I0730 10:10:13.369958806 15315 chttp2_transport.cc:1710] perform_stream_op[s=0x7f705c01d758]: RECV_INITIAL_METADATA I0730 10:10:13.369980235 15315 chttp2_transport.cc:1406] perform_stream_op_locked: RECV_INITIAL_METADATA; on_complete = (nil) I0730 10:10:13.370101165 15315 chttp2_transport.cc:1710] perform_stream_op[s=0x7f705c01d758]: CANCEL:{"created":"@1564506613.369999233","description":"Missing :authority or :path","file":"src/core/lib/surface/server.cc","file_line":772,"referenced_errors":[{"created":"@1564506613.369995848","description":"Failed processing incoming headers","file":"src/core/ext/filters/http/server/http_server_filter.cc","file_line":122,"referenced_errors":[{"created":"@1564506613.369993641","description":"Bad header","file":"src/core/ext/filters/http/server/http_server_filter.cc","file_line":154,"key":":method","value":"OPTIONS"},{"created":"@1564506613.369996896","description":"Missing header","file":"src/core/ext/filters/http/server/http_server_filter.cc","file_line":178,"key":"te"}]}]} I0730 10:10:13.370138153 15315 chttp2_transport.cc:1406] perform_stream_op_locked: CANCEL:{"created":"@1564506613.369999233","description":"Missing :authority or :path","file":"src/core/lib/surface/server.cc","file_line":772,"referenced_errors":[{"created":"@1564506613.369995848","description":"Failed processing incoming headers","file":"src/core/ext/filters/http/server/http_server_filter.cc","file_line":122,"referenced_errors":[{"created":"@1564506613.369993641","description":"Bad header","file":"src/core/ext/filters/http/server/http_server_filter.cc","file_line":154,"key":":method","value":"OPTIONS"},{"created":"@1564506613.369996896","description":"Missing header","file":"src/core/ext/filters/http/server/http_server_filter.cc","file_line":178,"key":"te"}]}]}; on_complete = 0x7f705c01f430 I0730 10:10:13.370166189 15315 chttp2_transport.cc:839] W:0x7f705c0162c0 SERVER [ipv4:192.168.0.17:38784] state IDLE -> WRITING [RST_STREAM]

mmahimtura picture mmahimtura on 30 Jul 2019

Can someone in the contributors please look at this issue? It is hampering me from moving forward. I have already done the analysis and found that the envoy proxy is not sending the 'TE' header as expected by gRPC.

bitsmaker picture bitsmaker on 31 Jul 2019

I don't think Envoy will add a TE header unless it is converting from non-gRPC or web-gRPC to H2-gRPC using the json transcoder filter or the gRPC web filter. Basically if you're changing the data such that the source would not send TE and the other endpoint expects it, Envoy will add TE. If the data is simply being proxied, it's up to the other endpoint to add TE.

cc @lizan for correctness

alyssawilk picture alyssawilk on 31 Jul 2019

Thanks for your response.

The source of the request is from a browser. I am guessing that's web-gRPC. The destination of the request is a C++ service. When I use non-TLS it works fine, I can see the TE header was added. But when I use TLS thats when the TE header goes missing.

If I am reading your statement correctly, in both my cases its going from web-gRPC to H2-gRPC. Shouldn't it add the TE header then?

mmahimtura picture mmahimtura on 31 Jul 2019

If your downstream is gRPC-web and your upstream is H2 GRPC you need the grpc web filter.
Looking at your config

      http_filters:
      - name: envoy.router
      - name: envoy.grpc_web
      - name: envoy.cors

Are your filters in inverted? The filters in Envoy config operate top down, which means you're routing to your upstream before applying gRPC web transformations.

alyssawilk picture alyssawilk on 31 Jul 2019

Thank you, I am new to envoy and web programming, please pardon my ignorance. Should I use this order then?

http_filters:
      - name: envoy.grpc_web
      - name: envoy.router
      - name: envoy.cors

If you can, could you please explain why it was working for the non-TLS scenario?

mmahimtura picture mmahimtura on 31 Jul 2019

I just tried using this order in the TLS scenario

http_filters:
          - name: envoy.grpc_web
          - name: envoy.router
          - name: envoy.cors

Its still missing the TE header and does not work

mmahimtura picture mmahimtura on 31 Jul 2019

First, your filters are still out of order - gRPC web will now work but CORS won't. I'm going to go fix this in #7779 since it has confused many people.
Second, I don't think the content type you're using is a content type Envoy respects

I believe the web-grpc filter only acts if the content type header matches the ones below.

const std::string GrpcWeb{"application/grpc-web"};                                               
const std::string GrpcWebProto{"application/grpc-web+proto"};                                    
const std::string GrpcWebText{"application/grpc-web-text"};                                      
const std::string GrpcWebTextProto{"application/grpc-web-text+proto"};

Looking at your request headers I was fooled into thinking you had content type set but you only have
'access-control-request-headers', 'content-type,x-grpc-web,x-user-agent'

I believe your client needs to inform Envoy this is a web-gRPC request for this to work. I can't speak to why the TLS path works, sorry. I'm not terribly familiar with Envoy gRPC transforms, and only checked in because it was linked to the potential non-gRPC TE issue.

alyssawilk picture alyssawilk on 31 Jul 2019

Thanks for your response but in the non-TLS (working) scenario, I see the same header for content-type 

2019-07-31 20:29:59.431][15][debug][http] [source/common/http/conn_manager_impl.cc:600] [C0][S9091586732190440070] request headers complete (end_stream=true):
':authority', 'yams.com:8080'
':path', '/auctioneer.Auctioneer/Sell'
':method', 'OPTIONS'
'connection', 'keep-alive'
'access-control-request-method', 'POST'
'origin', 'http://yams.com:4200'
'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36'
'access-control-request-headers', 'content-type,x-grpc-web,x-user-agent'
'accept', '*/*'
'referer', 'http://yams.com:4200/yams'
'accept-encoding', 'gzip, deflate'
'accept-language', 'en-US,en;q=0.9'

[2019-07-31 20:29:59.431][15][debug][http] [source/common/http/conn_manager_impl.cc:1092] [C0][S9091586732190440070] request end stream
[2019-07-31 20:29:59.431][15][trace][http] [source/common/http/conn_manager_impl.cc:857] [C0][S9091586732190440070] decode headers called: filter=0x35cf2c0 status=0
[2019-07-31 20:29:59.431][15][trace][http] [source/common/http/conn_manager_impl.cc:1254] [C0][S9091586732190440070] encode headers called: filter=0x35a6ec0 status=0
[2019-07-31 20:29:59.431][15][trace][http] [source/common/http/conn_manager_impl.cc:1254] [C0][S9091586732190440070] encode headers called: filter=0x35a7d80 status=0
[2019-07-31 20:29:59.431][15][debug][http] [source/common/http/conn_manager_impl.cc:1359] [C0][S9091586732190440070] encoding headers via codec (end_stream=true):
':status', '200'
'access-control-allow-origin', 'http://yams.com:4200'
'access-control-allow-methods', 'GET, PUT, DELETE, POST, OPTIONS'
'access-control-allow-headers', 'keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout'
'access-control-max-age', '1728000'
'access-control-expose-headers', 'custom-header-1,grpc-status,grpc-message'
'date', 'Wed, 31 Jul 2019 20:29:59 GMT'
'server', 'envoy'
mmahimtura picture mmahimtura on 31 Jul 2019

Please note that the client generating the requests was generated per the grpc-web documentation, ie using the protoc executable with --grpc-web_out=import_style=commonjs,mode=grpcwebtext. I did not hand-code it

mmahimtura picture mmahimtura on 31 Jul 2019

the CORS request doesn't have request body so there is no content-type in your request, you need CORS filter before the router to handle the CORS request correctly.

btw your cluster is not configured with TLS:

  clusters:
  - name: AuctioneerService
    connect_timeout: 0.25s
    type: strict_dns
    http2_protocol_options: {}
    lb_policy: round_robin
    hosts: [{ socket_address: { address: 0.0.0.0, port_value: 9090 }}]

So if you changed upstream with TLS, you need tls_context here too.

lizan picture lizan on 1 Aug 2019

Thanks for your response. Could you please confirm that my order should be 

http_filters:
          - name: envoy.cors
          - name: envoy.grpc_web
          - name: envoy.router

Also is there an example of the tls_context for the section you mentioned above? When you say upstream are you referring to my C++ service? If so, thats not using TLS. In that case do I still need the tls_context in clusters?

mmahimtura picture mmahimtura on 1 Aug 2019

I tried the above and I got the same thing. I really need help with this as I am at a complete standstill. Please note that non-TLS works without any of the changes suggested above. TLS does not because gRPC complains that the TE header is missing. Is there some code in envoy that strips that header?
Thanks again for all your help

bitsmaker picture bitsmaker on 2 Aug 2019

@bitsmaker interesting, are you able to make a quick reproducer with possibly docker-compose? I will take a deeper look to that.

lizan picture lizan on 8 Aug 2019

Thanks @lizan but as it turns out, I found what I was doing wrong. At some point in time I had set

cors:
                allow_origin:
                - "yams.com"

instead of

cors:
                allow_origin:
                - "*"

I am new to web related concepts and envoy. Thanks to you and @alyssawilk for looking and sorry if I wasted your time

mmahimtura picture mmahimtura on 8 Aug 2019

Glad you're sorted out!

alyssawilk picture alyssawilk on 12 Aug 2019

@mmahimtura Can you please help me out. I am facing the same issue.

kumaranshu72 picture kumaranshu72 on 10 Jun 2020

@kumaranshu72 I mentioned above that there was an issue in my file. Please see my comments above.

bitsmaker picture bitsmaker on 10 Jun 2020

@bitsmaker After adding the following config in envoy.yaml
tls_context: common_tls_context: tls_certificates: - certificate_chain: filename: "/etc/server.crt" private_key: filename: "/etc/server.key"
I am getting the following error.
{code: 2, message: "Http response at 400 or 500 level"}

Everything is working fine if I remove TLS from the server-side. But not working when TSL is required.
You can also check the details code I have mentioned here.
https://stackoverflow.com/questions/62303049/grpc-web-connectivity-issue-in-tls

kumaranshu72 picture kumaranshu72 on 10 Jun 2020
👍1

@kumaranshu72 Can yo please try changing the order of the filters to

http_filters:
          - name: envoy.cors
          - name: envoy.grpc_web
          - name: envoy.router
bitsmaker picture bitsmaker on 10 Jun 2020

@bitsmaker Tried the above No success.
On checking envoy logs i am getting following error.
TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
but the same certificate is working when I am creating grpc client and using it without grpc-web.

kumaranshu72 picture kumaranshu72 on 11 Jun 2020
👍2

@kumaranshu72 likely you have a different issue, can you open a new GH issue with your config/logs as detail as possible?

lizan picture lizan on 11 Jun 2020

@lizan created a new issues https://github.com/envoyproxy/envoy/issues/11582

kumaranshu72 picture kumaranshu72 on 13 Jun 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

karthequian picture karthequian  路  3Comments
dstrelau picture dstrelau  路  3Comments
sabiurr picture sabiurr  路  3Comments
alkov-ibm picture alkov-ibm  路  3Comments
boncheo picture boncheo  路  3Comments