Envoy: gRPC-JSON and CORS and OPTIONS & HEAD methods

[mattbailey]

Description:

gRPC-JSON transcoding (and http gRPC annotations) do not have support for the OPTIONS or HEAD http methods. Currently, if I send an OPTIONS request, the gRPC-JSON transcoder fails to handle the request:

[2019-04-16T20:20:04.741Z] "OPTIONS /hello HTTP/1.1" 503 UR 0 85 10 - "-" "curl/7.54.0" "46f2e4ff-4580-43b4-9538-2f76341cc040" "localhost:11000" "192.168.65.2:10
000"

HEAD behaves similarly, but less breaking for CORS preflight requests.

The result of this is that CORS is more or less unusable with gRPC-JSON transcoding for a number of scenarios that require preflight OPTIONS checks.

Is there some way to make the direct_response route handler pass through a specific filter like envoy.cors ?

[lizan]

@mattbailey what's your envoy config?

[mattbailey]

@lizan

Was experimenting with direct_response if you swap commented/uncommented on the routes:

admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address: { address: 0.0.0.0, port_value: 9901 }

static_resources:
  listeners:
  - name: listener1
    address:
      socket_address: { address: 0.0.0.0, port_value: 11000 }
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
          stat_prefix: grpc_json
          codec_type: AUTO
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              cors:
                allow_origin_regex:
                - "https?:\\/\\/localhost:\\d+"
                allow_credentials: true
                allow_methods: GET, PUT, DELETE, POST, PATCH, OPTIONS
                allow_headers: content-type,origin,authorization
                allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web
                expose_headers: grpc-status,grpc-message,x-envoy-upstream-service-time
              routes:
              - match: { prefix: "/" }
                route: { cluster: grpc, timeout: { seconds: 60 } }
              #- match: { prefix: "/", headers: [ { name: ":method", exact_match: "OPTIONS", invert_match: true } ] }
              #  route: { cluster: grpc, timeout: { seconds: 60 } }
              #- match: { prefix: "/", headers: [ { name: ":method", exact_match: "OPTIONS" } ] }
              #  direct_response: { status: 200 }
          access_log:
          - name: envoy.file_access_log
            config:
              path: "/dev/stdout"
          http_filters:
          - name: envoy.filters.http.jwt_authn
            config:
              providers:
                sessions:
                  issuer: http://dex/dex
                  audiences:
                    - example-app
                  remote_jwks:
                    http_uri:
                      uri: http://dex/dex/keys
                      cluster: local_jwks_cluster
                    cache_duration:
                      seconds: 300
                  forward: true
                  forward_payload_header: x-authorization-payload
              rules:
                - match:
                    prefix: /
                  requires:
                    provider_name: sessions
          - name: envoy.grpc_json_transcoder
            config:
              proto_descriptor: "/proto/proto.bin"
              services: ["helloworld.Greeter"]
              match_incoming_request_route: true
              print_options:
                always_print_primitive_fields: true
                always_print_enums_as_ints: true
          - name: envoy.cors
          - name: envoy.router

  clusters:
  - name: local_jwks_cluster
    connect_timeout: 1s
    dns_lookup_family: V4_ONLY
    type: STRICT_DNS
    load_assignment:
      cluster_name: local_jwks_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: dex
                port_value: 5556
  - name: grpc
    connect_timeout: 1.25s
    type: logical_dns
    lb_policy: round_robin
    dns_lookup_family: V4_ONLY
    http2_protocol_options: {}
    load_assignment:
      cluster_name: grpc
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: host.docker.internal
                port_value: 10000

[lizan]

@mattbailey You should have the cors filter before jwt_authn and grpc_json_transcoder. Does that work? Your access log indicating UR, means the request sent to upstream, and upstream reset the request.

[mattbailey]

Ok, simplified things a bit, but essentially the same problem. And, put envoy.cors at the top, as per @lizan

Config 1: direct_response for OPTIONS method matches

admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address: { address: 0.0.0.0, port_value: 9901 }

static_resources:
  listeners:
  - name: listener1
    address:
      socket_address: { address: 0.0.0.0, port_value: 11000 }
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
          stat_prefix: grpc_json
          codec_type: AUTO
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              cors:
                allow_origin: ["*"]
                allow_methods: GET, PUT, DELETE, POST, PATCH, OPTIONS
                allow_headers: authorization,keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web
                expose_headers: grpc-status,grpc-message,x-envoy-upstream-service-time
              routes:
              - match: { prefix: "/", headers: [ { name: ":method", exact_match: "OPTIONS", invert_match: true } ] }
                route: { cluster: grpc, timeout: { seconds: 60 } }
              - match: { prefix: "/", headers: [ { name: ":method", exact_match: "OPTIONS" } ] }
                direct_response: { status: 200 }
          access_log:
          - name: envoy.file_access_log
            config:
              path: "/dev/stdout"
          http_filters:
          - name: envoy.cors
          - name: envoy.filters.http.jwt_authn
            config:
              providers:
                sessions:
                  issuer: http://localhost:5556/dex
                  audiences:
                    - example-app
                  remote_jwks:
                    http_uri:
                      uri: http://localhost:5556/dex/keys
                      cluster: local_jwks_cluster
                    cache_duration:
                      seconds: 300
                  forward: true
                  forward_payload_header: x-authorization-payload
              rules:
                - match: { prefix: "/", headers: [ { name: ":method", exact_match: "OPTIONS", invert_match: true } ] }
                  requires:
                    provider_name: sessions
          - name: envoy.grpc_json_transcoder
            config:
              proto_descriptor: "/proto/proto.bin"
              services:
                - buffout.imageclean.v1.ImagecleanAPI
              match_incoming_request_route: true
              print_options:
                always_print_primitive_fields: true
                always_print_enums_as_ints: true
          - name: envoy.router

  clusters:
  - name: local_jwks_cluster
    connect_timeout: 1s
    dns_lookup_family: V4_ONLY
    type: STRICT_DNS
    load_assignment:
      cluster_name: local_jwks_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: host.docker.internal
                port_value: 5556
  - name: grpc
    connect_timeout: 1.25s
    type: logical_dns
    lb_policy: round_robin
    dns_lookup_family: V4_ONLY
    http2_protocol_options: {}
    load_assignment:
      cluster_name: grpc
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: host.docker.internal
                port_value: 10000

Config 1: OPTIONS request with Origin header

• Fails to emit CORS preflight headers
• Doesn't hit upstream grpc
  

❯ curl -i -XOPTIONS -H'Origin: https://localhost:123' localhost:11000/imageclean/v1/hello
HTTP/1.1 200 OK
location: http://localhost:11000/imageclean/v1/hello
date: Wed, 17 Apr 2019 20:01:24 GMT
server: envoy
content-length: 0

Config 1: GET request with Origin header

• CORS headers present
• Transcoding works, grpc server handles properly
  

❯ curl -i -H'Origin: http://localhost:123' -H 'Authorization: Bearer <redacted>' localhost:11000/imageclean/v1/hello
HTTP/1.1 200 OK
content-type: application/json
x-envoy-upstream-service-time: 3
grpc-status: 0
grpc-message:
content-length: 31
access-control-allow-origin: http://localhost:123
access-control-expose-headers: grpc-status,grpc-message,x-envoy-upstream-service-time
date: Wed, 17 Apr 2019 20:02:55 GMT
server: envoy

{"text":"Hello Kilgore Trout!"}

Config 2: No routing rules to change on OPTIONS method

admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address: { address: 0.0.0.0, port_value: 9901 }

static_resources:
  listeners:
  - name: listener1
    address:
      socket_address: { address: 0.0.0.0, port_value: 11000 }
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
          stat_prefix: grpc_json
          codec_type: AUTO
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              cors:
                allow_origin: ["*"]
                allow_methods: GET, PUT, DELETE, POST, PATCH, OPTIONS
                allow_headers: authorization,keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web
                expose_headers: grpc-status,grpc-message,x-envoy-upstream-service-time
              routes:
              - match: { prefix: "/" }
                route: { cluster: grpc, timeout: { seconds: 60 } }
          access_log:
          - name: envoy.file_access_log
            config:
              path: "/dev/stdout"
          http_filters:
          - name: envoy.cors
          - name: envoy.filters.http.jwt_authn
            config:
              providers:
                sessions:
                  issuer: http://localhost:5556/dex
                  audiences:
                    - example-app
                  remote_jwks:
                    http_uri:
                      uri: http://localhost:5556/dex/keys
                      cluster: local_jwks_cluster
                    cache_duration:
                      seconds: 300
                  forward: true
                  forward_payload_header: x-authorization-payload
              rules:
                - match: { prefix: "/", headers: [ { name: ":method", exact_match: "OPTIONS", invert_match: true } ] }
                  requires:
                    provider_name: sessions
          - name: envoy.grpc_json_transcoder
            config:
              proto_descriptor: "/proto/proto.bin"
              services:
                - buffout.imageclean.v1.ImagecleanAPI
              match_incoming_request_route: true
              print_options:
                always_print_primitive_fields: true
                always_print_enums_as_ints: true
          - name: envoy.router

  clusters:
  - name: local_jwks_cluster
    connect_timeout: 1s
    dns_lookup_family: V4_ONLY
    type: STRICT_DNS
    load_assignment:
      cluster_name: local_jwks_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: host.docker.internal
                port_value: 5556
  - name: grpc
    connect_timeout: 1.25s
    type: logical_dns
    lb_policy: round_robin
    dns_lookup_family: V4_ONLY
    http2_protocol_options: {}
    load_assignment:
      cluster_name: grpc
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: host.docker.internal
                port_value: 10000

Config 2: OPTIONS request with Origin header

• Emits CORS headsers
• Passes on to grpc service/transcoder, which fails.
  

❯ curl -i -XOPTIONS -H'Origin: https://localhost:123' localhost:11000/imageclean/v1/hello
HTTP/1.1 503 Service Unavailable
content-length: 85
content-type: text/plain
access-control-allow-origin: https://localhost:123
access-control-expose-headers: grpc-status,grpc-message,x-envoy-upstream-service-time
date: Wed, 17 Apr 2019 20:12:43 GMT
server: envoy

upstream connect error or disconnect/reset before headers. reset reason: remote reset

[2019-04-17T20:12:43.721Z] "OPTIONS /imageclean/v1/hello HTTP/1.1" 503 UR 0 85 4 - "-" "curl/7.54.0" "e51d4cc0-d6bb-4eaa-b25b-df12046eae7c" "localhost:11000" "192.168.65.2:10000"

Option 2: GET with Origin, works fine, same success.

[lizan]

Hmm, do you expect your upstream gRPC server handle CORS preflight request or Envoy to handle them? Does your protobuf descriptor have method matches OPTIONS request?

[mattbailey]

@lizan well, I guess that's what I'm getting at: google's http.proto doesn't handle OPTIONS at all: https://github.com/googleapis/googleapis/blob/master/google/api/http.proto

So, I can't even annotate handlers with OPTIONS methods. Notionally, I'd expect the gRPC-JSON transcoder to respond directly to OPTIONS requests with the appropriate Allow header, which it should be able to get from the descriptor, right?

For what it's worth, grpc-gateway has an example handling OPTIONS here: https://github.com/grpc-ecosystem/grpc-gateway/blob/master/examples/gateway/handlers.go#L36 but, of course, I can't manipulate handlers in the envoy transcoder, so I'm trying to find a work-around.

[lizan]

So, I can't even annotate handlers with OPTIONS methods. Notionally, I'd expect the gRPC-JSON transcoder to respond directly to OPTIONS requests with the appropriate Allow header, which it should be able to get from the descriptor, right?

If you want Envoy to handle CORS preflight requests, then the CORS filter is enough, so that's why you should put it on top, you don't even need a route to match OPTIONS request for CORS preflight request.

The OPTIONS request in your config 1 is not a CORS preflight request as it misses Access-Control-Request-Method and Access-Control-Request-Headers header.

[mattbailey]

(after talking with @lizan on envoy slack)

Looks like my problem was just that I wasn't simulating CORS preflight enough with curl. Config 2 works if you add additional headers to the request:

curl -i -XOPTIONS \
  -H'Access-Control-Request-Method: GET' \
  -H'Access-Control-Request-Headers: authorization, content-type' \
  -H'Origin: https://localhost:123' \
localhost:11000/imageclean/v1/hello

HTTP/1.1 200 OK
access-control-allow-origin: https://localhost:123
access-control-allow-methods: GET, PUT, DELETE, POST, PATCH, OPTIONS
access-control-allow-headers: authorization,keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web
access-control-expose-headers: grpc-status,grpc-message,x-envoy-upstream-service-time
date: Wed, 17 Apr 2019 21:56:12 GMT
server: envoy
content-length: 0

[MilgoTest]

May you please post your working config? I'm having the same problem but can't figure out how to get a 200 response for an OPTIONS request.

[mattbailey]

@MilgoTest Sorry! Did the worst thing ever: "I fixed it, but not gonna post how"

But also remember: my _real_ problem was that my curl testing was _inaccurate_ for simulating CORS properly.

static_resources:
  listeners:
  - name: listener1
    address:
      socket_address: { address: 0.0.0.0, port_value: 11000 }
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
          stat_prefix: grpc_json
          codec_type: AUTO
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              cors:
                allow_origin_regex:
                - "https?:\\/\\/.*\\.MYDOMAIN\\.COM"
                - "https?:\\/\\/MYDOMAIN\\.COM"
                - "https?:\\/\\/localhost:\\d+"
                allow_methods: GET, PUT, DELETE, POST, PATCH, OPTIONS
                allow_headers: authorization,keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web
                expose_headers: grpc-status,grpc-message,x-envoy-upstream-service-time
              routes:
              - match: { prefix: "/" }
                route: { cluster: grpc, timeout: { seconds: 60 } }
          access_log:
          - name: envoy.file_access_log
            config:
              path: "/dev/stdout"
          http_filters:
          - name: envoy.cors
          - name: envoy.grpc_web
          - name: envoy.grpc_json_transcoder
            config:
              proto_descriptor: "/proto/proto.bin"
              services:
                - SomeService.Method
              match_incoming_request_route: true
              print_options:
                always_print_primitive_fields: true
                always_print_enums_as_ints: true
          - name: envoy.router
  clusters:
  - name: grpc
    connect_timeout: 1.25s
    type: strict_dns
    lb_policy: round_robin
    dns_lookup_family: V4_ONLY
    http2_protocol_options: {}
    load_assignment:
      cluster_name: grpc
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 10000