Envoy: JWT Authentication fails for remote_jwks but works with local_jwks

[dcseifert]

Title: Filter envoy.filters.http.jwt_authn gets response code 400 (BadRequest) for remote_jwks uri

Description:
After configuring Envoy with external JWT Authentication a request containing a valid token fails with following logs (envoy; log-level: debug):

[2020-02-10 09:35:38.144][11][debug][conn_handler] [source/server/connection_handler_impl.cc:353] [C0] new connection
[2020-02-10 09:35:38.144][11][debug][conn_handler] [source/server/connection_handler_impl.cc:353] [C1] new connection
[2020-02-10 09:35:38.176][11][debug][connection] [source/common/network/connection_impl.cc:531] [C0] remote close
[2020-02-10 09:35:38.176][11][debug][connection] [source/common/network/connection_impl.cc:192] [C0] closing socket: 0
[2020-02-10 09:35:38.176][11][debug][conn_handler] [source/server/connection_handler_impl.cc:86] [C0] adding to cleanup list
[2020-02-10 09:35:38.177][11][debug][http] [source/common/http/conn_manager_impl.cc:263] [C1] new stream
[2020-02-10 09:35:38.178][11][debug][http] [source/common/http/conn_manager_impl.cc:731] [C1][S17748013466714094481] request headers complete (end_stream=true):
':authority', 'localhost:8080'
':path', '/api/apps'
':method', 'GET'
'authorization', 'Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IlF6bEVOVUpGTUVSQ01rTXpSakJHT1RReU1FRTJNa1JFTVRkR1JqQkRNVVl6UWtReE5FUTNOdyJ9.eyJpc3MiOiJodHRwczovL2Rldi1rZWxvbi5ldS5hdXRoMC5jb20vIiwic3ViIjoiRzVNMUg2a0RLcDY4UjBxMXcweW9wV0VhQkZHSDNmd01AY2xpZW50cyIsImF1ZCI6Imh0dHBzOi8vZGV2LWtlbG9uLmV1LmF1dGgwLmNvbS9hcGkvdjIvIiwiaWF0IjoxNTgxMzI0ODU4LCJleHAiOjE1ODE0MTEyNTgsImF6cCI6Ikc1TTFINmtES3A2OFIwcTF3MHlvcFdFYUJGR0gzZndNIiwic2NvcGUiOiJyZWFkOmNsaWVudF9ncmFudHMgY3JlYXRlOmNsaWVudF9ncmFudHMgZGVsZXRlOmNsaWVudF9ncmFudHMgdXBkYXRlOmNsaWVudF9ncmFudHMgcmVhZDp1c2VycyB1cGRhdGU6dXNlcnMgZGVsZXRlOnVzZXJzIGNyZWF0ZTp1c2VycyByZWFkOnVzZXJzX2FwcF9tZXRhZGF0YSB1cGRhdGU6dXNlcnNfYXBwX21ldGFkYXRhIGRlbGV0ZTp1c2Vyc19hcHBfbWV0YWRhdGEgY3JlYXRlOnVzZXJzX2FwcF9tZXRhZGF0YSBjcmVhdGU6dXNlcl90aWNrZXRzIHJlYWQ6Y2xpZW50cyB1cGRhdGU6Y2xpZW50cyBkZWxldGU6Y2xpZW50cyBjcmVhdGU6Y2xpZW50cyByZWFkOmNsaWVudF9rZXlzIHVwZGF0ZTpjbGllbnRfa2V5cyBkZWxldGU6Y2xpZW50X2tleXMgY3JlYXRlOmNsaWVudF9rZXlzIHJlYWQ6Y29ubmVjdGlvbnMgdXBkYXRlOmNvbm5lY3Rpb25zIGRlbGV0ZTpjb25uZWN0aW9ucyBjcmVhdGU6Y29ubmVjdGlvbnMgcmVhZDpyZXNvdXJjZV9zZXJ2ZXJzIHVwZGF0ZTpyZXNvdXJjZV9zZXJ2ZXJzIGRlbGV0ZTpyZXNvdXJjZV9zZXJ2ZXJzIGNyZWF0ZTpyZXNvdXJjZV9zZXJ2ZXJzIHJlYWQ6ZGV2aWNlX2NyZWRlbnRpYWxzIHVwZGF0ZTpkZXZpY2VfY3JlZGVudGlhbHMgZGVsZXRlOmRldmljZV9jcmVkZW50aWFscyBjcmVhdGU6ZGV2aWNlX2NyZWRlbnRpYWxzIHJlYWQ6cnVsZXMgdXBkYXRlOnJ1bGVzIGRlbGV0ZTpydWxlcyBjcmVhdGU6cnVsZXMgcmVhZDpydWxlc19jb25maWdzIHVwZGF0ZTpydWxlc19jb25maWdzIGRlbGV0ZTpydWxlc19jb25maWdzIHJlYWQ6aG9va3MgdXBkYXRlOmhvb2tzIGRlbGV0ZTpob29rcyBjcmVhdGU6aG9va3MgcmVhZDplbWFpbF9wcm92aWRlciB1cGRhdGU6ZW1haWxfcHJvdmlkZXIgZGVsZXRlOmVtYWlsX3Byb3ZpZGVyIGNyZWF0ZTplbWFpbF9wcm92aWRlciBibGFja2xpc3Q6dG9rZW5zIHJlYWQ6c3RhdHMgcmVhZDp0ZW5hbnRfc2V0dGluZ3MgdXBkYXRlOnRlbmFudF9zZXR0aW5ncyByZWFkOmxvZ3MgcmVhZDpzaGllbGRzIGNyZWF0ZTpzaGllbGRzIGRlbGV0ZTpzaGllbGRzIHJlYWQ6YW5vbWFseV9ibG9ja3MgZGVsZXRlOmFub21hbHlfYmxvY2tzIHVwZGF0ZTp0cmlnZ2VycyByZWFkOnRyaWdnZXJzIHJlYWQ6Z3JhbnRzIGRlbGV0ZTpncmFudHMgcmVhZDpndWFyZGlhbl9mYWN0b3JzIHVwZGF0ZTpndWFyZGlhbl9mYWN0b3JzIHJlYWQ6Z3VhcmRpYW5fZW5yb2xsbWVudHMgZGVsZXRlOmd1YXJkaWFuX2Vucm9sbG1lbnRzIGNyZWF0ZTpndWFyZGlhbl9lbnJvbGxtZW50X3RpY2tldHMgcmVhZDp1c2VyX2lkcF90b2tlbnMgY3JlYXRlOnBhc3N3b3Jkc19jaGVja2luZ19qb2IgZGVsZXRlOnBhc3N3b3Jkc19jaGVja2luZ19qb2IgcmVhZDpjdXN0b21fZG9tYWlucyBkZWxldGU6Y3VzdG9tX2RvbWFpbnMgY3JlYXRlOmN1c3RvbV9kb21haW5zIHJlYWQ6ZW1haWxfdGVtcGxhdGVzIGNyZWF0ZTplbWFpbF90ZW1wbGF0ZXMgdXBkYXRlOmVtYWlsX3RlbXBsYXRlcyByZWFkOm1mYV9wb2xpY2llcyB1cGRhdGU6bWZhX3BvbGljaWVzIHJlYWQ6cm9sZXMgY3JlYXRlOnJvbGVzIGRlbGV0ZTpyb2xlcyB1cGRhdGU6cm9sZXMgcmVhZDpwcm9tcHRzIHVwZGF0ZTpwcm9tcHRzIHJlYWQ6YnJhbmRpbmcgdXBkYXRlOmJyYW5kaW5nIHJlYWQ6bG9nX3N0cmVhbXMgY3JlYXRlOmxvZ19zdHJlYW1zIGRlbGV0ZTpsb2dfc3RyZWFtcyB1cGRhdGU6bG9nX3N0cmVhbXMiLCJndHkiOiJjbGllbnQtY3JlZGVudGlhbHMifQ.eLtOFZiBs6mBzB1e1dwEiDYbTo82o03rO_k1Z5Rkw0qlBawOsHtuAhFuJTOMqtexMOUOPGdsHXP39bm2uDG_omR6GbeePazA_zIhMu8aifEOk89s3tJezLZT3DDr1vmKJe0RbJjgH4WKoX9xLJgBQF2B9Dsrs2ViqQ9RDWrX_uSxpFOqKnOfMdQdaFpOYRSLVFK-VAoFDMQHWMnyEho7hyg__3Ph_2VdAiU4PjRm5978IRvwymvPNH4uC3xjS_WZIQZCZtvv-Qbj1Df40cj618Ax7UmYVt_vSbF0enKYPjkGFhbVeu277DXL9dHryrrQspN0MVFcpdYT40-FR2kLFA'
'user-agent', 'PostmanRuntime/7.22.0'
'accept', '*/*'
'cache-control', 'no-cache'
'postman-token', '7058e0af-40cb-4191-adf9-5f5cca54a124'
'accept-encoding', 'gzip, deflate, br'
'connection', 'keep-alive'

[2020-02-10 09:35:38.178][11][debug][http] [source/common/http/conn_manager_impl.cc:1276] [C1][S17748013466714094481] request end stream
[2020-02-10 09:35:38.178][11][debug][jwt] [source/extensions/filters/http/jwt_authn/filter.cc:124] Called Filter : setDecoderFilterCallbacks
[2020-02-10 09:35:38.178][11][debug][jwt] [source/extensions/filters/http/jwt_authn/filter.cc:46] Called Filter : decodeHeaders
[2020-02-10 09:35:38.178][11][debug][jwt] [source/extensions/filters/http/jwt_authn/matcher.cc:71] Prefix requirement '/api/apps' matched.
[2020-02-10 09:35:38.178][11][debug][jwt] [source/extensions/filters/http/jwt_authn/extractor.cc:188] extract authorizationBearer 
[2020-02-10 09:35:38.178][11][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:124] provider1: JWT authentication starts (allow_failed=false), tokens size=1
[2020-02-10 09:35:38.178][11][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:135] provider1: startVerify: tokens size 1
[2020-02-10 09:35:38.178][11][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:146] provider1: Verifying JWT token of issuer https://dev-kelon.eu.auth0.com/
[2020-02-10 09:35:38.178][11][debug][filter] [source/extensions/filters/http/common/jwks_fetcher.cc:55] fetch pubkey from [uri = https://dev-kelon.eu.auth0.com/.well-known/jwks.json]: start
[2020-02-10 09:35:38.178][11][debug][router] [source/common/router/router.cc:474] [C0][S11706397199394743165] cluster 'auth0' match for URL '/.well-known/jwks.json'
[2020-02-10 09:35:38.178][11][debug][router] [source/common/router/router.cc:614] [C0][S11706397199394743165] router decoding headers:
':path', '/.well-known/jwks.json'
':authority', 'dev-kelon.eu.auth0.com'
':method', 'GET'
':scheme', 'http'
'x-envoy-internal', 'true'
'x-forwarded-for', '10.244.0.120'
'x-envoy-expected-rq-timeout-ms', '5000'

[2020-02-10 09:35:38.178][11][debug][pool] [source/common/http/http1/conn_pool.cc:95] creating a new connection
[2020-02-10 09:35:38.178][11][debug][client] [source/common/http/codec_client.cc:34] [C2] connecting
[2020-02-10 09:35:38.178][11][debug][connection] [source/common/network/connection_impl.cc:691] [C2] connecting to 52.59.14.79:443
[2020-02-10 09:35:38.179][11][debug][connection] [source/common/network/connection_impl.cc:700] [C2] connection in progress
[2020-02-10 09:35:38.179][11][debug][pool] [source/common/http/conn_pool_base.cc:55] queueing request due to no available connections
[2020-02-10 09:35:38.179][11][debug][jwt] [source/extensions/filters/http/jwt_authn/filter.cc:73] Called Filter : decodeHeaders Stop
[2020-02-10 09:35:38.182][11][debug][connection] [source/common/network/connection_impl.cc:563] [C2] connected
[2020-02-10 09:35:38.183][11][debug][client] [source/common/http/codec_client.cc:72] [C2] connected
[2020-02-10 09:35:38.183][11][debug][pool] [source/common/http/http1/conn_pool.cc:244] [C2] attaching to next request
[2020-02-10 09:35:38.183][11][debug][router] [source/common/router/router.cc:1711] [C0][S11706397199394743165] pool ready
[2020-02-10 09:35:38.184][11][debug][router] [source/common/router/router.cc:1115] [C0][S11706397199394743165] upstream headers complete: end_stream=false
[2020-02-10 09:35:38.185][11][debug][http] [source/common/http/async_client_impl.cc:95] async http request response headers (end_stream=false):
':status', '400'
'server', 'awselb/2.0'
'date', 'Mon, 10 Feb 2020 09:35:38 GMT'
'content-type', 'text/html'
'content-length', '236'
'connection', 'close'
'x-envoy-upstream-service-time', '6'

[2020-02-10 09:35:38.185][11][debug][client] [source/common/http/codec_client.cc:104] [C2] response complete
[2020-02-10 09:35:38.185][11][debug][filter] [source/extensions/filters/http/common/jwks_fetcher.cc:90] onSuccess: fetch pubkey [uri = https://dev-kelon.eu.auth0.com/.well-known/jwks.json]: response status code 400
[2020-02-10 09:35:38.186][11][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:264] provider1: JWT token verification completed with: Jwks remote fetch is failed
[2020-02-10 09:35:38.186][11][debug][jwt] [source/extensions/filters/http/jwt_authn/filter.cc:84] Called Filter : check complete Jwks remote fetch is failed
[2020-02-10 09:35:38.186][11][debug][http] [source/common/http/conn_manager_impl.cc:1417] [C1][S17748013466714094481] Sending local reply with details jwt_authn_access_denied
[2020-02-10 09:35:38.186][11][debug][http] [source/common/http/conn_manager_impl.cc:1615] [C1][S17748013466714094481] encoding headers via codec (end_stream=false):
':status', '401'
'content-length', '27'
'content-type', 'text/plain'
'date', 'Mon, 10 Feb 2020 09:35:38 GMT'
'server', 'envoy'

[2020-02-10 09:35:38.186][11][debug][jwt] [source/extensions/filters/http/jwt_authn/filter.cc:39] Called Filter : onDestroy
[2020-02-10 09:35:38.186][11][debug][pool] [source/common/http/http1/conn_pool.cc:201] [C2] response complete
[2020-02-10 09:35:38.186][11][debug][pool] [source/common/http/http1/conn_pool.cc:206] [C2] saw upstream close connection
[2020-02-10 09:35:38.187][11][debug][connection] [source/common/network/connection_impl.cc:101] [C2] closing data_to_write=0 type=1
[2020-02-10 09:35:38.187][11][debug][connection] [source/common/network/connection_impl.cc:192] [C2] closing socket: 1
[2020-02-10 09:35:38.187][11][debug][client] [source/common/http/codec_client.cc:91] [C2] disconnect. resetting 0 pending requests
[2020-02-10 09:35:38.187][11][debug][pool] [source/common/http/http1/conn_pool.cc:136] [C2] client disconnected, failure reason: 
[2020-02-10 09:35:38.187][11][debug][connection] [source/common/network/connection_impl.cc:531] [C2] remote close
[2020-02-10 09:35:38.187][11][debug][grpc] [source/common/grpc/google_async_client_impl.cc:94] Client teardown, resetting streams
[2020-02-10 09:35:42.142][1][debug][main] [source/server/server.cc:174] flushing stats

I think the important line is:
onSuccess: fetch pubkey [uri = https://dev-kelon.eu.auth0.com/.well-known/jwks.json]: response status code 400

Fetching the jwks.json from https://dev-kelon.eu.auth0.com/.well-known/jwks.json manually works perfekt and if I configure envoy to use a local_jwks with the pasted content from the request:

                      - name: envoy.filters.http.jwt_authn
                        config:
                          providers:
                            provider1:
                              issuer: "https://dev-kelon.eu.auth0.com/"
                              local_jwks:
                                inline_string: '{"keys":[{"alg":"RS256","kty":"RSA","use":"sig","n":"v-ArEf2T0bg9M1002MPAf4mfUtG4_34Mc3dtPIZpzL81U-WEGDTtmrqp3iHbnLd3zfynwDTK8pygjLz8xRFsaYr-TYkri4dzUKz5c45P0tmv88I-qGOdRIhxL8It4XDdQV_fsGrskMLl9j9DLpU5Yfg9nm6pyIkqcDQglILubBXNkzk_JJpucoaF7GwRGZ79f9U1B2jsUIWqmXmtGOoQLyZWF3RcBibdFF6jhsHVKtxvZalhugd-wzZkLLlfNff-7f4NEumWCZn4dVh4vGAuzEDhstcCqJtRWHt6P-KQFVX-OAebwqvxdCa-6Oqsd39SrO28iTykmT-zawiCB3kDhw","e":"AQAB","kid":"QzlENUJFMERCMkMzRjBGOTQyMEE2MkREMTdGRjBDMUYzQkQxNEQ3Nw","x5t":"QzlENUJFMERCMkMzRjBGOTQyMEE2MkREMTdGRjBDMUYzQkQxNEQ3Nw","x5c":["MIIDBzCCAe+gAwIBAgIJe8NsUUuwcUTyMA0GCSqGSIb3DQEBCwUAMCExHzAdBgNVBAMTFmRldi1rZWxvbi5ldS5hdXRoMC5jb20wHhcNMjAwMjA3MTQyMDE2WhcNMzMxMDE2MTQyMDE2WjAhMR8wHQYDVQQDExZkZXYta2Vsb24uZXUuYXV0aDAuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv+ArEf2T0bg9M1002MPAf4mfUtG4/34Mc3dtPIZpzL81U+WEGDTtmrqp3iHbnLd3zfynwDTK8pygjLz8xRFsaYr+TYkri4dzUKz5c45P0tmv88I+qGOdRIhxL8It4XDdQV/fsGrskMLl9j9DLpU5Yfg9nm6pyIkqcDQglILubBXNkzk/JJpucoaF7GwRGZ79f9U1B2jsUIWqmXmtGOoQLyZWF3RcBibdFF6jhsHVKtxvZalhugd+wzZkLLlfNff+7f4NEumWCZn4dVh4vGAuzEDhstcCqJtRWHt6P+KQFVX+OAebwqvxdCa+6Oqsd39SrO28iTykmT+zawiCB3kDhwIDAQABo0IwQDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTUXNEacZNN51feJ6DPeoesHVGGoDAOBgNVHQ8BAf8EBAMCAoQwDQYJKoZIhvcNAQELBQADggEBAKnDhtzPUC4+yf7TXWKPEbnDawmwnzlMhha8964zNgS9Cb9RmmTdxPYJCMwA//NHDGQcPs6W31Noy52QFmEBuwt1hsJGQRF9/sAv0xFKA6++2etEbVFd0l49jiH5X3yXdA8XV95GV3rDUYXxKJYMZTaJccN28zs9uqdkPIqt0/Bx/HvwPZO6kHwT2Vhn+JuN8NBSCYtV5kbJBtx3shQzvs5OBoiHDvlG/UpMyb9yHZBQD6mlCIzYAf6EpI5TmY9GC44vMpIChtIqQDGpij9nY9NCcztuRWYchv7OVy9L1ri+GoV4bBRR1d1ODSxmHOnXDr8zmfcIG5b2KrxMhm2JCk4="]}]}'

The entire filter-chain also works perfectly.

Repro steps:
I am running envoy (v1.13.0) in k8s v1.16.6 with this deployment.

Most important my envy.yaml:

admin:
  access_log_path: "/dev/null"
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8001
static_resources:
  listeners:
    - address:
        socket_address:
          address: 0.0.0.0
          port_value: 8000
      filter_chains:
        - filters:
            - name: envoy.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
                codec_type: auto
                stat_prefix: ingress_http
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: backend
                      domains:
                        - "*"
                      cors:
                        allow_origin_string_match:
                          - safe_regex:
                              google_re2: {}
                              regex: \*
                        allow_methods: "*"
                        allow_headers: "*"
                        filter_enabled:
                          default_value:
                            numerator: 100
                            denominator: HUNDRED
                          runtime_key: cors.www.enabled
                        shadow_enabled:
                          default_value:
                            numerator: 0
                            denominator: HUNDRED
                          runtime_key: cors.www.shadow_enabled
                      routes:
                        - match:
                            prefix: "/"
                          route:
                            cluster: service
                http_filters:
                  - name: envoy.cors
                    typed_config: {}
                  - name: envoy.filters.http.jwt_authn
                    config:
                      providers:
                        provider1:
                          issuer: "https://dev-kelon.eu.auth0.com/"

                          # Works
                          local_jwks:
                            inline_string: '{"keys":[{"alg":"RS256","kty":"RSA","use":"sig","n":"v-ArEf2T0bg9M1002MPAf4mfUtG4_34Mc3dtPIZpzL81U-WEGDTtmrqp3iHbnLd3zfynwDTK8pygjLz8xRFsaYr-TYkri4dzUKz5c45P0tmv88I-qGOdRIhxL8It4XDdQV_fsGrskMLl9j9DLpU5Yfg9nm6pyIkqcDQglILubBXNkzk_JJpucoaF7GwRGZ79f9U1B2jsUIWqmXmtGOoQLyZWF3RcBibdFF6jhsHVKtxvZalhugd-wzZkLLlfNff-7f4NEumWCZn4dVh4vGAuzEDhstcCqJtRWHt6P-KQFVX-OAebwqvxdCa-6Oqsd39SrO28iTykmT-zawiCB3kDhw","e":"AQAB","kid":"QzlENUJFMERCMkMzRjBGOTQyMEE2MkREMTdGRjBDMUYzQkQxNEQ3Nw","x5t":"QzlENUJFMERCMkMzRjBGOTQyMEE2MkREMTdGRjBDMUYzQkQxNEQ3Nw","x5c":["MIIDBzCCAe+gAwIBAgIJe8NsUUuwcUTyMA0GCSqGSIb3DQEBCwUAMCExHzAdBgNVBAMTFmRldi1rZWxvbi5ldS5hdXRoMC5jb20wHhcNMjAwMjA3MTQyMDE2WhcNMzMxMDE2MTQyMDE2WjAhMR8wHQYDVQQDExZkZXYta2Vsb24uZXUuYXV0aDAuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv+ArEf2T0bg9M1002MPAf4mfUtG4/34Mc3dtPIZpzL81U+WEGDTtmrqp3iHbnLd3zfynwDTK8pygjLz8xRFsaYr+TYkri4dzUKz5c45P0tmv88I+qGOdRIhxL8It4XDdQV/fsGrskMLl9j9DLpU5Yfg9nm6pyIkqcDQglILubBXNkzk/JJpucoaF7GwRGZ79f9U1B2jsUIWqmXmtGOoQLyZWF3RcBibdFF6jhsHVKtxvZalhugd+wzZkLLlfNff+7f4NEumWCZn4dVh4vGAuzEDhstcCqJtRWHt6P+KQFVX+OAebwqvxdCa+6Oqsd39SrO28iTykmT+zawiCB3kDhwIDAQABo0IwQDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTUXNEacZNN51feJ6DPeoesHVGGoDAOBgNVHQ8BAf8EBAMCAoQwDQYJKoZIhvcNAQELBQADggEBAKnDhtzPUC4+yf7TXWKPEbnDawmwnzlMhha8964zNgS9Cb9RmmTdxPYJCMwA//NHDGQcPs6W31Noy52QFmEBuwt1hsJGQRF9/sAv0xFKA6++2etEbVFd0l49jiH5X3yXdA8XV95GV3rDUYXxKJYMZTaJccN28zs9uqdkPIqt0/Bx/HvwPZO6kHwT2Vhn+JuN8NBSCYtV5kbJBtx3shQzvs5OBoiHDvlG/UpMyb9yHZBQD6mlCIzYAf6EpI5TmY9GC44vMpIChtIqQDGpij9nY9NCcztuRWYchv7OVy9L1ri+GoV4bBRR1d1ODSxmHOnXDr8zmfcIG5b2KrxMhm2JCk4="]}]}'

                          # Fails with error 400
                          #remote_jwks:
                          #  http_uri:
                          #    uri: "https://dev-kelon.eu.auth0.com/.well-known/jwks.json"
                          #    cluster: auth0
                          #    timeout:
                          #      seconds: 5
                      rules:
                        - match:
                            prefix: /actuator/health
                        - match:
                            prefix: /api/login
                        - match:
                            prefix: /api/apps
                          requires:
                            provider_name: provider1
                  - name: envoy.ext_authz
                    config:
                      with_request_body:
                        max_request_bytes: 8192
                        allow_partial_message: true
                      failure_mode_allow: false
                      grpc_service:
                        google_grpc:
                          target_uri: "kelon.kelon.svc.cluster.local:9191"
                          stat_prefix: ext_authz
                        timeout: 0.5s
                  - name: envoy.router
                    typed_config: {}
  clusters:
    - name: service
      connect_timeout: 0.25s
      type: strict_dns
      lb_policy: round_robin
      load_assignment:
        cluster_name: service
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: appstore-backend-service.appstore.svc.cluster.local
                      port_value: 8080
    - name: auth0
      connect_timeout: 0.25s
      type: strict_dns
      lb_policy: round_robin
      load_assignment:
        cluster_name: auth0
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: dev-kelon.eu.auth0.com
                      port_value: 443

I think there might be something wrong with my cluster config of cluster auth0 here.
The example in the official docs has port 80 configured, which does not fit for https in my opinion and also does not work with Auth0 (301 Moved Permanently is returned by Auth0)

[qiwzhang]

Your jwks_uri is https, so your auth0 cluster need to specify TLS setting. Here is an example TLS settting in a cluster.

               "transportSocket": {
                    "name": "envoy.transport_sockets.tls",
                    "typedConfig": {
                        "@type": "type.googleapis.com/envoy.api.v2.auth.UpstreamTlsContext",
                        "commonTlsContext": {
                            "validationContext": {
                                "trustedCa": {
                                    "filename": "/etc/ssl/certs/ca-certificates.crt"
                                }
                            }
                        },
                        "sni": "dev-kelon.eu.auth0.com"
                    }

[dcseifert]

@qiwzhang This fixed my issue! Thank you so much!