Envoy: jwt_authn filter configuration issue

[vladimir22]

question

Title: * jwt_authn filter : Jwks remote fetch is failed *

Description:

Hello, I have problem with jwt_authn config. Unfortunatelly, jwt_authn documentation does not include full example envoy.yaml and I try to compose envoy.yaml based on lua example
I am trying to use different remote_jwks hosts (internal and www.googleapis.com/oauth2/v1/certs) and in every case I am getting response : "Jwks remote fetch is failed"

Request Example:

curl http://localhost:8000 --header "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImxlZ2FjeS10b2tlbi1rZXkiLCJ0eXAiOiJKV1QifQ.eyJqdGkiOiJmZWNlM2I2ZGZhYTc0YTdiYWVkMDMwMTZlYTEwN2U3YiIsInN1YiI6ImIwYmJkOTk2LWY4NzAtNGM3YS1iZWRjLWRlNzU1MGUzZmQ5OCIsInNjb3BlIjpbIm9wZW5pZCIsIm5vcnRoc3Rhci51c2VyIl0sImNsaWVudF9pZCI6ImFwaWdhdGV3YXljbGllbnRpZCIsImNpZCI6ImFwaWdhdGV3YXljbGllbnRpZCIsImF6cCI6ImFwaWdhdGV3YXljbGllbnRpZCIsImdyYW50X3R5cGUiOiJhdXRob3JpemF0aW9uX2NvZGUiLCJ1c2VyX2lkIjoiYjBiYmQ5OTYtZjg3MC00YzdhLWJlZGMtZGU3NTUwZTNmZDk4Iiwib3JpZ2luIjoibGRhcCIsInVzZXJfbmFtZSI6InJlYWRlcnVzZXIiLCJlbWFpbCI6InJlYWRlcnVzZXJAdXNlci5mcm9tLmxkYXAuY2YiLCJhdXRoX3RpbWUiOjE1NDA5MDk2NDQsInJldl9zaWciOiJkNzYxMGMwZiIsImlhdCI6MTU0MDkwOTY0NywiZXhwIjoxNTQwOTQ1NjQ3LCJpc3MiOiJodHRwOi8vaW5ncmVzcy5sb2NhbC91YWEvb2F1dGgvdG9rZW4iLCJ6aWQiOiJ1YWEiLCJhdWQiOlsiYXBpZ2F0ZXdheWNsaWVudGlkIiwibm9ydGhzdGFyIiwib3BlbmlkIl19.MO3zqxpQS6EvX6mUxYe6nR3Wyej3BschdXd0m5SIG_pRcPhrNoJVpENlIGVdSSqLugv5rAo_K5jkQWKOAeYdq_HLPCIYTyI4FmWZAc5TQzA54delRjSXprsrtGO7WdZwo9t4-zSR30UEx4zZPa-cJrzm8mO6tpEh4f1JGRaaWo34tuvD_Yi5X2c-SzW1E-r8C2NcB6vrnGhxdj3WQiJA_wIlEFASue50MnYBbtNZytgYJenkWarvJk2QPVK0WuCJZtFe47NKcVaz9ENTM4GtvQhpF1eYrOO216dSV0aiwv3ziU5-ND9XP7VtN4gBlHeQm2ZE_j6x2UYvsW_kE9xO_Q1212"

Jwks remote fetch is failed

Config:

envoy.yaml

static_resources:
listeners:

• name: main
  address:
  socket_address:
  address: 0.0.0.0
  port_value: 80
  filter_chains:
  
  
  
  • filters:
    
    
    
    
    
    • name: envoy.http_connection_manager
      
      
      config:
      
      
      stat_prefix: ingress_http
      
      
      codec_type: auto
      
      
      route_config:
      
      
      name: local_route
      
      
      virtual_hosts:
      
      
      
      
      
      
      
      • name: local_service
        
        
        
        domains:
        
        
        
        
        
        
        
        
        
        • "*"
          
          
          
          
          routes:
        
        
        
        
        
        • match:
          
          
          
          
          prefix: "/"
          
          
          
          
          route:
          
          
          
          
          cluster: web_service
          
          
          
          
          http_filters:
        
        
        
        
        
      
      
      
      
      • name: envoy.filters.http.jwt_authn
        
        
        
        config:
        
        
        
        providers:
        
        
        
        example_provider:
        
        
        
        issuer: http://ingress.local/uaa/oauth/token
        
        
        
        audiences:
        
        
        
        
        
        
        
        
        
        • apigatewayclientid
        
        
        
        
        
        • openid
          
          
          
          
          remote_jwks:
          
          
          
          
          http_uri:
          
          
          
          
          uri: https://www.googleapis.com/oauth2/v1/certs
          
          
          
          
          cluster: googleapis_cluster
          
          
          
          
          timeout:
          
          
          
          
          seconds: 5
          
          
          
          
          cache_duration:
          
          
          
          
          seconds: 10
          
          
          
          
          rules:
          
          
          
          
          
          
          
          
          
          
          
          • match:
            
            
            
            
            
            path: /
            
            
            
            
            
            requires:
            
            
            
            
            
            provider_name: example_provider
          
          
          
          
          
          
        
        
        
        
        
      
      
      
      
      • name: envoy.lua
        
        
        
        config:
        
        
        
        inline_code: |
        
        
        
        function envoy_on_request(request_handle)
        
        
        
        request_handle:headers():add("foo", "bar")
        
        
        
        end
        
        
        
        function envoy_on_response(response_handle)
        
        
        
        body_size = response_handle:body():length()
        
        
        
        response_handle:headers():add("response-body-size", tostring(body_size))
        
        
        
        end
      
      
      
      
      • name: envoy.router
        
        
        
        config: {}
      
      
      
      
    
    
    
  
  

clusters:

• name: web_service
  connect_timeout: 0.25s
  type: strict_dns # static
  lb_policy: round_robin
  hosts:
  
  
  
  • socket_address:
    
    address: web_service
    
    port_value: 80
  
  

• name: googleapis_cluster
  connect_timeout: 0.25s
  type: strict_dns
  lb_policy: round_robin
  hosts:
  
  
  
  • socket_address:
    
    address: googleapis.com
    
    port_value: 443
  
  

admin:
access_log_path: "/dev/null"
address:
socket_address:
address: 0.0.0.0
port_value: 8001

Logs:

proxy_1 |
proxy_1 | [2018-10-30 16:57:47.813][000014][debug][filter] [source/extensions/filters/http/jwt_authn/filter.cc:24] Called Filter : decodeHeaders
proxy_1 | [2018-10-30 16:57:47.813][000014][debug][filter] [source/extensions/filters/http/jwt_authn/matcher.cc:104] Path requirement '/' matched.
proxy_1 | [2018-10-30 16:57:47.813][000014][debug][filter] [source/extensions/filters/http/jwt_authn/authenticator.cc:102] Jwt authentication starts
proxy_1 | [2018-10-30 16:57:47.813][000014][debug][filter] [source/extensions/filters/http/common/jwks_fetcher.cc:39] fetch pubkey from [uri = https://www.googleapis.com/oauth2/v1/certs]: start
proxy_1 | [2018-10-30 16:57:47.813][000014][debug][router] [source/common/router/router.cc:261] [C0][S13404339381262252162] cluster 'googleapis_cluster' match for URL '/oauth2/v1/certs'
proxy_1 | [2018-10-30 16:57:47.813][000014][debug][router] [source/common/router/router.cc:317] [C0][S13404339381262252162] router decoding headers:
proxy_1 | ':path', '/oauth2/v1/certs'
proxy_1 | ':authority', 'www.googleapis.com'
proxy_1 | ':method', 'GET'
proxy_1 | ':scheme', 'http'
proxy_1 | 'x-envoy-internal', 'true'
proxy_1 | 'x-forwarded-for', '172.18.0.3'
proxy_1 | 'x-envoy-expected-rq-timeout-ms', '5000'
proxy_1 |
proxy_1 | [2018-10-30 16:57:47.813][000014][debug][pool] [source/common/http/http1/conn_pool.cc:78] creating a new connection
proxy_1 | [2018-10-30 16:57:47.813][000014][debug][client] [source/common/http/codec_client.cc:25] [C1] connecting
proxy_1 | [2018-10-30 16:57:47.813][000014][debug][connection] [source/common/network/connection_impl.cc:633] [C1] connecting to [2a00:1450:4013:c07::63]:443
proxy_1 | [2018-10-30 16:57:47.813][000014][debug][connection] [source/common/network/connection_impl.cc:646] [C1] immediate connection error: 99
proxy_1 | [2018-10-30 16:57:47.813][000014][debug][pool] [source/common/http/http1/conn_pool.cc:106] queueing request due to no available connections
proxy_1 | [2018-10-30 16:57:47.813][000014][debug][filter] [source/extensions/filters/http/jwt_authn/filter.cc:40] Called Filter : decodeHeaders Stop
proxy_1 | [2018-10-30 16:57:47.813][000014][debug][connection] [source/common/network/connection_impl.cc:449] [C1] raising immediate error
proxy_1 | [2018-10-30 16:57:47.814][000014][debug][connection] [source/common/network/connection_impl.cc:182] [C1] closing socket: 0
proxy_1 | [2018-10-30 16:57:47.814][000014][debug][client] [source/common/http/codec_client.cc:81] [C1] disconnect. resetting 0 pending requests
proxy_1 | [2018-10-30 16:57:47.814][000014][debug][pool] [source/common/http/http1/conn_pool.cc:122] [C1] client disconnected
proxy_1 | [2018-10-30 16:57:47.814][000014][debug][router] [source/common/router/router.cc:471] [C0][S13404339381262252162] upstream reset
proxy_1 | [2018-10-30 16:57:47.814][000014][debug][http] [source/common/http/async_client_impl.cc:94] async http request response headers (end_stream=false):
proxy_1 | ':status', '503'
proxy_1 | 'content-length', '57'
proxy_1 | 'content-type', 'text/plain'
proxy_1 |
proxy_1 | [2018-10-30 16:57:47.814][000014][debug][filter] [source/extensions/filters/http/common/jwks_fetcher.cc:71] onSuccess: fetch pubkey [uri = https://www.googleapis.com/oauth2/v1/certs]: response status code 503
proxy_1 | [2018-10-30 16:57:47.814][000014][debug][filter] [source/extensions/filters/http/jwt_authn/authenticator.cc:250] Jwt authentication completed with: Jwks remote fetch is failed
proxy_1 | [2018-10-30 16:57:47.814][000014][debug][filter] [source/extensions/filters/http/jwt_authn/filter.cc:50] Called Filter : check complete 24
proxy_1 | [2018-10-30 16:57:47.814][000014][debug][lua] [source/extensions/filters/http/lua/lua_filter.cc:306] yielding for full body
proxy_1 | [2018-10-30 16:57:47.814][000014][debug][lua] [source/extensions/filters/common/lua/lua.cc:38] coroutine yielded
proxy_1 | [2018-10-30 16:57:47.814][000014][debug][lua] [source/extensions/filters/http/lua/lua_filter.cc:52] resuming body due to end stream
proxy_1 | [2018-10-30 16:57:47.815][000014][debug][lua] [source/extensions/filters/common/lua/lua.cc:35] coroutine finished
proxy_1 | [2018-10-30 16:57:47.815][000014][debug][http] [source/common/http/conn_manager_impl.cc:1132] [C0][S9104162999134754958] encoding headers via codec (end_stream=false):
proxy_1 | ':status', '401'
proxy_1 | 'content-length', '27'
proxy_1 | 'content-type', 'text/plain'
proxy_1 | 'response-body-size', '27'
proxy_1 | 'date', 'Tue, 30 Oct 2018 16:57:47 GMT'
proxy_1 | 'server', 'envoy'
proxy_1 |
proxy_1 | [2018-10-30 16:57:47.815][000014][debug][filter] [source/extensions/filters/http/jwt_authn/filter.cc:17] Called Filter : onDestroy
proxy_1 | [2018-10-30 16:57:47.816][000014][debug][connection] [source/common/network/connection_impl.cc:500] [C0] remote close
proxy_1 | [2018-10-30 16:57:47.816][000014][debug][connection] [source/common/network/connection_impl.cc:182] [C0] closing socket: 0
proxy_1 | [2018-10-30 16:57:47.817][000014][debug][main] [source/server/connection_handler_impl.cc:63] [C0] adding to cleanup list
proxy_1 | [2018-10-30 16:57:50.234][000007][debug][main] [source/server/server.cc:139] flushing stats

[vladimir22]

inside proxy_1 container jwks host is accessible:

Logs:

curl -k https://www.googleapis.com/oauth2/v1/certs
{
"728f4016652079b9ed99861bb09bafc5a45baa86": "-----BEGIN CERTIFICATE-----\nMIIDJjCCAg6gAwIBAgIIXr5QVQhpB3kwDQYJKoZIhvcNAQEFBQAwNjE0MDIGA1UE\nAxMrZmVkZXJhdGVkLXNpZ25vbi5zeXN0ZW0uZ3NlcnZpY2VhY2NvdW50LmNvbTAe\nFw0xODEwMTYxNDQ5MTFaFw0xODExMDIwMzA0MTFaMDYxNDAyBgNVBAMTK2ZlZGVy\nYXRlZC1zaWdub24uc3lzdGVtLmdzZXJ2aWNl
YWNjb3VudC5jb20wggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCxtE2hhplffDYiixvq7N/KS364mH81FVDS\nH/3wam2Smrev7GB1qr2L6aZkRhm2s1SYio4QO+0gnfr4JWcNkjan94BEWSZ50cLX\nhBflId0VvaqvRrkO6oLmoujI4xdI0nJ9EcrxQuOjMvE7u38/QSx2Us85RCNxulOI\nij2n4v2KyQwIqlB0N9DgKvMcpCN9g3col16GwKGC59+575UFSSgNpFFQDnSHoOIV\n76WgL93PJyg0ZgkQ
m8AzkMLfO4evp19zmxoctSLi6mBHb+NyEDqTBI5UELTmpSBj\ngTdR1hNZyJOqRKjCIASsPHbWD6gHpDMivwxEzMprz36DYid76oTtAgMBAAGjODA2\nMAwGA1UdEwEB/wQCMAAwDgYDVR0PAQH/BAQDAgeAMBYGA1UdJQEB/wQMMAoGCCsG\nAQUFBwMCMA0GCSqGSIb3DQEBBQUAA4IBAQBCWm5aGCoO/iV/uNOU6uMEiEStudPD\nvqJOO/52HTvaKmyxpFWHNG7U1L+flXACdmJWwUr5db12TOEnLHRLYBWrNWNmr/
bm\njwuzeLK/OqqqYmTB+T2q6fj5mAUYBPR48Z21yHnggrtOx5X04MuUC2kKFnkpr5Yy\n7t3r6/0HyDMsWk3Uiudm9zraCgFibztjzyh0meGgnPKA5mTZQnrAwZNyCF6kKggi\n7LGzhFyRPFMJSkXCSvjUExdimCR+gQ3AqR5G46EfKyTEFcofkyNNutEkZ0pTe8pu\niFt9iQsqqXWdf342a+N5ueJSbXJ8i9tf5TWRxL97/RlIzly7wb2j1rt9\n-----END CERTIFICATE-----\n",
"8289d54280b76712de41cd2ef95972b123be9ac0": "-----BEGIN CERTIFICATE-----\nMIIDJjCCAg6gAwIBAgIIHiSmgNCe4EIwDQYJKoZIhvcNAQEFBQAwNjE0MDIGA1UE\nAxMrZmVkZXJhdGVkLXNpZ25vbi5zeXN0ZW0uZ3NlcnZpY2VhY2NvdW50LmNvbTAe\nFw0xODEwMjQxNDQ5MTFaFw0xODExMTAwMzA0MTFaMDYxNDAyBgNVBAMTK2ZlZGVy\nYXRlZC1zaWdub24uc3lzdGVtLmdzZXJ2aWNl
YWNjb3VudC5jb20wggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCPeue6aABtdQzCrAvfWU1Vi69YZuuJPu7L\nsKBN6rB80t/pD1MxCUOJ2sUVIuBJ7YGJ+MSzBLoHiqmxcyWsyhKjMjBse5KShBl1\njabm8URlGTIcHiYApm7NSLlcfktWWVu/ZMGnoqyMlZJJbkGfJ6f2kh5qYd04Ohf8\n8R0GGaqScNRFG66rcFvchWe50Y/wkJsdNnzplLZpLaGlgrd4Dx2+nXnkrWMowAb1\nYfz2OGR4VHrs6r2Y
P7IZkNo0I3yc5lHkjLD5ZAmjn6KMQByQdefVJ1gaNCCb/Z+8\nQGx/SAkSObHYOs15pxkvINQD90H0kjLlmt8VnQf2XQiRKCCBRJqRAgMBAAGjODA2\nMAwGA1UdEwEB/wQCMAAwDgYDVR0PAQH/BAQDAgeAMBYGA1UdJQEB/wQMMAoGCCsG\nAQUFBwMCMA0GCSqGSIb3DQEBBQUAA4IBAQAXw0JJPtxs1hanLTxSFI21ib4BDUuI\n+Aksnz1azijb6cKXVJAtDMQABpH2u6VTcnnGJ/nIQ4KeJHxMDL2w6wvViz9wwR
H3\nlru5D6Jks8IShBSeM80IOwrfxodEfju47B+G3Zxg/1qQ4QV0S3C+oMmcqQqHFK+l\nZ6glTHJrNLjH+xtiwav6jIGUhDByHiRdEOp1hYBWt7tpbeLb4Mhxyk0schaf6o47\niF3bur6WaEPUI8rUBDv0rXKS3leN3AJmmEEnRZPCoBrNDg1Esxxy/VNLHhhKRHoc\n+/K4sQuHeICO1a+yetfXIO6qa3rOOxEuL9ZpWvpSDqcRbYecDqa9CiZK\n-----END CERTIFICATE-----\n"
}
#

[qiwzhang]

There are two problems.

1) fetching JWKS fails. You need to use TLS. cluster googleapis_cluster need to setup TLC certificate to enable TLS.

2) For now, JWT filter doesn't support this type of JWKS. It only supports "RS256" and "ES256". Its JWKS looks like this file: https://github.com/google/jwt_verify_lib/blob/master/src/jwks_test.cc

[lizan]

for 2, you can just use https://www.googleapis.com/oauth2/v3/certs , this is in JWKS format.

[vladimir22]

Thank you very much, you helped me to find the local problem.

If you want I can try to make PR changes to Remote JWKS config example

in this file:

or you can do it by yourself, :

      http_filters:
      - name: envoy.filters.http.jwt_authn
        config:
         providers:
           example_provider:
             issuer: http://your-issuer/uaa/oauth/token
             audiences:
             - your_audience 
             - openid
             remote_jwks:
               http_uri:
                 uri: https://www.googleapis.com/oauth2/v3/certs 
                 cluster: googleapis_cluster
                 timeout:
                   seconds: 5
               cache_duration:
                 seconds: 10
         rules:
         - match:
             prefix: /
           requires:
             provider_name: example_provider

  - name: googleapis_cluster
    connect_timeout: 0.25s
    type: LOGICAL_DNS
    # Comment out the following line to test on v6 networks
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    hosts:
      - socket_address:
          address: googleapis.com
          port_value: 443
    tls_context: { sni: www.googleapis.com }

I think, this info would be helpful for learnig and configuring jwt_authn filter

[qiwzhang]

@vladimir22 please file a PR to submit your config sample. It will really help others.

[vladimir22]

I've created simple PR, please check it