Envoy: JWT Tokens cannot be verified

[fscz]

I recently installed Istio 1.6.2 and would like to set up JWT Auth. I am using the following configuration.

apiVersion: "security.istio.io/v1beta1"
kind: "RequestAuthentication"
metadata:
  name: default
  namespace: istio-system
spec:
  jwtRules:
  - issuer: "https://keycloak.${domain}/auth/realms/badatalab"
    jwksUri: "https://keycloak.${domain}/auth/realms/badatalab/protocol/openid-connect/certs"
    jwtHeaders:
    - "x-amzn-oidc-accesstoken"
---
apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
  name: control-api
spec:
  action: ALLOW
  selector:
   matchLabels:
     app: control-api
  rules:  
  - to:
    - operation:
        methods: ["GET"]
        paths: ["/api/keycloak/claim/*"]
  - to:
    - operation:
        methods: ["OPTIONS"]
  - when:
    - key: request.auth.claims[iss]
      values: ["https://keycloak.${domain}/auth/realms/badatalab"]

I tested a lot with this and unfortunately, the following JWTRule does not work:

- when:
    - key: request.auth.claims[iss]
      values: ["https://keycloak.${domain}/auth/realms/badatalab"]

It does also not work if I change values to ["*"]. I does work if I check for key: request.headers[User-Agent] with a specific value or wildcard.

Envoy log prints the following:

2020-06-19T17:18:06.278950Z debug   envoy jwt   [external/envoy/source/extensions/filters/http/jwt_authn/filter.cc:124] Called Filter : setDecoderFilterCallbacks
2020-06-19T17:18:06.279008Z debug   envoy jwt   [external/envoy/source/extensions/filters/http/jwt_authn/filter.cc:46] Called Filter : decodeHeaders
2020-06-19T17:18:06.279018Z debug   envoy jwt   [external/envoy/source/extensions/filters/http/jwt_authn/matcher.cc:71] Prefix requirement '/' matched.
2020-06-19T17:18:06.279023Z debug   envoy jwt   [external/envoy/source/extensions/filters/http/jwt_authn/extractor.cc:188] extract authorizationBearer 
2020-06-19T17:18:06.279034Z debug   envoy jwt   [external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:127] origins-0: JWT authentication starts (allow_failed=false), tokens size=0
2020-06-19T17:18:06.279037Z debug   envoy jwt   [external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:267] origins-0: JWT token verification completed with: Jwt is missing
2020-06-19T17:18:06.279041Z debug   envoy jwt   [external/envoy/source/extensions/filters/http/jwt_authn/verifier.cc:202] Called AllowMissingVerifierImpl.verify : verify
2020-06-19T17:18:06.279044Z debug   envoy jwt   [external/envoy/source/extensions/filters/http/jwt_authn/extractor.cc:188] extract authorizationBearer 
2020-06-19T17:18:06.279046Z debug   envoy jwt   [external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:127] _IS_ALLOW_MISSING_: JWT authentication starts (allow_failed=false), tokens size=0
2020-06-19T17:18:06.279049Z debug   envoy jwt   [external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:267] _IS_ALLOW_MISSING_: JWT token verification completed with: Jwt is missing
2020-06-19T17:18:06.279053Z debug   envoy jwt   [external/envoy/source/extensions/filters/http/jwt_authn/filter.cc:84] Called Filter : check complete OK
2020-06-19T17:18:06.279155Z debug   envoy rbac  [external/envoy/source/extensions/filters/http/rbac/rbac_filter.cc:74] checking request: requestedServerName: outbound_.80_._.control-api.default.svc.cluster.local, sourceIP: 172.19.164.128:34692, directRemoteIP: 172.19.164.128:34692, remoteIP: 172.19.130.211:0,localAddress: 172.19.161.29:8000, ssl: uriSanPeerCertificate: spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account, dnsSanPeerCertificate: , subjectPeerCertificate: , headers: ':authority', 'control-api.datalab-dev.ddc.telefonica.de'
':path', '/'
':method', 'GET'
'x-forwarded-for', '10.170.88.13,172.19.130.211'
'x-forwarded-proto', 'http'
'x-forwarded-port', '443'
'x-amzn-trace-id', 'Root=1-5eecf34e-b9264a259baea6da8138fa3c'
'x-amzn-oidc-data', 'eyJ0eXAiOiJKV1QiLCJraWQiOiIxM2ViYzg1OS1lMzA1LTQ2MmQtYTQwMi00MjZjNDNkMGNiYTIiLCJhbGciOiJFUzI1NiIsImlzcyI6Imh0dHBzOi8va2V5Y2xvYWsuZGF0YWxhYi1kZXYuZGRjLnRlbGVmb25pY2EuZGUvYXV0aC9yZWFsbXMvYmFkYXRhbGFiIiwiY2xpZW50IjoiYXdzIiwic2lnbmVyIjoiYXJuOmF3czplbGFzdGljbG9hZGJhbGFuY2luZzpldS1jZW50cmFsLTE6ODg2MTQyMjkxOTgwOmxvYWRiYWxhbmNlci9hcHAvYWxiL2EzNzc1OTA2Mjg4N2YyOWYiLCJleHAiOjE1OTI1ODcyMDZ9.eyJzdWIiOiI1ZDM4M2NkZC1iZmRmLTQzOGYtOTJhNi00MmI2ODg4ZjU4YTgiLCJlbWFpbF92ZXJpZmllZCI6dHJ1ZSwibmFtZSI6IkZhYmlhbiBTY2jDvHR6IiwicHJlZmVycmVkX3VzZXJuYW1lIjoibnExMDAxNzU2OSIsImdpdmVuX25hbWUiOiJGYWJpYW4iLCJmYW1pbHlfbmFtZSI6IlNjaMO8dHoiLCJlbWFpbCI6ImZhYmlhbi5zY2h1ZXR6LmV4dGVybmFsQHRlbGVmb25pY2EuY29tIiwiZXhwIjoxNTkyNTg3MjA2LCJpc3MiOiJodHRwczovL2tleWNsb2FrLmRhdGFsYWItZGV2LmRkYy50ZWxlZm9uaWNhLmRlL2F1dGgvcmVhbG1zL2JhZGF0YWxhYiJ9.PqdI10pj1axPLtvi30WLfGdq9UbMD3wnqewew1qzaFK4e3ykWLR741Xt5pm_FTDRj9jUBNIq0-MYZq5IKP7Esw=='
'x-amzn-oidc-identity', '5d383cdd-bfdf-438f-92a6-42b6888f58a8'
'x-amzn-oidc-accesstoken', 'eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICItd0pSQVFuX2xBdFFuMXJrQS1oWWJJaUN6dGREQ0xQQTBWZzFfdTg4RUZNIn0.eyJqdGkiOiI1MDEwNGRmMS1iODU4LTQ2YmMtYTY1OS04ZjFkNjQyNzUxZjgiLCJleHAiOjE1OTI1OTgxMjIsIm5iZiI6MCwiaWF0IjoxNTkyNTYyMTIzLCJpc3MiOiJodHRwczovL2tleWNsb2FrLmRhdGFsYWItZGV2LmRkYy50ZWxlZm9uaWNhLmRlL2F1dGgvcmVhbG1zL2JhZGF0YWxhYiIsInN1YiI6IjVkMzgzY2RkLWJmZGYtNDM4Zi05MmE2LTQyYjY4ODhmNThhOCIsInR5cCI6IkJlYXJlciIsImF6cCI6ImF3cyIsImF1dGhfdGltZSI6MTU5MjU2MjEyMywic2Vzc2lvbl9zdGF0ZSI6IjJmOTBkZDE4LTI3NzUtNDg4ZC1hMDA5LTY0N2FiOTFlMTczOSIsImFjciI6IjEiLCJhbGxvd2VkLW9yaWdpbnMiOlsiIl0sInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIl19LCJzY29wZSI6Im9wZW5pZCBwcm9maWxlIGVtYWlsIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsIm5hbWUiOiJGYWJpYW4gU2Now7x0eiIsImNvbnRyb2wiOlsiY29udHJvbC1kdW1teS1ncm91cCJdLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJucTEwMDE3NTY5IiwiZ2l2ZW5fbmFtZSI6IkZhYmlhbiIsImZhbWlseV9uYW1lIjoiU2Now7x0eiIsImVtYWlsIjoiZmFiaWFuLnNjaHVldHouZXh0ZXJuYWxAdGVsZWZvbmljYS5jb20ifQ.BwDtqT9zZo-v76GrbXFE6l8moCy9SeBxAMgNX67Bf2EPwyWPTZv3VTopTovyDoSBY4dVtGoyieWbJwGyaNaj0HDwwzxJWn6C24SpRr4WaEqACuBPUJqzRJkZ4LW2rjDzsbbsZg2uNIri0XLnviAZIcC7o_KRcvuuH2hJMIs0xzJT-i5iL0oMYHvQulQsguTTGcSN3ObZnwj-tL9xN18JcAQe_SHm8b_0ESsxw8zeNJQdmk6UEGjUmC_yiUxbAFl6eCjfLO6bLFsfZMWBYFw8SXJA3j9zeBo5aQf6SDJ8YhB8YNmGYvL8TWZVbaYdYsWztNHyN7TIN1V5dlAk2bEJug'
'cache-control', 'max-age=0'
'upgrade-insecure-requests', '1'
'user-agent', 'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3750.0 Iron Safari/537.36'
'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8'
'accept-encoding', 'gzip, deflate, br'
'accept-language', 'de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7'
'cookie', 'experimentation_subject_id=IjFkMTgzNjAwLTcxNzktNDYxNy1hNjMwLTE2MDEyMTgyOTRjNCI%3D--4cbf1e820401a364db2c85cfceadc3e5dc0f6d6a; control_ng_csrftoken=tKvyS9WRXr1GCUqhd3TamR1xnXNh5Cl24gIuoijfCAouJyF96sbbA8Q1g8o600xy; control_ng_sessionid=qbazcnkpj4niqnwewrcuez8r219amvl5'
'x-envoy-external-address', '172.19.130.211'
'x-request-id', '121ee64f-0c7e-491b-b70f-f2b1d9eb23b7'
'x-b3-traceid', 'a012bccc54117216b17b8311780f5b9c'
'x-b3-spanid', 'b17b8311780f5b9c'
'x-b3-sampled', '0'
'content-length', '0'
'x-forwarded-client-cert', 'By=spiffe://cluster.local/ns/default/sa/control-api;Hash=1fc3c9525cfd6d3c96bedc338fa9524a0dbff7100f2e5dd75eae6084a633ff89;Subject="";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account'
, dynamicMetadata: filter_metadata {
  key: "istio_authn"
  value {
    fields {
      key: "source.namespace"
      value {
        string_value: "istio-system"
      }
    }
    fields {
      key: "source.principal"
      value {
        string_value: "cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
      }
    }
    fields {
      key: "source.user"
      value {
        string_value: "cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
      }
    }
  }
}

2020-06-19T17:18:06.279184Z debug   envoy rbac  [external/envoy/source/extensions/filters/http/rbac/rbac_filter.cc:117] enforced denied
2020-06-19T17:18:06.279226Z debug   envoy jwt   [external/envoy/source/extensions/filters/http/jwt_authn/filter.cc:39] Called Filter : onDestroy
2020-06-19T17:19:56.382270Z debug   envoy jwt   [external/envoy/source/extensions/filters/http/jwt_authn/filter_config.cc:9] Loaded JwtAuthConfig: providers {
  key: "origins-0"
  value {
    issuer: "https://keycloak.datalab-dev.ddc.telefonica.de/auth/realms/badatalab"
    local_jwks {
      inline_string: "{\"keys\":[{\"kid\":\"-wJRAQn_lAtQn1rkA-hYbIiCztdDCLPA0Vg1_u88EFM\",\"kty\":\"RSA\",\"alg\":\"RS256\",\"use\":\"sig\",\"n\":\"hcikReISUPOiCSZ-2--RR8TLOHSFaDiWGhQV267u4DrFNgnHcdszU4lE1Z3DmJjAVPlzZPbSa9xF7D8DWew8JNJT2qYTranbaYYUrN3Kt3glQ6pGy1hxwnz0i9gQlH4Zjrq0mYkJnTgTj4qIYOk4hFFqmo2-_Kfq9Knx1NuuNBpjcRSvFwcYvNzPzgLoCB3ldAxf9sctMrroIac-xXMcOm4TPfTzoJv5RJ1gXAY68-yo3iGFRPbdY5qVkn59khOarqvO0tfobA_wvSxVbKiTtXk-_p4OSg8ci-ie4pDmqUM-8-N6oGkaVNeITN2q-9Jxg0DzQTGNAi8SM_GiwVydtw\",\"e\":\"AQAB\",\"x5c\":[\"MIICoTCCAYkCBgFwFSOArDANBgkqhkiG9w0BAQsFADAUMRIwEAYDVQQDDAliYWRhdGFsYWIwHhcNMjAwMjA1MTEzNTEzWhcNMzAwMjA1MTEzNjUzWjAUMRIwEAYDVQQDDAliYWRhdGFsYWIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCFyKRF4hJQ86IJJn7b75FHxMs4dIVoOJYaFBXbru7gOsU2Ccdx2zNTiUTVncOYmMBU+XNk9tJr3EXsPwNZ7Dwk0lPaphOtqdtphhSs3cq3eCVDqkbLWHHCfPSL2BCUfhmOurSZiQmdOBOPiohg6TiEUWqajb78p+r0qfHU2640GmNxFK8XBxi83M/OAugIHeV0DF/2xy0yuughpz7Fcxw6bhM99POgm/lEnWBcBjrz7KjeIYVE9t1jmpWSfn2SE5quq87S1+hsD/C9LFVsqJO1eT7+ng5KDxyL6J7ikOapQz7z43qgaRpU14hM3ar70nGDQPNBMY0CLxIz8aLBXJ23AgMBAAEwDQYJKoZIhvcNAQELBQADggEBAHaHuXu4Nv0u3yjx6eyCyf64PTQNQUVPQOIPEqNlZzLOzJg1RGMZehjINyjleU8DfFisE5GlNA+ETBWj3Kc/jpMKmjaTjgJegHuCGxeuTZ5SF8UEhGvXwe8/b91q1pqvYq6TuRjLPf9ZwsubsL7tYC23OnkEZrJOtUxua4Xu3M9OyBvBCZhI8EJBmybcjA9eea2MyXD8URGWGGnIkfVut5Z+03MJB3waywYC8AWoCTkf/BVnGq/htimOBqRgBB/5apT2sa0qGrQcl64ADxRsgpZEVMhnnd8DQ5TgTqPTWE24HDLoZvbup9KEnCkq29nzV2TNFbvOAN8M0qqgJ2b5maY=\"],\"x5t\":\"BFKRUU-wOHWXRKqyIzLznPhwAFg\",\"x5t#S256\":\"SyY8vkk5zjbo7j9foe1lmx2-qqK8-bZwPZGZktdrnB4\"}]}"
      183412668: "envoy.api.v2.core.DataSource"
    }
    payload_in_metadata: "https://keycloak.datalab-dev.ddc.telefonica.de/auth/realms/badatalab"
    183412668: "envoy.config.filter.http.jwt_authn.v2alpha.JwtProvider"
  }
}
rules {
  match {
    prefix: "/"
    183412668: "envoy.api.v2.route.RouteMatch"
  }
  requires {
    requires_any {
      requirements {
        provider_name: "origins-0"
        183412668: "envoy.config.filter.http.jwt_authn.v2alpha.JwtRequirement"
      }
      requirements {
        allow_missing {
        }
        183412668: "envoy.config.filter.http.jwt_authn.v2alpha.JwtRequirement"
      }
      183412668: "envoy.config.filter.http.jwt_authn.v2alpha.JwtRequirementOrList"
    }
    183412668: "envoy.config.filter.http.jwt_authn.v2alpha.JwtRequirement"
  }
  183412668: "envoy.config.filter.http.jwt_authn.v2alpha.RequirementRule"
}
183412668: "envoy.config.filter.http.jwt_authn.v2alpha.JwtAuthentication"

This leads me to believe that in my RequestAuthentication, my custom JWT Header is somehow ignored. The header is specified with:

jwtHeaders:
- "x-amzn-oidc-accesstoken"

Any ideas?

[qiwzhang]

Could you verify following is include din envoy jwt_authn filter config:
` from_header name: x-amzn-oidc-accesstoken
If no, please make sure Istio generate such config.

[fscz]

How do I check that?

[qiwzhang]

Here is the config from your log

providers {
  key: "origins-0"
  value {
    issuer: "https://keycloak.datalab-dev.ddc.telefonica.de/auth/realms/badatalab"
    local_jwks {
      inline_string:
"{\"keys\":[{\"kid\":\"-wJRAQn_lAtQn1rkA-hYbIiCztdDCLPA0Vg1_u88EFM\",\"kty\":\"RSA\",\"alg\":\"RS256\",\"use\":\"sig\",\"n\":\"hcikReISUPOiCSZ-2--RR8TLOHSFaDiWGhQV267u4DrFNgnHcdszU4lE1Z3DmJjAVPlzZPbSa9xF7D8DWew8JNJT2qYTranbaYYUrN3Kt3glQ6pGy1hxwnz0i9gQlH4Zjrq0mYkJnTgTj4qIYOk4hFFqmo2-_Kfq9Knx1NuuNBpjcRSvFwcYvNzPzgLoCB3ldAxf9sctMrroIac-xXMcOm4TPfTzoJv5RJ1gXAY68-yo3iGFRPbdY5qVkn59khOarqvO0tfobA_wvSxVbKiTtXk-_p4OSg8ci-ie4pDmqUM-8-N6oGkaVNeITN2q-9Jxg0DzQTGNAi8SM_GiwVydtw\",\"e\":\"AQAB\",\"x5c\":[\"MIICoTCCAYkCBgFwFSOArDANBgkqhkiG9w0BAQsFADAUMRIwEAYDVQQDDAliYWRhdGFsYWIwHhcNMjAwMjA1MTEzNTEzWhcNMzAwMjA1MTEzNjUzWjAUMRIwEAYDVQQDDAliYWRhdGFsYWIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCFyKRF4hJQ86IJJn7b75FHxMs4dIVoOJYaFBXbru7gOsU2Ccdx2zNTiUTVncOYmMBU+XNk9tJr3EXsPwNZ7Dwk0lPaphOtqdtphhSs3cq3eCVDqkbLWHHCfPSL2BCUfhmOurSZiQmdOBOPiohg6TiEUWqajb78p+r0qfHU2640GmNxFK8XBxi83M/OAugIHeV0DF/2xy0yuughpz7Fcxw6bhM99POgm/lEnWBcBjrz7KjeIYVE9t1jmpWSfn2SE5quq87S1+hsD/C9LFVsqJO1eT7+ng5KDxyL6J7ikOapQz7z43qgaRpU14hM3ar70nGDQPNBMY0CLxIz8aLBXJ23AgMBAAEwDQYJKoZIhvcNAQELBQADggEBAHaHuXu4Nv0u3yjx6eyCyf64PTQNQUVPQOIPEqNlZzLOzJg1RGMZehjINyjleU8DfFisE5GlNA+ETBWj3Kc/jpMKmjaTjgJegHuCGxeuTZ5SF8UEhGvXwe8/b91q1pqvYq6TuRjLPf9ZwsubsL7tYC23OnkEZrJOtUxua4Xu3M9OyBvBCZhI8EJBmybcjA9eea2MyXD8URGWGGnIkfVut5Z+03MJB3waywYC8AWoCTkf/BVnGq/htimOBqRgBB/5apT2sa0qGrQcl64ADxRsgpZEVMhnnd8DQ5TgTqPTWE24HDLoZvbup9KEnCkq29nzV2TNFbvOAN8M0qqgJ2b5maY=\"],\"x5t\":\"BFKRUU-wOHWXRKqyIzLznPhwAFg\",\"x5t#S256\":\"SyY8vkk5zjbo7j9foe1lmx2-qqK8-bZwPZGZktdrnB4\"}]}"
      183412668: "envoy.api.v2.core.DataSource"
    }
    payload_in_metadata:
"https://keycloak.datalab-dev.ddc.telefonica.de/auth/realms/badatalab"
    183412668: "envoy.config.filter.http.jwt_authn.v2alpha.JwtProvider"
  }
}

It did not define "from_headers".

Please take a look at this doc
https://github.com/envoyproxy/envoy/blob/master/api/envoy/extensions/filters/http/jwt_authn/v3/config.proto#L144
on how to config jwt_authn

You may need to check Istio doc on how to config "from_headers" for jwt.

On Mon, Jun 22, 2020 at 10:07 AM fscz [e-mail] wrote:

How do I check that?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/envoyproxy/envoy/issues/11672#issuecomment-647651223,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACZ6TB3KHVIJDFQGE45UIXTRX6FVJANCNFSM4ODI2TBQ
.

[fscz]

The documentation is wrong. At the top of https://istio.io/latest/docs/reference/config/security/jwt/#JWTRule
it reads.

issuer: https://example.com
jwksUri: https://example.com/.secret/jwks.json
jwtHeaders:
- "x-goog-iap-jwt-assertion"

However, correct is:

issuer: "xyz"
jwksUri: "abc"
fromHeaders:
  - name: x-amzn-oidc-accesstoken

[qiwzhang]

Please file an issue with Issue. This is Envoy repo.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.