Serving: Use better probe path than "/" for activator and KPA

Created on 31 Oct 2019 · 22Comments · Source: knative/serving

kind of issue this is:

/kind question

Activator fails probe when istio end user authentication policy is applied.

Expected Behavior

When istio end user authentication policy is applied for a service target (excluding /metrics and /healthz path), the service should respond back with "Origin authentication failed" if valid jwt is not provided.

Actual Behavior

Activator fails probe and responds with "context deadline exceeded".

Steps to Reproduce the Problem

Deploy a service
Apply the below end user auth policy

apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  name: policy-name
spec:
  targets:
  - name: svc-target
  origins:
  - jwt:
      issuer: "valid issuer"
      jwksUri: "some jwks uri"
      triggerRules:
      - excluded_paths:
        - prefix: /metrics 
        - prefix: /healthz       
  principalBinding: USE_ORIGIN

Hit the knative service end point. After a while you will receive "context deadline exceeded" error

Additional Info

Pods come up
If the policy is applied after the initial call which goes through the activator, I get the expected error as it goes through ingress gateway.
Activator is probing the root path like http://clusterip:port/ . If my service is also at the root path. I cannot exclude the root path from authentication.
If my service is not at the root level am able to exclude the root path and everything is working as expected.

arenetworking kinfeature kingood-first-issue kinquestion

Source

itsmurugappan

All 22 comments

@itsmurugappan: The label(s) kind/proposal cannot be applied. These labels are supported: ``

In response to this:

Activator fails probe when istio end user authentication policy is applied.

Expected Behavior

When istio end user authentication policy is applied for a service target (excluding /metrics and /healthz path), the service should respond back with "Origin authentication failed" if valid jwt is not provided.

Actual Behavior

Activator fails probe and responds with "context deadline exceeded".

Steps to Reproduce the Problem

Deploy a service

Apply the below end user auth policy
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
 name: policy-name
spec:
 targets:
 - name: svc-target
 origins:
 - jwt:
     issuer: "valid issuer"
     jwksUri: "some jwks uri"
     triggerRules:
     - excluded_paths:
       - prefix: /metrics 
       - prefix: /healthz       
 principalBinding: USE_ORIGIN
Hit the knative service end point. After a while you will receive "context deadline exceeded" error

Additional Info

Pods come up

If the policy is applied after the initial call which goes through the activator, I get the expected error as it goes through ingress gateway.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

knative-prow-robot on 31 Oct 2019

@itsmurugappan: The label(s) kind/proposal cannot be applied. These labels are supported: ``

In response to this:

Activator fails probe when istio end user authentication policy is applied.

Expected Behavior

When istio end user authentication policy is applied for a service target (excluding /metrics and /healthz path), the service should respond back with "Origin authentication failed" if valid jwt is not provided.

Actual Behavior

Activator fails probe and responds with "context deadline exceeded".

Steps to Reproduce the Problem

Deploy a service

Apply the below end user auth policy
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
 name: policy-name
spec:
 targets:
 - name: svc-target
 origins:
 - jwt:
     issuer: "valid issuer"
     jwksUri: "some jwks uri"
     triggerRules:
     - excluded_paths:
       - prefix: /metrics 
       - prefix: /healthz       
 principalBinding: USE_ORIGIN
Hit the knative service end point. After a while you will receive "context deadline exceeded" error

Additional Info

Pods come up

If the policy is applied after the initial call which goes through the activator, I get the expected error as it goes through ingress gateway.

Activator is probing the root path like http://clusterip:port/ . If my service is also at the root path. I cannot exclude the root path from authentication.

knative-prow-robot on 31 Oct 2019

@itsmurugappan: The label(s) kind/proposal cannot be applied. These labels are supported: ``

In response to this:

In what area(s)?
/area API

kind of issue this is:

/kind question

Activator fails probe when istio end user authentication policy is applied.

Expected Behavior

When istio end user authentication policy is applied for a service target (excluding /metrics and /healthz path), the service should respond back with "Origin authentication failed" if valid jwt is not provided.

Actual Behavior

Activator fails probe and responds with "context deadline exceeded".

Steps to Reproduce the Problem

Deploy a service

Apply the below end user auth policy
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
 name: policy-name
spec:
 targets:
 - name: svc-target
 origins:
 - jwt:
     issuer: "valid issuer"
     jwksUri: "some jwks uri"
     triggerRules:
     - excluded_paths:
       - prefix: /metrics 
       - prefix: /healthz       
 principalBinding: USE_ORIGIN
Hit the knative service end point. After a while you will receive "context deadline exceeded" error

Additional Info

Pods come up

If the policy is applied after the initial call which goes through the activator, I get the expected error as it goes through ingress gateway.

Activator is probing the root path like http://clusterip:port/ . If my service is also at the root path. I cannot exclude the root path from authentication.

If my service is not at the root level am able to exclude the root path and everything is working as expected.

knative-prow-robot on 31 Oct 2019

@itsmurugappan: The label(s) kind/proposal cannot be applied. These labels are supported: ``

In response to this:

kind of issue this is:

/kind question

Activator fails probe when istio end user authentication policy is applied.

Expected Behavior

When istio end user authentication policy is applied for a service target (excluding /metrics and /healthz path), the service should respond back with "Origin authentication failed" if valid jwt is not provided.

Actual Behavior

Activator fails probe and responds with "context deadline exceeded".

Steps to Reproduce the Problem

Deploy a service

Apply the below end user auth policy
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
 name: policy-name
spec:
 targets:
 - name: svc-target
 origins:
 - jwt:
     issuer: "valid issuer"
     jwksUri: "some jwks uri"
     triggerRules:
     - excluded_paths:
       - prefix: /metrics 
       - prefix: /healthz       
 principalBinding: USE_ORIGIN
Hit the knative service end point. After a while you will receive "context deadline exceeded" error

Additional Info

Pods come up

If the policy is applied after the initial call which goes through the activator, I get the expected error as it goes through ingress gateway.

Activator is probing the root path like http://clusterip:port/ . If my service is also at the root path. I cannot exclude the root path from authentication.

If my service is not at the root level am able to exclude the root path and everything is working as expected.

knative-prow-robot on 31 Oct 2019

Cc @tcnghia

itsmurugappan on 19 Nov 2019

I think we can make the prober uses a more specific path to make it easier for operators to whitelist.

tcnghia on 19 Nov 2019

👍1

I think we can make the prober uses a more specific path to make it easier for operators to whitelist.

@tcnghia after I discussed with you in kubecon, I made this change in my cluster. I am able to whitelist the prober endpoint and enforce auth on other paths. Is it ok , if i send a pull request for this ?

itsmurugappan on 24 Dec 2019

@itsmurugappan Yes, feel free to send a PR. Serving 0.12.x will cut January 21 so we probably want to have the PR land next week.

tcnghia on 9 Jan 2020

Thank you. Will send it soon.

itsmurugappan on 9 Jan 2020

@itsmurugappan I just opened https://github.com/knative/pkg/pull/984 to save you the trouble of updating across repos.

tcnghia on 9 Jan 2020

Thank you @tcnghia . Please correct me if I am wrong.
I made the change in this func as below

    httpDest := url.URL{
        Scheme: "http",
        Host:   dest,
        Path:   "/zzzprobezzz",
    }
    // NOTE: changes below may require changes to testing/roundtripper.go to make unit tests passing.
    return prober.Do(ctx, rw.transport, httpDest.String(),
        prober.WithHeader(network.ProbeHeaderName, queue.Name),
        prober.WithHeader(network.UserAgentKey, network.ActivatorUserAgent),
        prober.ExpectsBody(queue.Name),
        prober.ExpectsStatusCodes([]int{http.StatusOK}))

Since the url was getting created here, I added the path. I will change this to use 'WithPath' method you had created. Also please let me know if the path name is ok ?

itsmurugappan on 9 Jan 2020

@itsmurugappan yes, that won't require my change in knative/pkg#984. In fact I think it's clearer than using WithPath.

Please extract your path into a constant in the file. We can probably use something like /_internal/knative/activator/probe as a starting point. Others will chime in when they review.

Thanks alot.

tcnghia on 9 Jan 2020

👍1

@itsmurugappan: The label(s) kind/proposal cannot be applied, because the repository doesn't have them

In response to this:

kind of issue this is:

/kind question

Activator fails probe when istio end user authentication policy is applied.

Expected Behavior

When istio end user authentication policy is applied for a service target (excluding /metrics and /healthz path), the service should respond back with "Origin authentication failed" if valid jwt is not provided.

Actual Behavior

Activator fails probe and responds with "context deadline exceeded".

Steps to Reproduce the Problem

Deploy a service

Apply the below end user auth policy
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
 name: policy-name
spec:
 targets:
 - name: svc-target
 origins:
 - jwt:
     issuer: "valid issuer"
     jwksUri: "some jwks uri"
     triggerRules:
     - excluded_paths:
       - prefix: /metrics 
       - prefix: /healthz       
 principalBinding: USE_ORIGIN
Hit the knative service end point. After a while you will receive "context deadline exceeded" error

Additional Info

Pods come up

If the policy is applied after the initial call which goes through the activator, I get the expected error as it goes through ingress gateway.

Activator is probing the root path like http://clusterip:port/ . If my service is also at the root path. I cannot exclude the root path from authentication.

If my service is not at the root level am able to exclude the root path and everything is working as expected.

knative-prow-robot on 23 Jan 2020

We should follow up https://github.com/knative/serving/pull/6505 with making other probe paths consistent with what we used in https://github.com/knative/serving/pull/6505 .

tcnghia on 23 Jan 2020

/assign shreejad

tcnghia on 24 Feb 2020

Unless we have a strong case for 0.13, let's shoot for 0.14 here.

mattmoor on 25 Feb 2020

0.14 is good.

vagababov on 25 Feb 2020

FWIW, path /healthz seems to me like a better default. It's probably more commonly used and people have already whitelisted it.

yolocs on 17 Mar 2020

👍1

PR addressing this issue [#7445] was closed. As the PR was not "linked" to the issue, this issue did not get closed automatically.
@vagababov, @tcnghia How can I close this issue? I don't see the "Close Issue" button beside "Comment" at the bottom of this page.