Docs: Need instructions about how to configure mTLS within Knative cluster

[ZhiminXiang]

Describe the change you'd like to see
As Istio 1.1 was landed within Knative, Istio mTLS should be unblocked. (see more about the mTLS issue).

So we need instructions about how to configure Istio mTLS within Knative cluster.

[rgregg]

@ZhiminXiang is this something you could draft instructions for?

[ZhiminXiang]

Currently we still see E2E test failures when enabling mTLS (https://github.com/knative/serving/pull/5648). I think we need to root cause the E2E test failures to make sure mTLS works before starting the instruction.

[knative-housekeeping-robot]

Issues go stale after 90 days of inactivity.
Mark the issue as fresh by adding the comment /remove-lifecycle stale.
Stale issues rot after an additional 30 days of inactivity and eventually close.
If this issue is safe to close now please do so by adding the comment /close.

Send feedback to Knative Productivity Slack channel or file an issue in knative/test-infra.

/lifecycle stale

[tshak]

:wave: We're struggling to get strict mTLS to play nicely with Knative. Seeing as #5648 is resolved I would love to see documentation on this scenario.

[knative-housekeeping-robot]

Stale issues rot after 30 days of inactivity.
Mark the issue as fresh by adding the comment /remove-lifecycle rotten.
Rotten issues close after an additional 30 days of inactivity.
If this issue is safe to close now please do so by adding the comment /close.

Send feedback to Knative Productivity Slack channel or file an issue in knative/test-infra.

/lifecycle rotten

[grounded042]

I’m also interested in documentation on getting mTLS up and running as it’s something we’re needing and there doesn’t seem to be a solid source on how to go about doing that.

[abrennan89]

@ZhiminXiang is there any update on instructions for this yet? Do you plan to create them?

[nak3]

@abrennan89 sorry for the delay. I will create the doc.

/assign

[tshak]

Thanks @nak3. In particular my challenge is getting strict mTLS setup. Happy to help review and test the docs when they're ready. 👍

[nak3]

@tshak Thank you so much! I sent the draft here https://github.com/knative/docs/pull/2593

It seems that DestinationRule does not work as expected on Istio 1.5 (I could access to ksvc without it) but it works on Istio 1.6 as described in blog.

[tshak]

Great work! I'm going to get to this within the next couple of days on an Istio 1.6 cluster using the IstioOperator. On initial glance, I'm a bit concerned with section: https://github.com/nak3/docs/blob/blog-mtls/blog/articles/configuraing-istio-mtls-on-knative-serving.md#step-2-enable-strict-mtls-on-entire-mesh-and-permissive-on-the-system-namespace

From a security perspective, setting PERMISSIVE anywhere sets off alarms. Is it not possible that a compromised service could craft a request to the activator to get access to another service that they normally would not have access to?

There may not be a way around it, but at least it'd be good to document any security risks with this approach if valid. May be good to have an Istio team member review as well.

Thanks again - I look forward to giving this a spin! 👍

[nak3]

Thank you for your comment. Actually we just started talking about it https://github.com/knative/net-istio/issues/150.

I tested it and all STRICT mode works by disabling side injection to {istio-}webhook.
With said, it will be very complicated to configure DestinationRules, so I used enableAutoMtls to true in the blog post.

[tshak]

After following the blog post, I was unable to update config-istio. This is because I'm running net-certmanager. Patching the net-certmanager-webhook to disable the istio sidecar resolved the issue. I'd add a note about this as I imagine many people are running net-certmanager as well.

Otherwise everything seems to be working so far. This helped a lot. Thanks!

[tshak]

Also, you may want to note that if someone doesn't want to run the cluster-local-gateway (which is the more secure default), that they'll need to update config-istio to specify local-gateway.mesh: "mesh"

See also: https://github.com/knative/net-istio/blob/82f58eb4fb0f8fa44d2b59065058d0adf874a134/config/config.yaml#L69

[tshak]

The basics appear working but I can't seem to configure a working authorization policy with knative. For example, let's say that I want to only allow intranamespace traffic, plus traffic from the istio-ingressgateway:

kind: AuthorizationPolicy
metadata:
  name: intranamespace
  namespace: demo
spec:
  action: ALLOW
  rules:
  - from:
    - source:
        namespaces: ["demo", "istio-system"]

This does not work as expected with knative services. I get a 403 with the message: RBAC: access denied. It does however work when I add knative-serving to the namespace list. However, this effectively opens access up to all knative services regardless of the namespace that it's hosted in, defeating the purpose of the policy. Maybe we can track this as a separate issue but I wanted to raise it up here in case I was doing something obviously wrong.

[tshak]

A bit more about my scenario: I have a namespace which has one app that serves traffic via the istio-ingressgateway. In the same namespace I have multiple backend services that have the serving.knative.dev/visibility: cluster-local set that should only be accessed from within the namespace. I could craft an AuthorizationPolicy with a selector for only the externally exposed service. My principle concern is that when one grants access to the knative-serving namespace that it effectively grants access to all knative services in the mesh which is probably not obvious to most operators.

[nak3]

Thank you so much for your feeedback @tshak. It is really helpful and totally make sense.

I'd add a note about this as I imagine many people are running net-certmanager as well.

Yes, definitely. I updated my article.

Also, you may want to note that if someone doesn't want to run the cluster-local-gateway (which is the more secure default), that they'll need to update config-istio to specify local-gateway.mesh: "mesh"

Yes, I will write the doc about it.

The basics appear working but I can't seem to configure a working authorization policy with knative.

I think that there are two solutions.

a) you can block by specify more strict policy as described in the second example in https://github.com/knative/serving/issues/8364#issuecomment-645904009.
b) Use "allowing path" instead of namespace as described in Allowing access from system pods by paths.

[nak3]

@tshak If you like, would you mind reviewing in https://github.com/knative/docs/pull/2583 ? Especially, as you also faced the issue, I'm wondering if Allowing access from system pods by namespace should be documented or not.

[tshak]

Thanks for all the help thus far! Overall #2583 looks good. I thought that I had tested the path approach based on the GKE examples and it still didn't work. I'll try to test this again soon.

It would be nice to have a dedicated set of conformance tests that are easy for anyone running Knative + Istio + STRICT mTLS to run on their clusters to prove that their policies are configured correctly. Or maybe something like istioctl analyze but more purpose built for Knative. It's easy to make a policy, istio, or knative configuration change and break critical functionality and/or create a security hole. 🤔

[Shashankft9]

@nak3 about allowing access from system pods by namespace:
By applying the whitelist-by-namespaces policy that you have mentioned in the doc, we practically do allow /metrics and /healthz , but we also allow curl requests from anywhere - be it a non-mesh pod or a mesh-pod, since they all pass through activator pod in ns knative-serving. So, isn't it advisable to use only the whitelist-by-paths policy from the doc otherwise it defeats the purpose of applying these policies, correct?

[nak3]

@Shashankft9 Yeah, I think we should remove the allow list by namespace example as it is confusable and not so it is easy to create a security hole. I will update the draft of the doc.

[abrennan89]

Requested input from @nak3 on whether this can be closed since a related PR was merged. Not sure what's left here to do from a docs perspective.

[nak3]

Thank you for the reminder. https://github.com/knative/docs/pull/3118 should have addressed this. So let's close this.

/close

[knative-prow-robot]

@nak3: Closing this issue.

In response to this:

Thank you for the reminder. https://github.com/knative/docs/pull/3118 should have addressed this. So let's close this.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.