Eventing: [BUG] Natss Channel doesn't work with Istio 1.1.x

[whynowy]

Describe the bug
After installing Istio 1.1.x in a k8s cluster, both Natss controller and dispatcher in knative-eventing namespace can not connect to natss service.

{"level":"error","ts":"2019-04-09T18:46:00.914Z","logger":"provisioner","caller":"clusterchannelprovisioner/controller.go:42","msg":"Connect() failed: ","eventing.knative.dev/clusterChannelProvisioner":"natss","eventing.knative.dev/clusterChannelProvisionerComponent":"Controller","error":"read tcp 100.104.99.25:52518->100.71.171.214:4222: i/o timeout","stacktrace":"github.com/knative/eventing/contrib/natss/pkg/controller/clusterchannelprovisioner.ProvideController\n\t/go/src/github.com/knative/eventing/contrib/natss/pkg/controller/clusterchannelprovisioner/controller.go:42\nmain.main\n\t/go/src/github.com/knative/eventing/contrib/natss/pkg/controller/main.go:53\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}

Expected behavior
Natss channel works.

To Reproduce
Upgrade Istio to 1.1

Knative release version
V0.5.0

Additional context
No.

[abbi-gaurav]

We discovered this recently while upgrading Kyma to Istio 1.1. It seems this issue was discovered in the past with Istio. For some unknown reason, NATSS started working with Istio at some 0.x.

Here are some related issues
https://github.com/istio/istio/issues/10832
https://github.com/istio/old_issues_repo/issues/338
https://github.com/nats-io/nats-operator/issues/88

For Kyma there was no other option but to disable the Istio sidecar. I also contacted the natss community and they recommend not to have any proxy between natss client and server.

[whynowy]

@abbi-gaurav : Thanks for the information I disabled sidecar injection for natss statefulset and now it's working.

What is the reason we put Natss on the mesh?

[radufa]

A temporary simple solution to secure NATSS is to use "Authenticated Users": https://nats.io/documentation/streaming/nats-streaming-tls/
That means the the client pods (NATSS Controller + Dispatcher) and the NATSS Server pod should use the same "user" based on a shared secret done at the installation time. Of course, for a production ready solution, you have to thing how to rotate the credentials, etc.

We hoped that with the supporting of mTLS for TCP connections in Istio 1.1, NATSS would be part of the mesh too.
In Kyma we are using now the "shared secret" solution.

[akashrv]

/close

[knative-prow-robot]

@akashrv: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.