Eventing: Channels reply doesn't work on minikube

[slinkydeveloper]

Describe the bug
The channels reply functionality doesn't work. I tried both InMemoryChannel and KafkaChannel. After the subscriber service replies, the dispatcher receives the message but it fails to forward it. Dispatcher log (IMC):

{"level":"info","ts":"2019-10-14T13:33:22.723Z","logger":"inmemorychannel_dispatcher.in-memory-channel-dispatcher","caller":"channel/message_dispatcher.go:112","msg":"Dispatching message to http://example-inbound-channel.default.svc.cluster.local","knative.dev/controller":"in-memory-channel-dispatcher"}
{"level":"error","ts":"2019-10-14T13:33:52.715Z","logger":"inmemorychannel_dispatcher.in-memory-channel-dispatcher","caller":"fanout/fanout_handler.go:121","msg":"Fanout had an error","knative.dev/controller":"in-memory-channel-dispatcher","error":"Failed to forward reply Post http://example-inbound-channel.default.svc.cluster.local: dial tcp 10.108.102.77:80: i/o timeout","stacktrace":"knative.dev/eventing/pkg/channel/fanout.(*Handler).dispatch\n\t/home/prow/go/src/knative.dev/eventing/pkg/channel/fanout/fanout_handler.go:121\nknative.dev/eventing/pkg/channel/fanout.createReceiverFunction.func1.1\n\t/home/prow/go/src/knative.dev/eventing/pkg/channel/fanout/fanout_handler.go:95"}

Dispatcher log (KafkaChannel master version):

{"level":"info","ts":"2019-10-14T10:18:04.993Z","logger":"controller","caller":"channel/message_dispatcher.go:116","msg":"Dispatching message to http://example-inbound-channel.default.svc.cluster.local"}
{"level":"warn","ts":"2019-10-14T10:18:34.993Z","logger":"controller","caller":"dispatcher/dispatcher.go:213","msg":"Error in consumer group","error":"Failed to forward reply Post http://example-inbound-channel.default.svc.cluster.local: dial tcp 10.98.216.68:80: i/o timeout"}

The ip you see in the logs is the dispatcher cluster ip

Expected behavior
I apply a subscription like:

apiVersion: messaging.knative.dev/v1alpha1
kind: Subscription
metadata:
  name: sub-1
  namespace: default
spec:
  channel:
    apiVersion: messaging.knative.dev/v1alpha1
    kind: InMemoryChannel
    name: example-inbound-channel
  reply:
    channel:
      apiVersion: messaging.knative.dev/v1alpha1
      kind: InMemoryChannel
      name: example-outbound-channel
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1alpha1
      kind: Service
      name: example-service

And I subscribe a logger service to example-outbound-channel. The example-service just forwards what it receives. It's tested and it works correctly.

I expect that when i send a message to the inbound channel, the logger correctly publishes the event, but it doesn't

Knative release version
Knative Serving/Eventing 0.9.0
Kubernetes 1.16
Istio 1.1.7
Minikube 1.4.0

Additional context
Since sequences creates subscriptions like above, they are broken too. Example Sequence:

apiVersion: messaging.knative.dev/v1alpha1
kind: Sequence
metadata:
  name: sequence
spec:
  channelTemplate:
    apiVersion: messaging.knative.dev/v1alpha1
    kind: InMemoryChannel
  steps:
    - ref:
        apiVersion: serving.knative.dev/v1alpha1
        kind: Service
        name: example-service
    - ref:
        apiVersion: serving.knative.dev/v1alpha1
        kind: Service
        name: event-display

[slinkydeveloper]

I experience the same problem with eventing v0.8.0

[yolocs]

Purely speculating here. Do you have cluster-local-gateway deployed? Like mentioned https://github.com/knative/serving/blob/master/DEVELOPMENT.md#deploy-istio

[slinkydeveloper]

Yes. Also services are available as usual, pinging them directly

[slinkydeveloper]

$ kubectl get pods -n istio-system 
NAME                                     READY   STATUS    RESTARTS   AGE
cluster-local-gateway-7f498585cf-92nmd   1/1     Running   0          2m45s
cluster-local-gateway-7f498585cf-rb2tf   1/1     Running   0          2m45s
istio-ingressgateway-577f6f9c49-ncnmt    1/2     Running   0          2m30s
istio-ingressgateway-577f6f9c49-zv2bt    2/2     Running   0          2m45s
istio-pilot-6ddb67676c-vtpt7             1/1     Running   0          2m45s

[lionelvillard]

stupid question: I suppose example-outbound-channel exists and accessible?

[n3wscott]

I think you also need one in the namespace you are attempting to communicate in.

[slinkydeveloper]

stupid question: I suppose example-outbound-channel exists and accessible?

Yes. All subscriptions, channels and services are marked as ready

I think you also need one in the namespace you are attempting to communicate in.

I've created a container and i've sent requests using curl. Example with sequence: curl -d '"Hello!"' -H "Content-type: application/json" -H "Ce-id: 1" -H "Ce-type: example.kfn" -H "Ce-specversion: 0.2" -H "Ce-source: external" http://sequence-kn-sequence-0-kn-channel.default.svc.cluster.local
All ksvc and channels lives in the same namespace

[slinkydeveloper]

There's any it test i can try to run locally to check it? It could a good help to diagnose the issue (or at least understand if there's any)

[lionelvillard]

you can try running this: https://github.com/knative/docs/tree/master/docs/eventing/samples/sequence/sequence-reply-to-event-display

[slinkydeveloper]

I already tried using sequences (the one you see on top does the same thing that manually chaining subscriptions, and it doesn't work for the same problem too) but they don't work. I'm wondering if there is an it test that i can run and debug locally

[vaikas]

There are e2e tests here that you should be able to run on your cluster.
https://github.com/knative/eventing/tree/master/test

[slinkydeveloper]

I experience the same error running the e2e test TestSequence with go test -tags=e2e -timeout=20m -parallel=1 -v -run TestSequence ./test/e2e

{"level":"error","ts":"2019-10-14T18:17:58.037Z","logger":"inmemorychannel_dispatcher.in-memory-channel-dispatcher","caller":"fanout/fanout_handler.go:120","msg":"Fanout had an error","knative.dev/controller":"in-memory-channel-dispatcher","error":"failed to forward reply to http://e2e-sequence-kn-sequence-1-kn-channel.test-sequence-68mtm.svc.cluster.local: Post http://e2e-sequence-kn-sequence-1-kn-channel.test-sequence-68mtm.svc.cluster.local: dial tcp 10.99.121.86:80: i/o timeout","stacktrace":"knative.dev/eventing/pkg/channel/fanout.(*Handler).dispatch\n\t/home/francesco/go/src/knative.dev/eventing/pkg/channel/fanout/fanout_handler.go:120\nknative.dev/eventing/pkg/channel/fanout.createReceiverFunction.func1.1\n\t/home/francesco/go/src/knative.dev/eventing/pkg/channel/fanout/fanout_handler.go:94"}

[vaikas]

I'm in the process of setting up a repro also. My PR is failing also:
https://github.com/knative/eventing/pull/2032
Though, not necessarily related :)

[slinkydeveloper]

This is the script i use to setup the minikube cluster: setup.zip

Sometime there is a known race condition when installing serving: in case it happens, repeat the serving installation command and then install manually eventing

[lionelvillard]

ah: https://knative.slack.com/archives/C9JP909F0/p1568723863013900

I had the same problem a while ago. I switched to kind

[slinkydeveloper]

noob question: what is kind? :) this one? https://github.com/kubernetes-sigs/kind

[lionelvillard]

yup: https://github.com/kubernetes-sigs/kind

here the script I use: https://github.com/lionelvillard/knative-examples/blob/master/hack/setup-knative-kind.sh

[slinkydeveloper]

I didn't managed to run my application on kind for other reasons.

Anyway, I tried to ping the kafka-ch-dispatcher service from inside the kafka-ch-dispatcher container and it just hangs:

kafka-ch-dispatcher   ClusterIP   10.108.90.101    <none>        80/TCP    113s

[root@kafka-ch-dispatcher-f5595b4fc-wvz6d /]# ping 10.108.90.101
PING 10.108.90.101 (10.108.90.101) 56(84) bytes of data.

And as expected, it doesn't work.

[kameshsampath]

I tired with v0.9.0 and minikube 1.4.0 and get the same issue

[konstantin-dzreev]

I'm getting a similar issue.

Knative release version
Knative Serving/Eventing: 0.8.0, 0.9.0
Kubernetes: 1.14, 1.16
Gloo: 0.21.1
Minikube: 1.4.0, 1.5.2
Ubuntu: 19.04

There is a very simple setup: a broker, a service and a trigger without filters. Once a message gets pushed to the broker, it reaches the service, but a reply event causes an error:

$ kubectl logs -f imc-dispatcher-dd84879d7-xbqrw

{"level":"info","ts":"2019-11-14T06:05:36.357Z","logger":"inmemorychannel-dispatcher.in-memory-channel-dispatcher","caller":"provisioners/message_receiver.go:140","msg":"Received request for default-kn2-trigger-kn-channel.default.svc.cluster.local","knative.dev/controller":"in-memory-channel-dispatcher"}
{"level":"info","ts":"2019-11-14T06:05:36.357Z","logger":"inmemorychannel-dispatcher.in-memory-channel-dispatcher","caller":"provisioners/message_receiver.go:147","msg":"Request mapped to channel: default/default-kn2-trigger-kn-channel","knative.dev/controller":"in-memory-channel-dispatcher"}
{"level":"info","ts":"2019-11-14T06:05:36.357Z","logger":"inmemorychannel-dispatcher.in-memory-channel-dispatcher","caller":"provisioners/message_dispatcher.go:112","msg":"Dispatching message to http://default-broker-filter.default.svc.cluster.local/triggers/default/conversion-m10e/e4d1bc76-0698-11ea-b70b-686d8c7fa7f5","knative.dev/controller":"in-memory-channel-dispatcher"}
{"level":"info","ts":"2019-11-14T06:05:36.358Z","logger":"inmemorychannel-dispatcher.in-memory-channel-dispatcher","caller":"provisioners/message_dispatcher.go:112","msg":"Dispatching message to http://default-broker-filter.default.svc.cluster.local/triggers/default/event-display/f6c6923b-0696-11ea-b70b-686d8c7fa7f5","knative.dev/controller":"in-memory-channel-dispatcher"}
{"level":"info","ts":"2019-11-14T06:05:37.335Z","logger":"inmemorychannel-dispatcher.in-memory-channel-dispatcher","caller":"provisioners/message_dispatcher.go:112","msg":"Dispatching message to http://default-kn2-ingress-kn-channel.default.svc.cluster.local","knative.dev/controller":"in-memory-channel-dispatcher"}
{"level":"error","ts":"2019-11-14T06:06:07.335Z","logger":"inmemorychannel-dispatcher.in-memory-channel-dispatcher","caller":"fanout/fanout_handler.go:121","msg":"Fanout had an error","knative.dev/controller":"in-memory-channel-dispatcher","error":"Failed to forward reply Post http://default-kn2-ingress-kn-channel.default.svc.cluster.local: dial tcp 10.110.251.33:80: i/o timeout","stacktrace":"github.com/knative/eventing/pkg/provisioners/fanout.(*Handler).dispatch\n\t/home/prow/go/src/github.com/knative/eventing/pkg/provisioners/fanout/fanout_handler.go:121\ngithub.com/knative/eventing/pkg/provisioners/fanout.createReceiverFunction.func1.1\n\t/home/prow/go/src/github.com/knative/eventing/pkg/provisioners/fanout/fanout_handler.go:95"}

[konstantin-dzreev]

P.S. Switching to kind mentioned above fixed the issue

[xphoniex]

I'm facing this very same issue:

{"level":"error","ts":"2019-12-28T16:15:59.944Z","logger":"inmemorychannel_dispatcher.in-memory-channel-dispatcher","caller":"fanout/fanout_handler.go:120","msg":"Fanout had an error","knative.dev/controller":"in-memory-channel-dispatcher","error":"failed to forward reply to http://test-sequence-kn-sequence-1-kn-channel.event-example.svc.cluster.local: Post http://test-sequence-kn-sequence-1-kn-channel.event-example.svc.cluster.local: dial tcp 10.110.73.157:80: i/o timeout","stacktrace":"knative.dev/eventing/pkg/channel/fanout.(*Handler).dispatch\n\t/home/prow/go/src/knative.dev/eventing/pkg/channel/fanout/fanout_handler.go:120\nknative.dev/eventing/pkg/channel/fanout.createReceiverFunction.func1.1\n\t/home/prow/go/src/knative.dev/eventing/pkg/channel/fanout/fanout_handler.go:94"}

There's a strange networking bug in minikube where a pod cannot reach itself through a service. I think that's what's happening here. (For reference https://github.com/kubernetes/minikube/issues/2460 & https://github.com/kubernetes/minikube/issues/1568)

[slinkydeveloper]

FYI I've created a little script to run kind + knative: https://gist.github.com/slinkydeveloper/193b44e8b15492c8584863411e1bc53e

[prasanthpagolu]

Thanks @slinkydeveloper I had to make minor changes for newer version and also for few bugs that I encountered, attaching it here if it may help others:
kind-knative.sh.zip

[iyinoluwaayoola]

I'm facing this very same issue:

{"level":"error","ts":"2019-12-28T16:15:59.944Z","logger":"inmemorychannel_dispatcher.in-memory-channel-dispatcher","caller":"fanout/fanout_handler.go:120","msg":"Fanout had an error","knative.dev/controller":"in-memory-channel-dispatcher","error":"failed to forward reply to http://test-sequence-kn-sequence-1-kn-channel.event-example.svc.cluster.local: Post http://test-sequence-kn-sequence-1-kn-channel.event-example.svc.cluster.local: dial tcp 10.110.73.157:80: i/o timeout","stacktrace":"knative.dev/eventing/pkg/channel/fanout.(*Handler).dispatch\n\t/home/prow/go/src/knative.dev/eventing/pkg/channel/fanout/fanout_handler.go:120\nknative.dev/eventing/pkg/channel/fanout.createReceiverFunction.func1.1\n\t/home/prow/go/src/knative.dev/eventing/pkg/channel/fanout/fanout_handler.go:94"}

There's a strange networking bug in minikube where a pod cannot reach itself through a service. I think that's what's happening here. (For reference kubernetes/minikube#2460 & kubernetes/minikube#1568)

Indeed it's related, this fixed it for me. https://github.com/kubernetes/minikube/issues/1568#issuecomment-308674617

[slinkydeveloper]

This fixed it for me: https://github.com/kubernetes/minikube/issues/1568#issuecomment-308674617

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[slinkydeveloper]

/reopen

[knative-prow-robot]

@slinkydeveloper: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.