Eventing: [BUG] {"error":"error sending cloudevent: 404 Not Found"}
Describe the bug
I followed this guide https://knative.dev/docs/eventing/broker-trigger/ and https://knative.dev/docs/eventing/samples/kafka/channel/index.html
with both I get:
$ kubectl run -i --tty --rm debug --image=yauritux/busybox-curl --restart=Never --
If you don't see a command prompt, try pressing enter.
/home # curl -v "http://default-broker.default.svc.cluster.local/" \
> -X POST \
> -H "X-B3-Flags: 1" \
> -H "CE-SpecVersion: 0.2" \
> -H "CE-Type: dev.knative.foo.bar" \
> -H "CE-Time: 2018-04-05T03:56:24Z" \
> -H "CE-ID: 45a8b444-3213-4758-be3f-540bf93f85ff" \
> -H "CE-Source: dev.knative.example" \
> -H 'Content-Type: application/json' \
> -d '{ "much": "wow" }'
* About to connect() to default-broker.default.svc.cluster.local port 80 (#0)
* Trying 10.4.11.119...
* Adding handle: conn: 0x1180eda8
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x1180eda8) send_pipe: 1, recv_pipe: 0
* Connected to default-broker.default.svc.cluster.local (10.4.11.119) port 80 (#0)
> POST / HTTP/1.1
> User-Agent: curl/7.30.0
> Host: default-broker.default.svc.cluster.local
> Accept: */*
> X-B3-Flags: 1
> CE-SpecVersion: 0.2
> CE-Type: dev.knative.foo.bar
> CE-Time: 2018-04-05T03:56:24Z
> CE-ID: 45a8b444-3213-4758-be3f-540bf93f85ff
> CE-Source: dev.knative.example
> Content-Type: application/json
> Content-Length: 17
>
* upload completely sent off: 17 out of 17 bytes
< HTTP/1.1 400 Bad Request
< date: Mon, 04 Nov 2019 13:54:33 GMT
< content-length: 51
< content-type: text/plain; charset=utf-8
< x-envoy-upstream-service-time: 24
* Server envoy is not blacklisted
< server: envoy
<
* Connection #0 to host default-broker.default.svc.cluster.local left intact
{"error":"error sending cloudevent: 404 Not Found"}
on the broker ingress:
$ kubectl logs default-broker-ingress-54d485746c-9wrmt ingress
2019/11/04 13:56:49 Registering 1 clients
2019/11/04 13:56:49 Registering 0 informer factories
2019/11/04 13:56:49 Registering 0 informers
{"level":"info","ts":"2019-11-04T13:56:49.537Z","caller":"logging/config.go:108","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n \"level\": \"info\",\n \"development\": false,\n \"outputPaths\": [\"stdout\"],\n \"errorOutputPaths\": [\"stderr\"],\n \"encoding\": \"json\",\n \"encoderConfig\": {\n \"timeKey\": \"ts\",\n \"levelKey\": \"level\",\n \"nameKey\": \"logger\",\n \"callerKey\": \"caller\",\n \"messageKey\": \"msg\",\n \"stacktraceKey\": \"stacktrace\",\n \"lineEnding\": \"\",\n \"levelEncoder\": \"\",\n \"timeEncoder\": \"iso8601\",\n \"durationEncoder\": \"\",\n \"callerEncoder\": \"\"\n }\n}\n"}
{"level":"info","ts":"2019-11-04T13:56:49.537Z","caller":"logging/config.go:109","msg":"Logging level set to info"}
{"level":"info","ts":"2019-11-04T13:56:49.562Z","logger":"broker_ingress","caller":"ingress/main.go:103","msg":"Starting the Broker Ingress","commit":"5fe2b0a"}
{"level":"info","ts":"2019-11-04T13:56:49.569Z","logger":"broker_ingress","caller":"metrics/exporter.go:94","msg":"Flushing the existing exporter before setting up the new exporter.","commit":"5fe2b0a"}
{"level":"info","ts":"2019-11-04T13:56:49.569Z","logger":"broker_ingress","caller":"metrics/prometheus_exporter.go:37","msg":"Created Opencensus Prometheus exporter with config: &{knative.dev/eventing broker_ingress prometheus 5000000000 9090 false false { }}. Start the server for Prometheus exporter.","commit":"5fe2b0a"}
{"level":"info","ts":"2019-11-04T13:56:49.570Z","logger":"broker_ingress","caller":"metrics/exporter.go:103","msg":"Successfully updated the metrics exporter; old config: <nil>; new config &{knative.dev/eventing broker_ingress prometheus 5000000000 9090 false false { }}","commit":"5fe2b0a"}
{"level":"info","ts":"2019-11-04T13:56:49.570Z","logger":"broker_ingress","caller":"configmap/store.go:157","msg":"tracing-config config \"config-tracing\" config was added or updated: &config.Config{Backend:\"none\", ZipkinEndpoint:\"\", StackdriverProjectID:\"\", Debug:false, SampleRate:0.1}","commit":"5fe2b0a"}
{"level":"info","ts":"2019-11-04T13:56:49.663Z","logger":"broker_ingress","caller":"ingress/main.go:169","msg":"Starting informers.","commit":"5fe2b0a"}
{"level":"warn","ts":1572875810.3738859,"logger":"fallback","caller":"http/transport.go:502","msg":"got an error from receiver fn","error":"error sending cloudevent: 404 Not Found"}
{"level":"warn","ts":1572875810.3740385,"logger":"fallback","caller":"http/transport.go:594","msg":"error returned from invokeReceiver","error":"error sending cloudevent: 404 Not Found"}
based on this yaml;
---
apiVersion: v1
kind: ConfigMap
metadata:
name: config-autoscaler
namespace: knative-serving
data:
enable-scale-to-zero: "false"
---
apiVersion: v1
kind: ConfigMap
metadata:
name: default-ch-webhook
namespace: knative-eventing
data:
default-ch-config: |
clusterDefault:
apiVersion: messaging.knative.dev/v1alpha1
kind: InMemoryChannel
---
apiVersion: v1
kind: ConfigMap
metadata:
name: config-domain
namespace: knative-serving
data:
svc.cluster.local: ""
---
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: my-service
namespace: default
spec:
template:
spec:
containers:
- # This corresponds to
# https://github.com/knative/eventing-contrib/blob/v0.2.1/cmd/message_dumper/dumper.go.
image: gcr.io/knative-releases/github.com/knative/eventing-sources/cmd/message_dumper@sha256:ab5391755f11a5821e7263686564b3c3cd5348522f5b31509963afb269ddcd63
---
apiVersion: eventing.knative.dev/v1alpha1
kind: Trigger
metadata:
name: my-service-trigger
namespace: default
spec:
broker: default # Defaulted by the Webhook.
filter:
attributes:
type: dev.knative.foo.bar
subscriber:
ref:
apiVersion: serving.knative.dev/v1
kind: Service
name: my-service
---
apiVersion: eventing.knative.dev/v1alpha1
kind: Broker
metadata:
name: default
spec:
channelTemplateSpec:
apiVersion: messaging.knative.dev/v1alpha1
kind: InMemoryChannel
---
apiVersion: sources.eventing.knative.dev/v1alpha1
kind: ContainerSource
metadata:
name: heartbeats-sender
spec:
template:
spec:
containers:
- image: index.docker.io/daisyycguo/heartbeats-6790335e994243a8d3f53b967cdd6398
name: heartbeats-sender
args:
- --sink=http://default-broker.default.svc.cluster.local
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
sink:
apiVersion: eventing.knative.dev/v1alpha1
kind: Broker
name: default
Expected behavior
The cloudevent should pass through the broker to the service my-service
To Reproduce
Install v0.9.0 or v0.10.0 and apply the above yaml
Knative release version
Same issue with v0.9.0 or v0.10.0 and InMemory and Kafka
grebois
👍2
All 18 comments
What is the YAML of the Broker? kubectl get broker default -n default -o yaml
Do you have Istio installed?
Harwayne
on 4 Nov 2019
Can you change the ce specversion to be 0.3 and try again?
nachocano
on 5 Nov 2019
@Harwayne sure, same problem with istio 1.3.3 btw
$ helm ls
NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
istio 2 Mon Nov 4 14:46:59 2019 DEPLOYED istio-1.3.4 1.3.4 istio-system
istio-init 2 Mon Nov 4 14:46:57 2019 DEPLOYED istio-init-1.3.4 1.3.4 istio-system
$ kubectl get broker default -n default -o yaml
apiVersion: eventing.knative.dev/v1alpha1
kind: Broker
metadata:
annotations:
eventing.knative.dev/creator: system:serviceaccount:knative-eventing:eventing-controller
eventing.knative.dev/lastModifier: system:serviceaccount:knative-eventing:eventing-controller
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"eventing.knative.dev/v1alpha1","kind":"Broker","metadata":{"annotations":{},"name":"default","namespace":"default"},"spec":{"channelTemplateSpec":{"apiVersion":"messaging.knative.dev/v1alpha1","kind":"InMemoryChannel"}}}
creationTimestamp: "2019-11-04T15:47:05Z"
generation: 1
labels:
eventing.knative.dev/namespaceInjected: "true"
name: default
namespace: default
resourceVersion: "41142"
selfLink: /apis/eventing.knative.dev/v1alpha1/namespaces/default/brokers/default
uid: 5a06dbd4-ff1a-11e9-a0b2-42010a9c0086
spec:
channelTemplateSpec:
apiVersion: messaging.knative.dev/v1alpha1
kind: InMemoryChannel
status:
IngressChannel:
apiVersion: messaging.knative.dev/v1alpha1
kind: InMemoryChannel
name: default-kne-ingress
namespace: default
address:
hostname: default-broker.default.svc.cluster.local
url: http://default-broker.default.svc.cluster.local
conditions:
- lastTransitionTime: "2019-11-04T15:47:06Z"
status: "True"
type: Addressable
- lastTransitionTime: "2019-11-04T15:47:13Z"
status: "True"
type: FilterReady
- lastTransitionTime: "2019-11-04T15:47:07Z"
status: "True"
type: IngressChannelReady
- lastTransitionTime: "2019-11-04T15:47:10Z"
status: "True"
type: IngressReady
- lastTransitionTime: "2019-11-04T15:47:07Z"
status: "True"
type: IngressSubscriptionReady
- lastTransitionTime: "2019-11-04T15:47:13Z"
status: "True"
type: Ready
- lastTransitionTime: "2019-11-04T15:47:05Z"
status: "True"
type: TriggerChannelReady
triggerChannel:
apiVersion: messaging.knative.dev/v1alpha1
kind: InMemoryChannel
name: default-kne-trigger
namespace: default
@nachocano same problem with 0.3 and 1.0
grebois
on 5 Nov 2019
I narrowed down the problem to istio being enabled on the namespace, when I disable it I stop getting the 404s and everything works as expected :/?
apiVersion: v1
kind: Namespace
metadata:
name: sandbox
labels:
istio-injection: disable
knative-eventing-injection: enabled
With istio:
$ kubectl --namespace sandbox attach curl -it
Defaulting container name to curl.
Use 'kubectl describe pod/curl -n sandbox' to see all of the containers in this pod.
If you don't see a command prompt, try pressing enter.
[ root@curl:/ ]$ curl -v "http://default-broker.sandbox.svc.cluster.local/" \
> -X POST \
> -H "X-B3-Flags: 1" \
> -H "CE-SpecVersion: 0.2" \
> -H "CE-Type: dev.knative.foo.bar" \
> -H "CE-Time: 2018-04-05T03:56:24Z" \
> -H "CE-ID: 45a8b444-3213-4758-be3f-540bf93f85ff" \
> -H "CE-Source: dev.knative.example" \
> -H 'Content-Type: application/json' \
> -d '{ "much": "wow" }'
> POST / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: default-broker.sandbox.svc.cluster.local
> Accept: */*
> X-B3-Flags: 1
> CE-SpecVersion: 0.2
> CE-Type: dev.knative.foo.bar
> CE-Time: 2018-04-05T03:56:24Z
> CE-ID: 45a8b444-3213-4758-be3f-540bf93f85ff
> CE-Source: dev.knative.example
> Content-Type: application/json
> Content-Length: 17
>
< HTTP/1.1 400 Bad Request
< date: Wed, 06 Nov 2019 14:37:45 GMT
< content-length: 51
< content-type: text/plain; charset=utf-8
< x-envoy-upstream-service-time: 13
< server: envoy
<
{"error":"error sending cloudevent: 404 Not Found"}
Without istio:
$ kubectl --namespace sandbox attach curl -it
Defaulting container name to curl.
Use 'kubectl describe pod/curl -n sandbox' to see all of the containers in this pod.
If you don't see a command prompt, try pressing enter.
[ root@curl:/ ]$ curl -v "http://default-broker.sandbox.svc.cluster.local/" \
> -X POST \
> -H "X-B3-Flags: 1" \
> -H "CE-SpecVersion: 0.2" \
> -H "CE-Type: dev.knative.foo.bar" \
> -H "CE-Time: 2018-04-05T03:56:24Z" \
> -H "CE-ID: 45a8b444-3213-4758-be3f-540bf93f85ff" \
> -H "CE-Source: dev.knative.example" \
> -H 'Content-Type: application/json' \
> -d '{ "much": "wow" }'
> POST / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: default-broker.sandbox.svc.cluster.local
> Accept: */*
> X-B3-Flags: 1
> CE-SpecVersion: 0.2
> CE-Type: dev.knative.foo.bar
> CE-Time: 2018-04-05T03:56:24Z
> CE-ID: 45a8b444-3213-4758-be3f-540bf93f85ff
> CE-Source: dev.knative.example
> Content-Type: application/json
> Content-Length: 17
>
< HTTP/1.1 202 Accepted
< Content-Length: 0
< Date: Wed, 06 Nov 2019 14:39:25 GMT
<
[ root@curl:/ ]$
@nachocano @Harwayne any ideas?
grebois
on 6 Nov 2019
If it's not too difficult, could you try seeing if you can have two pods communicate in a namespace with Istio enabled to see if it's only related to Broker, or Istio issue in general?
vaikas
on 8 Nov 2019
@vaikas of course;
[ root@curl:/ ]$ curl -vvvv http://default-kne-ingress-kn-channel.sandbox.svc.cluster.local
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: default-kne-ingress-kn-channel.sandbox.svc.cluster.local
> Accept: */*
>
< HTTP/1.1 404 Not Found
< date: Fri, 08 Nov 2019 19:02:30 GMT
< server: envoy
< content-length: 0
<
[ root@curl:/ ]$ curl -vvvvv default-broker
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: default-broker
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< date: Fri, 08 Nov 2019 19:02:56 GMT
< content-length: 0
< x-envoy-upstream-service-time: 1
< server: envoy
<
[ root@curl:/ ]$ curl -vvvvv imc-dispatcher.knative-eventing.svc.cluster.local
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: imc-dispatcher.knative-eventing.svc.cluster.local
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< date: Fri, 08 Nov 2019 19:06:27 GMT
< content-length: 0
< x-envoy-upstream-service-time: 0
< server: envoy
<
$ curl -vvvvvv hello-display.event-example
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: hello-display.event-example
> Accept: */*
>
< HTTP/1.1 404 Not Found
< x-powered-by: Express
< content-security-policy: default-src 'none'
< x-content-type-options: nosniff
< content-type: text/html; charset=utf-8
< content-length: 139
< date: Fri, 08 Nov 2019 19:13:24 GMT
< x-envoy-upstream-service-time: 27
< server: envoy
<
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot GET /</pre>
</body>
</html>
all work even with different namespaces
grebois
on 8 Nov 2019
One more interesting thing, it also works from an istio enabled namespace to a non istio namespace:
$ curl -v "default-broker.event-example/" \
> -X POST \
> -H "X-B3-Flags: 1" \
> -H "CE-SpecVersion: 0.2" \
> -H "CE-Type: dev.knative.foo.bar" \
> -H "CE-Time: 2018-04-05T03:56:24Z" \
> -H "CE-ID: 45a8b444-3213-4758-be3f-540bf93f85ff" \
> -H "CE-Source: dev.knative.example" \
> -H 'Content-Type: application/json' \
> -d '{ "much": "wow" }'
> POST / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: default-broker.event-example
> Accept: */*
> X-B3-Flags: 1
> CE-SpecVersion: 0.2
> CE-Type: dev.knative.foo.bar
> CE-Time: 2018-04-05T03:56:24Z
> CE-ID: 45a8b444-3213-4758-be3f-540bf93f85ff
> CE-Source: dev.knative.example
> Content-Type: application/json
> Content-Length: 17
>
< HTTP/1.1 202 Accepted
< content-length: 0
< date: Fri, 08 Nov 2019 19:17:27 GMT
< x-envoy-upstream-service-time: 14
< server: envoy
<
There are also some errors in the broker ingress pod:
{"level":"error","ts":1573247491.782166,"logger":"fallback","caller":"http/transport.go:586","msg":"failed to get non-nil event from MessageToEvent","stacktrace":"knative.dev/eventing/vendor/github.com/cloudevents/sdk-go/pkg/cloudevents/transport/http.(*Transport).ServeHTTP\n\t/home/prow/go/src/knative.dev/eventing/vendor/github.com/cloudevents/sdk-go/pkg/cloudevents/transport/http/transport.go:586\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2375\nknative.dev/eventing/vendor/go.opencensus.io/plugin/ochttp.(*Handler).ServeHTTP\n\t/home/prow/go/src/knative.dev/eventing/vendor/go.opencensus.io/plugin/ochttp/server.go:86\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2774\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1878"}
{"level":"error","ts":1573247496.3941765,"logger":"fallback","caller":"http/transport.go:586","msg":"failed to get non-nil event from MessageToEvent","stacktrace":"knative.dev/eventing/vendor/github.com/cloudevents/sdk-go/pkg/cloudevents/transport/http.(*Transport).ServeHTTP\n\t/home/prow/go/src/knative.dev/eventing/vendor/github.com/cloudevents/sdk-go/pkg/cloudevents/transport/http/transport.go:586\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2375\nknative.dev/eventing/vendor/go.opencensus.io/plugin/ochttp.(*Handler).ServeHTTP\n\t/home/prow/go/src/knative.dev/eventing/vendor/go.opencensus.io/plugin/ochttp/server.go:86\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2774\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1878"}
W1108 21:12:03.775204 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 1758298 (1760660)
{"level":"warn","ts":1573247540.045236,"logger":"fallback","caller":"http/transport.go:502","msg":"got an error from receiver fn","error":"error sending cloudevent: 404 Not Found"}
{"level":"warn","ts":1573247540.045319,"logger":"fallback","caller":"http/transport.go:594","msg":"error returned from invokeReceiver","error":"error sending cloudevent: 404 Not Found"}
could this be related to https://github.com/cloudevents/sdk-go/issues/124 ?
grebois
on 8 Nov 2019
Is this resolved or still an issue?
akashrv
on 5 Dec 2019
@akashrv I'm still having this issue with the latest Istio 1.4.1 and Knative 0.10
grebois
on 5 Dec 2019
hi grebois, not sure if this will help but I had a similar issue, with similar error message type and it was because I did not have the Istio cluster local gateway enabled, which I think you need to be able to sink to a Knative Service.
josiemundi
on 9 Dec 2019
Hi @josiemundi actually that was one of my first thoughts, and I even deleted the default local gateway and created my own but didn't work out, would you try the yaml's I posted and try them in your setup just to double check?
grebois
on 10 Dec 2019
hi @grebois I tried your example and it works for me (both in istio labelled namespace and non-istio labelled). Do you have the istio-system pod 'cluster-local-gateway' up? As per this example under 'Updating your install to use cluster local gateway).
It should look like something like this if you kubectl get pods -n istio-system:
istio-system cluster-local-gateway-xxxxxx
istio-system istio-ingressgateway-xxxxxx
istio-system istio-pilot-xxxxx
I ran the yamls in a new namespace so changed the yamls a little. I also didn't include the type filter in your trigger, just left that out. Also, used the default broker that is initiated once you label the namespace with knative-eventing. Perhaps you can just try deploying the source, trigger and service yamls and see if that works.
So... try running this:
# Namespace for sample application with eventing enabled
apiVersion: v1
kind: Namespace
metadata:
name: knative-eventing-heartbeats-source
labels:
knative-eventing-injection: enabled
#containersource
apiVersion: sources.eventing.knative.dev/v1alpha1
kind: ContainerSource
metadata:
name: heartbeats-sender
namespace: knative-eventing-heartbeats-source
spec:
template:
spec:
containers:
- image: index.docker.io/daisyycguo/heartbeats-6790335e994243a8d3f53b967cdd6398
name: heartbeats-sender
args:
- --sink=http://default-broker.knative-eventing-heartbeats-source.svc.cluster.local
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
sink:
apiVersion: eventing.knative.dev/v1alpha1
kind: Broker
name: default
#trigger
apiVersion: eventing.knative.dev/v1alpha1
kind: Trigger
metadata:
name: my-service-trigger
namespace: knative-eventing-heartbeats-source
spec:
broker: default # Defaulted by the Webhook.
subscriber:
ref:
apiVersion: serving.knative.dev/v1
kind: Service
name: my-service
#service
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: my-service
namespace: knative-eventing-heartbeats-source
spec:
template:
spec:
containers:
- # This corresponds to
# https://github.com/knative/eventing-contrib/blob/v0.2.1/cmd/message_dumper/dumper.go.
image: gcr.io/knative-releases/github.com/knative/eventing-sources/cmd/message_dumper@sha256:ab5391755f11a5821e7263686564b3c3cd5348522f5b31509963afb269ddcd63
Then with the filter added into the trigger on type I tried it...
curl:
curl -v "http://default-broker.knative-eventing-heartbeats-source.svc.cluster.local" \
> -X POST \
> -H "X-B3-Flags: 1" \
> -H "CE-SpecVersion: 0.2" \
> -H "CE-Type: dev.knative.foo.bar" \
> -H "CE-Time: 2018-04-05T03:56:24Z" \
> -H "CE-ID: 45a8b444-3213-4758-be3f-540bf93f85ff" \
> -H "CE-Source: dev.knative.example" \
> -H 'Content-Type: application/json' \
> -d '{ "much": "wow" }'
> POST / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: default-broker.knative-eventing-heartbeats-source.svc.cluster.local
> Accept: */*
> X-B3-Flags: 1
> CE-SpecVersion: 0.2
> CE-Type: dev.knative.foo.bar
> CE-Time: 2018-04-05T03:56:24Z
> CE-ID: 45a8b444-3213-4758-be3f-540bf93f85ff
> CE-Source: dev.knative.example
> Content-Type: application/json
> Content-Length: 17
>
< HTTP/1.1 202 Accepted
< Content-Length: 0
< Date: Wed, 11 Dec 2019 11:18:44 GMT
Let me know if that works for you.
josiemundi
on 11 Dec 2019
@josiemundi Thanks a lot for the follow up, this is what's on my istio-system namespace;
$ kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
cluster-local-gateway-6bf78dcf6d-bj44q 1/1 Running 0 2d19h
heapster-55d9b967c9-ztdbt 2/2 Running 0 2d19h
istio-citadel-66ddfd755-k8vbf 1/1 Running 0 2d19h
istio-galley-5f49858479-rvkh6 1/1 Running 0 2d19h
istio-ingressgateway-74747c4b5b-t2xd8 1/1 Running 0 2d19h
istio-init-crd-10-1.4.0-8qlm6 0/1 Completed 0 2d19h
istio-init-crd-11-1.4.0-r4bq4 0/1 Completed 0 2d19h
istio-init-crd-14-1.4.0-48b5v 0/1 Completed 0 2d19h
istio-pilot-75d8b87cbf-72786 2/2 Running 0 2d19h
istio-policy-59499c9447-dwql7 2/2 Running 0 2d19h
istio-sidecar-injector-7984b6f548-p8b8j 1/1 Running 0 2d19h
istio-telemetry-5c6bd8bb76-2fjqz 2/2 Running 0 2d19h
istio-tracing-56c7f85df7-h48lv 1/1 Running 0 2d19h
kiali-7b5c8f79d8-6992l 1/1 Running 0 2d19h
prometheus-74d8b55f54-lwsf8 1/1 Running 0 2d19h
zipkin-7684997f4b-hfjw2 1/1 Running 0 2d23h
The following yaml;
# Namespace for sample application with eventing enabled
apiVersion: v1
kind: Namespace
metadata:
name: knative-eventing-heartbeats-source
labels:
knative-eventing-injection: enabled
---
#containersource
apiVersion: sources.eventing.knative.dev/v1alpha1
kind: ContainerSource
metadata:
name: heartbeats-sender
namespace: knative-eventing-heartbeats-source
spec:
template:
spec:
containers:
- image: index.docker.io/daisyycguo/heartbeats-6790335e994243a8d3f53b967cdd6398
name: heartbeats-sender
args:
- --sink=http://default-broker.knative-eventing-heartbeats-source.svc.cluster.local
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
sink:
apiVersion: eventing.knative.dev/v1alpha1
kind: Broker
name: default
---
#trigger
apiVersion: eventing.knative.dev/v1alpha1
kind: Trigger
metadata:
name: my-service-trigger
namespace: knative-eventing-heartbeats-source
spec:
broker: default # Defaulted by the Webhook.
subscriber:
ref:
apiVersion: serving.knative.dev/v1
kind: Service
name: my-service
---
#service
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: my-service
namespace: knative-eventing-heartbeats-source
spec:
template:
spec:
containers:
- # This corresponds to
# https://github.com/knative/eventing-contrib/blob/v0.2.1/cmd/message_dumper/dumper.go.
image: gcr.io/knative-releases/github.com/knative/eventing-sources/cmd/message_dumper@sha256:ab5391755f11a5821e7263686564b3c3cd5348522f5b31509963afb269ddcd63
works as you mention;
$ kubectl -n knative-eventing-heartbeats-source run -i --tty --rm debug --image=yauritux/busybox-curl --restart=Never --
If you don't see a command prompt, try pressing enter.
/home #
/home #
/home # curl -v "http://default-broker.knative-eventing-heartbeats-source.svc.cluster.local" \
> -X POST \
> -H "X-B3-Flags: 1" \
> -H "CE-SpecVersion: 0.2" \
> -H "CE-Type: dev.knative.foo.bar" \
> -H "CE-Time: 2018-04-05T03:56:24Z" \
> -H "CE-ID: 45a8b444-3213-4758-be3f-540bf93f85ff" \
> -H "CE-Source: dev.knative.example" \
> -H 'Content-Type: application/json' \
> -d '{ "much": "wow" }'
* About to connect() to default-broker.knative-eventing-heartbeats-source.svc.cluster.local port 80 (#0)
* Trying 10.4.13.2...
* Adding handle: conn: 0x10a2de20
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x10a2de20) send_pipe: 1, recv_pipe: 0
* Connected to default-broker.knative-eventing-heartbeats-source.svc.cluster.local (10.4.13.2) port 80 (#0)
> POST / HTTP/1.1
> User-Agent: curl/7.30.0
> Host: default-broker.knative-eventing-heartbeats-source.svc.cluster.local
> Accept: */*
> X-B3-Flags: 1
> CE-SpecVersion: 0.2
> CE-Type: dev.knative.foo.bar
> CE-Time: 2018-04-05T03:56:24Z
> CE-ID: 45a8b444-3213-4758-be3f-540bf93f85ff
> CE-Source: dev.knative.example
> Content-Type: application/json
> Content-Length: 17
>
* upload completely sent off: 17 out of 17 bytes
< HTTP/1.1 202 Accepted
< Content-Length: 0
< Date: Thu, 12 Dec 2019 08:47:23 GMT
<
* Connection #0 to host default-broker.knative-eventing-heartbeats-source.svc.cluster.local left intact
/home # exit
pod "debug" deleted
but as soon as I enable istio, I get:
# curl -v "http://default-broker.knative-eventing-heartbeats-source.svc.cluster.local" \
> -X POST \
> -H "X-B3-Flags: 1" \
> -H "CE-SpecVersion: 0.2" \
> -H "CE-Type: dev.knative.foo.bar" \
> -H "CE-Time: 2018-04-05T03:56:24Z" \
> -H "CE-ID: 45a8b444-3213-4758-be3f-540bf93f85ff" \
> -H "CE-Source: dev.knative.example" \
> -H 'Content-Type: application/json' \
> -d '{ "much": "wow" }'
* About to connect() to default-broker.knative-eventing-heartbeats-source.svc.cluster.local port 80 (#0)
* Trying 10.4.11.156...
* Adding handle: conn: 0x1029ae20
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x1029ae20) send_pipe: 1, recv_pipe: 0
* Connected to default-broker.knative-eventing-heartbeats-source.svc.cluster.local (10.4.11.156) port 80 (#0)
> POST / HTTP/1.1
> User-Agent: curl/7.30.0
> Host: default-broker.knative-eventing-heartbeats-source.svc.cluster.local
> Accept: */*
> X-B3-Flags: 1
> CE-SpecVersion: 0.2
> CE-Type: dev.knative.foo.bar
> CE-Time: 2018-04-05T03:56:24Z
> CE-ID: 45a8b444-3213-4758-be3f-540bf93f85ff
> CE-Source: dev.knative.example
> Content-Type: application/json
> Content-Length: 17
>
* upload completely sent off: 17 out of 17 bytes
< HTTP/1.1 400 Bad Request
< date: Thu, 12 Dec 2019 09:08:40 GMT
< content-length: 51
< content-type: text/plain; charset=utf-8
< x-envoy-upstream-service-time: 22
* Server envoy is not blacklisted
< server: envoy
<
* Connection #0 to host default-broker.knative-eventing-heartbeats-source.svc.cluster.local left intact
{"error":"error sending cloudevent: 404 Not Found"}
this is the final yaml:
# Namespace for sample application with eventing enabled
apiVersion: v1
kind: Namespace
metadata:
name: knative-eventing-heartbeats-source
labels:
knative-eventing-injection: enabled
istio-injection: enabled
---
#containersource
apiVersion: sources.eventing.knative.dev/v1alpha1
kind: ContainerSource
metadata:
name: heartbeats-sender
namespace: knative-eventing-heartbeats-source
spec:
template:
spec:
containers:
- image: index.docker.io/daisyycguo/heartbeats-6790335e994243a8d3f53b967cdd6398
name: heartbeats-sender
args:
- --sink=http://default-broker.knative-eventing-heartbeats-source.svc.cluster.local
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
sink:
apiVersion: eventing.knative.dev/v1alpha1
kind: Broker
name: default
---
#trigger
apiVersion: eventing.knative.dev/v1alpha1
kind: Trigger
metadata:
name: my-service-trigger
namespace: knative-eventing-heartbeats-source
spec:
broker: default # Defaulted by the Webhook.
subscriber:
ref:
apiVersion: serving.knative.dev/v1
kind: Service
name: my-service
---
#service
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: my-service
namespace: knative-eventing-heartbeats-source
spec:
template:
spec:
containers:
- # This corresponds to
# https://github.com/knative/eventing-contrib/blob/v0.2.1/cmd/message_dumper/dumper.go.
image: gcr.io/knative-releases/github.com/knative/eventing-sources/cmd/message_dumper@sha256:ab5391755f11a5821e7263686564b3c3cd5348522f5b31509963afb269ddcd63
I'm currently running istio 1.4.0
grebois
on 12 Dec 2019
@grebois can you try just running the original namespace yaml without Istio enabled and then run it. Then manually label the namespace with:
kubectl label namespace knative-eventing-heartbeats-source istio-injection=enabled
I also had trouble when I tried deploying the yaml with Istio enabled but when I manually do it, it works fine.
josiemundi
on 12 Dec 2019
@grebois I just tried this again with your original namespace yaml and it worked. Here below a check that the Istio injection was enabled and the result below:
kubectl get namespace -L istio-injection
NAME STATUS AGE ISTIO-INJECTION
default Active 17m
docker Active 15m
istio-system Active 9m24s
knative-eventing Active 8m57s
knative-eventing-heartbeats-source Active 6m44s enabled
knative-monitoring Active 8m55s
knative-serving Active 8m59s enabled
Result:
[ root@curl:/ ]$ curl -v "http://default-broker.knative-eventing-heartbeats-source.svc.cluster.local" \
> -X POST \
> -H "X-B3-Flags: 1" \
> -H "CE-SpecVersion: 0.2" \
> -H "CE-Type: dev.knative.foo.bar" \
> -H "CE-Time: 2018-04-05T03:56:24Z" \
> -H "CE-ID: 45a8b444-3213-4758-be3f-540bf93f85ff" \
> -H "CE-Source: dev.knative.example" \
> -H 'Content-Type: application/json' \
> -d '{ "much": "wow" }'
> POST / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: default-broker.knative-eventing-heartbeats-source.svc.cluster.local
> Accept: */*
> X-B3-Flags: 1
> CE-SpecVersion: 0.2
> CE-Type: dev.knative.foo.bar
> CE-Time: 2018-04-05T03:56:24Z
> CE-ID: 45a8b444-3213-4758-be3f-540bf93f85ff
> CE-Source: dev.knative.example
> Content-Type: application/json
> Content-Length: 17
>
< HTTP/1.1 202 Accepted
< Content-Length: 0
< Date: Thu, 12 Dec 2019 20:40:31 GMT
The only difference I can see is that I am using the knative Istio-lean.yaml and the cluster-local-gateway only, I could try it with full Istio.
josiemundi
on 12 Dec 2019
Hi @josiemundi, are you sure the istio proxy is running? I create a new cluster, installed istio using https://knative.dev/docs/install/installing-istio/#installing-istio-with-sidecar-injection and afterwards created the cluster local gateway with https://knative.dev/docs/install/installing-istio/#updating-your-install-to-use-cluster-local-gateway, but when I apply the yaml, only the queue-proxy is there;
$ kubectl get pods -n knative-eventing-heartbeats-source
NAME READY STATUS RESTARTS AGE
containersource-heartbeats-0b79b933-1ff1-11ea-a240-42010a959bsx 1/1 Running 0 18s
default-broker-filter-c6654bccf-lpsct 1/1 Running 0 17s
default-broker-ingress-f8684f644-mqxvs 1/1 Running 0 17s
my-service-w7brm-deployment-59978ffd58-4vlh4 2/2 Running 0 17s
$ kubectl delete --all pods -n knative-eventing-heartbeats-source
pod "containersource-heartbeats-0b79b933-1ff1-11ea-a240-42010a959bsx" deleted
pod "default-broker-filter-c6654bccf-lpsct" deleted
pod "default-broker-ingress-f8684f644-mqxvs" deleted
pod "my-service-w7brm-deployment-59978ffd58-4vlh4" deleted
$ kubectl get pods -n knative-eventing-heartbeats-source
NAME READY STATUS RESTARTS AGE
containersource-heartbeats-0b79b933-1ff1-11ea-a240-42010a9nfmkw 1/1 Running 0 40s
default-broker-filter-c6654bccf-g5xq8 1/1 Running 0 39s
default-broker-ingress-f8684f644-wckqh 1/1 Running 0 39s
my-service-w7brm-deployment-59978ffd58-dr4x6 2/2 Running 0 39s
$ kubectl get namespace -L istio-injection
NAME STATUS AGE ISTIO-INJECTION
default Active 16m
istio-system Active 9m56s disabled
knative-eventing Active 7m24s
knative-eventing-heartbeats-source Active 94s enabled
knative-serving Active 7m54s enabled
kube-node-lease Active 16m
kube-public Active 16m
kube-system Active 16m
$ kubectl logs my-service-w7brm-deployment-59978ffd58-dr4x6 -n knative-eventing-heartbeats-source
Error from server (BadRequest): a container name must be specified for pod my-service-w7brm-deployment-59978ffd58-dr4x6, choose one of: [user-container queue-proxy]
👀1
😕1
This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.
![github-actions[bot] picture](https://avatars.githubusercontent.com/in/15368?v=4&s=40)
github-actions[bot]
on 25 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
lionelvillard
路
12Comments
antoineco
路
11Comments
nachocano
路
11Comments
capri-xiyue
路
8Comments
rhuss
路
4Comments