Eventing: Make Knative eventing channels more serverless and scalable

[aslom]

Problem
Today channels do not scale in response to number of events: if channel is not used it occupies resources (memory, cpu) and may stop processing events (if hitting memory, cpu limits)

Persona:
Which persona is this feature for?

System Integrator, Event consumer (developer)

Exit Criteria

In-memory channel and one other channel (such as Kafka) can scale down and up

Time Estimate (optional):
How many developer-days do you think this may take to resolve?

weeks

Additional context (optional)
Add any other context about the feature request here.

Ref https://github.com/knative/eventing/issues/2152

[Harwayne]

We have discussed using Knative Services in particular in the Broker Deployments. Knative Services would allow it to scale down to zero and up as needed (which would probably help Broker performance too). Our hesitation is that currently, Knative Eventing does not require Knative Serving. Switching to Knative Services would make it a hard dependency.

I think the first step would be to create a PodSpecable informer-lister. That would allow us to run our data plane pieces as either Deployments or Knative Services, without having a run time dependency on Knative Serving. The controllers would choose which to use via a ConfigMap.

This is the only reference I can find to PodSpecable, but that is a test file.

[aslom]

I really like idea to have it configurable duck-typed for channel (and sources?) and avoiding hard dependency on ksvc (keeping things simple by default)

Eventually all channel implementations could share then the same basic code so they can Deployments or Knative Services and be scalable :)

[nachocano]

I really like idea to have it configurable duck-typed for channel (and sources?) and avoiding hard dependency on ksvc (keeping things simple by default)

Eventually all channel implementations could share then the same basic code so they can Deployments or Knative Services and be scalable :)

Some initial work I did on broker ingress and filter deployments to make them "configurable": either Deployment or ksvc. Very initial work, but I guess whatever we decide for channel/sources, we can do the same for brokers.
The ideal thing as @Harwayne mentioned would be not to add a strong dependency to serving and have some sort of dynamic tracker.

[aslom]

@nachocano @Harwayne what do you mean exactly by strong dependency? Does it mean that code is not importing libraries from serving?

[Harwayne]

@nachocano @Harwayne what do you mean exactly by _strong_ dependency? Does it mean that code is not importing libraries from serving?

The controller must run correctly if it is set to use Deployments and the Knative Serving CRDs are _not_ installed.

Using a straightforward, auto-generated, informer-lister for ksvcs, it would fail loop saying that it cannot list that kind int he given API version.

Ideally, we would also not have a compile time dependency, but I am more flexible on that.

[lionelvillard]

I do have a dynamic informer: https://github.com/lionelvillard/knative-functions-controller/tree/master/pkg/dynamic

Maybe I can move it to pkg if not there yet.

[lionelvillard]

Full disclosure, the original code is there: https://github.com/kubernetes/client-go/tree/master/dynamic

I just made it injection compatible and at the time I used an old client go

[aslom]

After reviewing with @lionelvillard it seems that the easiest first step is to modify eventing/config/channels/in-memory-channel/500-dispatcher.yaml
to make it into ksvc and see dispatcher can be started? @Harwayne @nachocano

That requires to disable checks that assume deployment (reconcile in eventing/pkg/reconciler/inmemorychannel/controller/inmemorychannel.go).

I am going to open WIP PR to have it for testing and review.

If that works then next step is to use ConfigMap to configure dispatcher to use deployment or ksvc - that will change how "static" is dispatcher.

[aslom]

Update on experimenting - after converting imc-dispatcher into ksvc seems to run into race condition between two containers inside that try to bind to 9090 (see below).

Going to investigate more and see if I can get disable for now binding to 9090 to get imc dispatcher started and be able to dispatch one event.

(base) aslom@m eventing (channels-ksvc) $ k -n knative-eventing logs imc-dispatcher-wmjxg-deployment-8458d9b9cb-gq95d -c queue-proxy
{"level":"info","ts":"2019-11-08T19:20:14.593Z","caller":"logging/config.go:100","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n  \"level\": \"info\",\n  \"development\": false,\n  \"outputPaths\": [\"stdout\"],\n  \"errorOutputPaths\": [\"stderr\"],\n  \"encoding\": \"json\",\n  \"encoderConfig\": {\n    \"timeKey\": \"ts\",\n    \"levelKey\": \"level\",\n    \"nameKey\": \"logger\",\n    \"callerKey\": \"caller\",\n    \"messageKey\": \"msg\",\n    \"stacktraceKey\": \"stacktrace\",\n    \"lineEnding\": \"\",\n    \"levelEncoder\": \"\",\n    \"timeEncoder\": \"iso8601\",\n    \"durationEncoder\": \"\",\n    \"callerEncoder\": \"\"\n  }\n}"}
{"level":"info","ts":"2019-11-08T19:20:14.593Z","caller":"logging/config.go:101","msg":"Logging level set to info"}
{"level":"warn","ts":"2019-11-08T19:20:14.593Z","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: open /var/run/ko/HEAD: permission denied"}
{"level":"error","ts":"2019-11-08T19:20:14.594Z","logger":"queueproxy","caller":"queue/main.go:448","msg":"Failed to bring up queue-proxy, shutting down.","knative.dev/key":"knative-eventing/imc-dispatcher-wmjxg","knative.dev/pod":"imc-dispatcher-wmjxg-deployment-8458d9b9cb-gq95d","error":"metrics server failed: listen tcp :9090: bind: address already in use","errorVerbose":"listen tcp :9090: bind: address already in use\nmetrics server failed\nmain.main.func3\n\t/home/prow/go/src/knative.dev/serving/cmd/queue/main.go:423\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1337","stacktrace":"main.main\n\t/home/prow/go/src/knative.dev/serving/cmd/queue/main.go:448\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}

(base) aslom@m eventing (channels-ksvc) $ k -n knative-eventing logs imc-dispatcher-wmjxg-deployment-8458d9b9cb-gq95d -c user-container
2019/11/08 19:18:43 Registering 3 clients
2019/11/08 19:18:43 Registering 2 informer factories
2019/11/08 19:18:43 Registering 1 informers
2019/11/08 19:18:43 Registering 1 controllers
{"level":"info","ts":"2019-11-08T19:18:44.052Z","caller":"logging/config.go:108","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n  \"level\": \"info\",\n  \"development\": false,\n  \"outputPaths\": [\"stdout\"],\n  \"errorOutputPaths\": [\"stderr\"],\n  \"encoding\": \"json\",\n  \"encoderConfig\": {\n    \"timeKey\": \"ts\",\n    \"levelKey\": \"level\",\n    \"nameKey\": \"logger\",\n    \"callerKey\": \"caller\",\n    \"messageKey\": \"msg\",\n    \"stacktraceKey\": \"stacktrace\",\n    \"lineEnding\": \"\",\n    \"levelEncoder\": \"\",\n    \"timeEncoder\": \"iso8601\",\n    \"durationEncoder\": \"\",\n    \"callerEncoder\": \"\"\n  }\n}\n"}
{"level":"info","ts":"2019-11-08T19:18:44.052Z","caller":"logging/config.go:109","msg":"Logging level set to info"}
{"level":"info","ts":"2019-11-08T19:18:44.061Z","logger":"inmemorychannel_dispatcher.in-memory-channel-dispatcher","caller":"dispatcher/controller.go:86","msg":"Setting up event handlers","commit":"e79eab7","knative.dev/controller":"in-memory-channel-dispatcher"}
{"level":"info","ts":"2019-11-08T19:18:44.062Z","logger":"inmemorychannel_dispatcher","caller":"profiling/server.go:60","msg":"Profiling enabled: false","commit":"e79eab7"}
{"level":"info","ts":"2019-11-08T19:18:44.084Z","logger":"inmemorychannel_dispatcher","caller":"metrics/exporter.go:94","msg":"Flushing the existing exporter before setting up the new exporter.","commit":"e79eab7"}
{"level":"info","ts":"2019-11-08T19:18:44.084Z","logger":"inmemorychannel_dispatcher","caller":"metrics/prometheus_exporter.go:37","msg":"Created Opencensus Prometheus exporter with config: &{knative.dev/inmemorychannel-dispatcher inmemorychannel_dispatcher prometheus 5000000000 <nil> 9090 false   {   false}}. Start the server for Prometheus exporter.","commit":"e79eab7"}
{"level":"info","ts":"2019-11-08T19:18:44.084Z","logger":"inmemorychannel_dispatcher","caller":"metrics/exporter.go:103","msg":"Successfully updated the metrics exporter; old config: <nil>; new config &{knative.dev/inmemorychannel-dispatcher inmemorychannel_dispatcher prometheus 5000000000 <nil> 9090 false   {   false}}","commit":"e79eab7"}
{"level":"info","ts":"2019-11-08T19:18:44.084Z","logger":"inmemorychannel_dispatcher.in-memory-channel-dispatcher","caller":"configmap/store.go:157","msg":"tracing-config config \"config-tracing\" config was added or updated: &config.Config{Backend:\"none\", ZipkinEndpoint:\"\", StackdriverProjectID:\"\", Debug:false, SampleRate:0.1}","commit":"e79eab7","knative.dev/controller":"in-memory-channel-dispatcher"}
{"level":"info","ts":"2019-11-08T19:18:44.162Z","logger":"inmemorychannel_dispatcher","caller":"sharedmain/main.go:179","msg":"Starting informers.","commit":"e79eab7"}
{"level":"info","ts":"2019-11-08T19:18:44.262Z","logger":"inmemorychannel_dispatcher","caller":"sharedmain/main.go:185","msg":"Starting controllers...","commit":"e79eab7"}
...

[aslom]

After talking @lionelvillard one idea to consider for future: allow channel dispatcher to be namespace bound ksvc (and not shared across namespaces) as that should improve scalability to have separately scaled dispatchers and make accounting easier for multi-tenant setups?

[aslom]

I think found the right place to disable metrics server however currently it seems to be required?

Removed

          - name: METRICS_DOMAIN
            value: knative.dev/inmemorychannel-dispatcher

but that leas to panic - going to do quick code change to make metrics optional so ksvc diaptcher can start without requiring port 9090 (and confilicting with serving)

(base) aslom@m eventing (channels-ksvc) $ k -n knative-eventing logs imc-dispatcher-7rd8t-deployment-8555bfd5d-tprs9 -c user-container
2019/11/11 20:52:46 Registering 3 clients
2019/11/11 20:52:46 Registering 2 informer factories
2019/11/11 20:52:46 Registering 1 informers
2019/11/11 20:52:46 Registering 1 controllers
{"level":"info","ts":"2019-11-11T20:52:46.442Z","caller":"logging/config.go:108","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n  \"level\": \"info\",\n  \"development\": false,\n  \"outputPaths\": [\"stdout\"],\n  \"errorOutputPaths\": [\"stderr\"],\n  \"encoding\": \"json\",\n  \"encoderConfig\": {\n    \"timeKey\": \"ts\",\n    \"levelKey\": \"level\",\n    \"nameKey\": \"logger\",\n    \"callerKey\": \"caller\",\n    \"messageKey\": \"msg\",\n    \"stacktraceKey\": \"stacktrace\",\n    \"lineEnding\": \"\",\n    \"levelEncoder\": \"\",\n    \"timeEncoder\": \"iso8601\",\n    \"durationEncoder\": \"\",\n    \"callerEncoder\": \"\"\n  }\n}\n"}
{"level":"info","ts":"2019-11-11T20:52:46.442Z","caller":"logging/config.go:109","msg":"Logging level set to info"}
{"level":"info","ts":"2019-11-11T20:52:46.447Z","logger":"inmemorychannel_dispatcher.in-memory-channel-dispatcher","caller":"dispatcher/controller.go:86","msg":"Setting up event handlers","commit":"e79eab7","knative.dev/controller":"in-memory-channel-dispatcher"}
{"level":"info","ts":"2019-11-11T20:52:46.448Z","logger":"inmemorychannel_dispatcher","caller":"profiling/server.go:60","msg":"Profiling enabled: false","commit":"e79eab7"}
panic: The environment variable "METRICS_DOMAIN" is not set

If this is a process running on Kubernetes, then it should be specifying
this via:

  env:
  - name: METRICS_DOMAIN
    value: knative.dev/some-repository

If this is a Go unit test consuming metric.Domain() then it should add the
following import:

import (
    _ "knative.dev/pkg/metrics/testing"
)

goroutine 1 [running]:
knative.dev/eventing/vendor/knative.dev/pkg/metrics.Domain(0x19, 0xc00004204a)
    /Users/aslom/Documents/awsm/go/src/knative.dev/eventing/vendor/knative.dev/pkg/metrics/config.go:266 +0xfa
knative.dev/eventing/vendor/knative.dev/pkg/metrics.UpdateExporterFromConfigMap(0x191b947, 0x1a, 0xc0000bd020, 0x14)
    /Users/aslom/Documents/awsm/go/src/knative.dev/eventing/vendor/knative.dev/pkg/metrics/exporter.go:67 +0x26
knative.dev/eventing/vendor/knative.dev/pkg/injection/sharedmain.MainWithConfig(0x1caf8c0, 0xc0000bc050, 0x191b947, 0x1a, 0xc0001961e0, 0xc000583f80, 0x1, 0x1)
    /Users/aslom/Documents/awsm/go/src/knative.dev/eventing/vendor/knative.dev/pkg/injection/sharedmain/main.go:171 +0xcb1
knative.dev/eventing/vendor/knative.dev/pkg/injection/sharedmain.MainWithContext(0x1caf8c0, 0xc0000bc050, 0x191b947, 0x1a, 0xc000583f80, 0x1, 0x1)
    /Users/aslom/Documents/awsm/go/src/knative.dev/eventing/vendor/knative.dev/pkg/injection/sharedmain/main.go:111 +0x21d
knative.dev/eventing/vendor/knative.dev/pkg/injection/sharedmain.Main(0x191b947, 0x1a, 0xc000583f80, 0x1, 0x1)
    /Users/aslom/Documents/awsm/go/src/knative.dev/eventing/vendor/knative.dev/pkg/injection/sharedmain/main.go:97 +0x9d
main.main()
    /Users/aslom/Documents/awsm/go/src/knative.dev/eventing/cmd/in_memory/channel_dispatcher/main.go:29 +0x67

[aslom]

I could not find a diagram describing Knative data-plane architecture - do you know any good blog post? links?

I have put together a simple diagram showing current architecture:

and what it would look with ksvc and scaling:

You can edit it here:
https://docs.google.com/presentation/d/13V8EE9nHEgyKrrKMMvUdeSymWFeEdRFCg_qv7n-7QRo/edit?usp=sharing

[mikehelmick]

related to #1555

[mikehelmick]

/cc @grantr

[nachocano]

I think found the right place to disable metrics server however currently it seems to be required?

Removed

          - name: METRICS_DOMAIN
            value: knative.dev/inmemorychannel-dispatcher

but that leas to panic - going to do quick code change to make metrics optional so ksvc diaptcher can start without requiring port 9090 (and confilicting with serving)

Yes, I think I had this same issue and IIRC it was because the queue-proxy container binds the port 9090. For initial testing I agree you can easily comment out the metrics.UpdateExporter part and see if you can get things running.

[aslom]

@nachocano as a workaround I have added PROMETHEUS_PORT to allow overriding default 9090
https://github.com/knative/eventing/compare/master...aslom:channels-ksvc
and with this I can start imc-dispatcher as a ksvc but then it runs into problems i am investigating - probably conflicting with imc-controller and imc-dispatcher Deployment

[lionelvillard]

/milestone 0.12.0

[knative-prow-robot]

@lionelvillard: You must be a member of the knative/knative-milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.

In response to this:

/milestone 0.12.0

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[aslom]

Related issues that may make it easier is to allow configuring dispatcher per-user namespace: https://github.com/knative/eventing/issues/2171 and https://github.com/knative/eventing/pull/2333

[aslom]

I have wrote down ideas about re-usable components for channels / delivery
https://docs.google.com/presentation/d/1KxWNXrMki2JgC-D92cuWuedR77fh4ZGODHrW-XNgwR4/edit?usp=sharing

[slinkydeveloper]

@aslom for what regards kafka channel, I already wrote a document a while ago about it with a complete plan, and i'm working on it now: https://docs.google.com/document/d/1ve1pAfs1uPRQVi8VxrUbpchrbSf7S3N2_seoIU9XBx4/edit?usp=sharing

[aslom]

@markfisher shred link to diagram from previous discussion this topic: https://docs.google.com/drawings/d/1muhTBFfzuV1JrEEj-6hmWLWFjk7Sq-Khb6jC1q_hOM4/edit

[markfisher]

That doc might be confusing given the scope/granularity of Channel today. Basically a Channel in the current model would encompass all of the components between the 2 yellow Service boxes in that diagram, other than the Subscription and Function. The purpose of the diagram (as well as I can recall) was to propose that "channel" could be a much simpler-scoped, lower level building block that acts only as a way to read and/or write Cloud Events to/from some external system. This was inspired by the Channel Adapter Enterprise Integration Pattern for defining a pluggable abstraction layer between the "application" domain - Knative in this case - and external messaging systems. One of the original motivations for the diagram was a recognition of significant overlap between e.g. a KafkaSource and a KafkaChannel (where the difference is simply whether one end or both ends are exposed for interaction with other "internal" components).

If the abstraction boundary is defined at that level, it might make it easier to use Knative terminology on one side of the abstraction, and the terminology of the external system on the other (implementation) side, thus alleviating some of the confusion that was mentioned on the WG call today.

[aslom]

Updating current status of autoscaling and channels: https://docs.google.com/document/d/18Ma-uqsB0SYzwvRZ_rz5c8m21JAXGP0oLZpr8W0Jwak/edit#heading=h.ga53edbcomm2

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[aslom]

/reopen

[slinkydeveloper]

Is this still relevant? Since I see you worked on this in more specific issues, can we close this more general issue?