Eventing: E2E Metrics for Importer/Broker/Trigger model

[nachocano]

Problem
This is a placeholder feature for metrics-related tasks that will be driven by the eventing-observability subgroup and scoped down to 0.9, 0.9+ releases.
Here are some initial tasks we identified:

• [ ] Infrastructure setup, scraping of metrics - Prometheus
  
  
  
  • Add prometheus operator issue #1925
  
  
  • [x] knative-observability changes #52
  
  
  • [ ] documentation in eventing and docs
  
  
  • Control plane
    
    
    
    
    
    • [x] Broker Ingress/Filter #5378
    
    
    
  
  
  • [x] Eventing Controller #5378
  
  
  • [x] ApiServerSource #1747
  
  
  • [x] CronJobSource #1747
  
  
  • [x] ContainerSource #1747
  
  
  • [ ] KafkaSource
  
  
  • [ ] GitHubSource
  
  
  • [x] PullSubscription
  
  
  • [x] Storage Source #246
  
  
  • [ ] AWSSqsSource
  
  
  • [x] InMemory Channel #1747
  
  
  • [x] Kafka Channel #617
  
  
  • [x] NATSS Channel #614
  
  
  • [x] Channel (from knative-gcp)
  
  

• [x] Infrastructure setup and scraping of metrics - Stackdriver

  • [x] Configurable metrics backend #1726
  • [x] Broker ingress/filter changes metrics refactoring / Use stats reporter #1739, #1735
  • [x] pkg changes #612
  • [x] Eventing changes to support newer pkg #1779
  • [x] CloudEvents SDK changes to surface HTTP status codes #177
  • [x] event_processing_latencies not working #1753
  • [x] Tag broker metrics by type #1423
  • [x] Use responseCode tag in Broker ingress/filter metrics instead of result #1784
  • [x] Add broker label selector in Triggers #1806
  • [x] Serialization/Deserialization of configMaps in pkg #656
  • [x] Source stats_reporter in pkg #648

• [ ] Data plane metrics

  • [x] Broker (event_count)
  • [x] Trigger (event_count, event_processing_latencies, event_dispatch_latencies)
  • [ ] Importers (event_count)
  • [x] ApiServerSource #1786, #1864, #1872
  • [x] CronJobSource #1790
  • [ ] KafkaSource
  • [ ] GitHubSource
  • [ ] AWSSqsSource
  • [x] PullSubscription #250
  • [x] Storage #250
  • [ ] Channels (event_count)
  • [ ] InMemoryChannel (issue #1818) (needs big refactor --> done in #1933)
  • [x] Channel (from knative-gcp)
  • [ ] Kafka (needs refactor --> migrate to CE sdk)
  • [ ] NATSS (needs refactor --> migrate to CE sdk)

• [ ] Graphana Dashboards

  • [x] Broker/ Trigger (issue #1821) #1966
  • [x] Importers (issue #1822) #1966
  • [ ] Channels (issue #1823)

• [ ] E2E tests for metrics

  • [ ] Infrastructure setup (prometheus) (issue #1920)
  • [ ] Broker (issue #1819)
  • [ ] Trigger (issue #1820)
  • [ ] Sources
    
    
    
    • [ ] ApiServerSource (issue #1815)
    
    
    • [ ] CronJobSource (issue #1816)
    
    
    • [ ] KafkaSource
    
    
    • [ ] GitHubSource
    
    
    • [ ] AWSSqsSource
    
    
    • [x] PubSub (from knative-gcp, issue https://github.com/google/knative-gcp/issues/289)
    
    
    • [x] Storage (from knative-gcp, issue https://github.com/google/knative-gcp/issues/288)
    
    
  • [ ] Channels
    
    
    
    • [ ] InMemoryChannel (issue #1817)
    
    
    • [ ] Channel (from knative-gcp, issue https://github.com/google/knative-gcp/issues/287)
    
    
    • [ ] Kafka
    
    
    • [ ] NATSS
    
    

• [ ] Documentation

  • [x] Metrics available (Broker/Trigger, Core Sources) #1891
  • [x] Accessing metrics #1891
    
    
    
    • [x] prometheus
    
    
    • [x] stackdriver
    
    
  • [ ] Instrumentation
    
    
    
    • [ ] Source
    
    
    • [ ] Channel
    
    
  • [ ] Doc in knative/docs issue #1792

Persona:
Which persona is this feature for?
Event consumer, Event producer, and Operator.

Exit Criteria
Users should be able to see metrics in the grafana/prometheus UI and stackdriver. Documentation should be available in knative.dev, indicating what metrics are defined and how to add new ones, as well as how to configure the metrics infrastructure. E2E tests should be in place as well.

Additional context (optional)
Some notes on the eventing-observability subgroup https://docs.google.com/document/d/178iKE5_qtnXQfYcXp-NeKWKguoM_GLKuz3E2w96F0gI/edit?usp=sharing

[nachocano]

/assign @nachocano
/assign @sayanh

[knative-prow-robot]

@nachocano: GitHub didn't allow me to assign the following users: sayanh.

Note that only knative members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @nachocano
/assign @sayanh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nachocano]

fyi @sayanh
apparently we cannot have two persons assigned.

[nachocano]

fyi @Harwayne

[akashrv]

/milestone v0.9.0

[akashrv]

The way we defined the metrics for Broker, could we define for Importers and add it to the data plane contract of the importer (@n3wscott @nachocano ) ?

[nachocano]

The way we defined the metrics for Broker, could we define for Importers and add it to the data plane contract of the importer (@n3wscott @nachocano ) ?

I think we can do that.

[sayanh]

Infrastructure setup and scraping of metrics (e.g., prometheus operator, or via configMaps, as with serving)

Working on a first prototype with configmaps. After discussion with @nachocano, few more concrete stuff came up:

• [ ] Have a working Prometheus scrape config for eventing pods
• [ ] Have a reconciler dashboard
• [ ] Create a common repo for dashboards ( as patching from serving is not ideal)

[nachocano]

• [ ] Create a common repo for dashboards ( as patching from serving is not ideal)

I think this one refers to something like have a common repo for cross-observability things. There is one called knative-observability, but not sure about the work being done there.
I think for 0.9 we'll have to patch, but we can start the discussions on what is the right approach.

Thanks for working on this @sayanh

[akashrv]

The way we defined the metrics for Broker, could we define for Importers and add it to the data plane contract of the importer (@n3wscott @nachocano ) ?

I think we can do that.

Then lets add it to the list of tasks (checkboxes) and follow-up with @n3wscott to add to importer spec.

[nachocano]

Current documentation: https://github.com/knative/eventing/blob/master/docs/metrics.md

[nachocano]

I'm looking at Infrastructure setup and scraping of metrics - Stackdriver subtask

[nachocano]

Other tasks that I'm looking at

• [x] CloudEvents SDK changes to surface HTTP status codes
• [x] Use responseCode tag in broker ingress/filter metrics instead of result
• [x] Refactor broker ingress/filter to use Stats Reporter
• [x] Refactor pkg for stackdriver support

[sayanh]

Working on:
Data plane metrics -> Importers -> CronJobSource
Infrastructure setup -> Control plane -> NATSS Channel

[nachtmaar]

Working on:
Infrastructure setup -> Control plane -> GitHubSource
Data plane metrics -> Importers -> GitHubSource

[Harwayne]

/project Observability In progress

[akashrv]

/milestone v0.10.0

[marcobebway]

Working on:
Infrastructure setup -> Control plane -> NATSS Channel

Agreed with Sayan about it: https://github.com/knative/eventing/issues/1693#issuecomment-526759177.

Update

• This is the related PR: https://github.com/knative/eventing-contrib/pull/614.

[nachocano]

Working on:
Infrastructure setup -> Control plane -> GitHubSource
Data plane metrics -> Importers -> GitHubSource

@nachtmaar was wondering if you were able to finish these ones, or if there are things pending?

[nachtmaar]

Working on:
Infrastructure setup -> Control plane -> GitHubSource
Data plane metrics -> Importers -> GitHubSource

@nachtmaar was wondering if you were able to finish these ones, or if there are things pending?

Hey @nachocano. I am currently only working on Control Plane. Did not start with Data plane yet.
Here is a quick status update:

• I already refactored GitHubSource to make use of knative.dev/pkg/controller runtime.
• For that I had to refactor the whole Reconciler, it was still using sigs.k8s.io/controller-runtime
• Metrics are working now with knative.dev/pkg/controller runtime
• Currently trying to port all reconciler test to use TableTest which is a big rewrite too. Got the first two tests working.
• Reconciler works after refactoring, but still having a few issues (emitting events e.g.). For that reason I started working on reconciler tests, which were not working anymore with controller refactoring
• Don't have a PR yet, but all my code is at https://github.com/nachtmaar/eventing-contrib/tree/pkg-controller-refactoring

[nachocano]

Working on:
Infrastructure setup -> Control plane -> GitHubSource
Data plane metrics -> Importers -> GitHubSource

@nachtmaar was wondering if you were able to finish these ones, or if there are things pending?

Hey @nachocano. I am currently only working on Control Plane. Did not start with Data plane yet.
Here is a quick status update:

• I already refactored GitHubSource to make use of knative.dev/pkg/controller runtime.
• For that I had to refactor the whole Reconciler, it was still using sigs.k8s.io/controller-runtime
• Metrics are working now with knative.dev/pkg/controller runtime
• Currently trying to port all reconciler test to use TableTest which is a big rewrite too. Got the first two tests working.
• Reconciler works after refactoring, but still having a few issues (emitting events e.g.). For that reason I started working on reconciler tests, which were not working anymore with controller refactoring
• Don't have a PR yet, but all my code is at https://github.com/nachtmaar/eventing-contrib/tree/pkg-controller-refactoring

Thanks @nachtmaar for the detailed update! Sounds great, thanks for taking care of this. And yeah, whenever you have the tests working, let's get the control plane part in, and then we can focus on data plane in another PR.

[nachocano]

@yolocs is starting to work on Data Plane metrics for InMemoryChannel

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.