Tribler: deployment monitoring and epic progress dashboard

[synctext]

To better organise ourselves we need more critical information in 1 place.

The coming time we aim to close #1 finally. Our progress towards this goal and how stable we are should be captured in a Tribler-at-a-Glance dashboard. Example from Jenkins:

https://medium.com/kj187/jenkins-job-dashing-widget-cc72feeed654

https://www.level-up.one/6-of-my-favorite-jenkins-plugins/

https://www.datadoghq.com/blog/monitor-jenkins-datadog/

Tribler critical information candidates:

• stability issues
  
  
  
  • crash reports from the wild (24h, last week, last month) latest devel, latest stable version and all versions
  
  
  • Application tester with random clicker number of faults (24h, last week, last month) latest devel, latest stable version
  
  
  • burn-in testing of running Tribler for 1 week: total CPU cycle, peak memory, total disk IO (crash or run-away resource usage)
  
  
  • issues pending
  
  
• performance monitoring
  
  
  
  1. Anonymous end-to-end download performance (latest devel, latest stable version)
  
  
  2. Exit node based download speed
  
  
  3. Start of download delay, (non-)anonymous mode
  
  
  4. First time startup time
  
  
• deployment monitoring
  
  
  
  • explorer Trustchain, growth of blocks in (last minute ! ; last hour ; last day; last week ; all time)
  
  
  • exit node status (CPU, connections, idle slots, memory?)
  
  
  • traffic stats
  
  
  • metadata status: keyword searches, channel gossip community
  
  

[devos50]

Interesting visualisations! Somewhat related to https://github.com/Tribler/tribler/issues/3508 (at least the TrustChain deployment monitoring).

e2e anonymous download is an excellent candidate for performance monitoring and should not take long to setup. I think @ichorid addressed this a while ago actually but it has not been actively monitored since then. In fact, making us (more) aware of failing tests/validation experiments is becoming a necessity as the number of different tests that run with fixed time intervals is growing.

I think we have to address this issue rather sooner than later. The problem is that if we do not do it, we will have a proliferation of different tools. Currently, we have the TrustChain explorer, Tribler user statistics, the error reporter and all tests/monitors on Jenkins. There might be some opportunity to merge some tools, which eases maintenance.

metadata status: keyword searches, channel gossip community

This might be a dangerous one to monitor and could be a violation of ones privacy expectations of Tribler.

[synctext]

Please look at FileCoin slipped roadmap. After Release 7.5 I'm considering that we work together on the first _Jenkins dashboard_ for 2 weeks:

• [ ] arrange hardware monitors with obscene awesomeness, due to size (@synctext)
• [ ] Anonymous end-to-end download performance (latest devel, latest stable version) (@egbertbouman)
• [ ] crash reports from the wild (24h, last week, last month) latest devel, latest stable version and all versions (@ichorid)
• [ ] Application tester with random clicker number of faults (24h, last week, last month) latest devel, latest stable version (@devos50)
• [ ] IPv8 traffic stats with total of unique number of public keys in last (24h, last week, last month) within discovery community (heard about only, responsive) (@qstokkink)
• [ ] explorer Trustchain, growth of blocks in (last minute ! ; last hour ; last day; last week ; all time) (@grimadas)
  

[qstokkink]

Can we decide on some software/library to use (or to make) to graph all of this data? All sorts of dashboard creation tools exist.

For example: https://dzone.com/articles/build-beautiful-console-dashboards-with-sampler

[devos50]

Most of this data can either be extracted from our existing Jenkins Job using the API, or from our running Trustchain explorer backend, also with API requests. One of the question we should also answer, is whether we want a dedicated website for this. Jenkins unfortunately does not provide the tools for such real-time data, and integration of this dashboard in Jenkins would just be a new job with succeed/fail status.

arrange hardware monitors with obscene awesomeness, due to size

We should secure a prominent spot at the coffee machine ☕️

[qstokkink]

I propose starting with something "easy". Exposing GitHub events through tribler.org:

• Add a webhook to GitHub for the Tribler repository (sending POST requests to the tribler.org domain).
• Add a new page (tribler.org/githubevents?) which renders all GitHub events (possibly with websockets for live updates).

The idea is that we can reuse the resulting backend for another (bigger and better) dashboard and we'll have something to look at in the mean time.

[devos50]

One way to get more insights into our user count is by analysing the crawled TrustChain data. The plot below is generated based on our current dataset, with over 80.000 users and 123 million records. The (major) releases of Tribler are annotated. Note how our 7.5.0 release resulted in an increase in new user count.

Parsing this 97GB database, however, is computationally intensive and could be done on a daily basis for example. A dashboard could include this static image.

[synctext]

In 2006-2009 we had initial deployment monitoring. Included in Zeilemaker master thesis.

[xoriole]

Based on data we already have

[kozlovsky]

Yesterday I did a little research on this topic, and now I want to suggest a way to show anonymized performance statistics. It may be the following set of technologies:

1. Custom client-side code to prepare anonymized statistics
2. Dedicated server with custom API as an entry point
3. InfluxDB for storing anonymized data
4. Grafana for displaying beautiful graphs

The most popular tool for gathering and processing metrics is Prometheus. It has has a big community and is widely used for gathering server metrics. Prometheus if often compared with InfluxDB (see the comparison on official Prometheus doc). While Prometheus is more popular, in my opinion, InfluxDB is better suited to our needs for the following reasons:

1) Prometheus pulls metrics from the known number of server instances. In our case, we cannot pull statistics from client machines and want to push instead. While it is possible to use Prometheus with additional tools like Prometheus Aggregation Gateway, it in some way goes against Prometheus philosophy. On the other side, InfluxDB expects that the data are pushed, which is better suited to our needs.

2) Prometheus data storage is ephemeral and not intended to be stored for a long time. InfluxDB data are persistent and can be used to compare changes in gathered statistics on long time intervals.

Grafana is a very popular open-source tool for graph visualization, which can be used with Prometheus, InfluxDB, and multiple other data sources. It allows constructing powerful dashboards with different types of graphs and charts.

If we decide to use this set of tools, I think I can take on this task. I see the following sub-tasks here to be implemented:
1) a client-side code for preparing anonymized statistics
2) a client-side code to send gathered statistics to our dedicated server
3) a custom server API to collect anonymized statistics
4) a server code which implements API as mentioned above, aggregate collected data and put it into InfluxDB instance
5) deploy a dedicated server with statistics gathering API, deploy InfluxDB instance (probably on some different machine)
6) deploy Grafana instance
7) make Grafana dashboard

Later we can use Grafana to display all graphs, not only user statistics but also server builds, etc.

What do you think?

[synctext]

• Health monitoring of client state
• Health monitoring of our website, Github, statistics servers
• Health monitoring of bootstrap servers and crawler servers

Pitfall: everything we want with our self-organising research project is easier to do in a central server... Primarily use our crawlers as early warning infrastructure! (IPv8 is designed for network health monitoring) Then we need to emphasise crawler intelligence and stats aggregation.

Are we not re-creating this from scratch? https://jenkins-ci.tribler.org/job/Test_BootstrapServers/lastSuccessfulBuild/artifact/walk_rtts.png

First, anonymity is our existential feature. How to do this? (True anonymity might be impossible, OFF switch by default)
We could show the user inside the debug panel the exact history and record which will be shared in private with our debug servers optionally? Can we protect against Internet address leakage? Many steps in future I guess to re-usage our Tor-like stuff while debugging our Tor-like stuff :-)

This needs to be opt-in for production releases and can hopefully be opt-out for nightly builds and Beta versions. What about Release Candidates?

InfluxDB: 34,082 commits, 19.5k of stars on Github. This is a general time-series database solution, we still need to make custom code for deployment monitoring?

This seems quite complex tooling. Afraid of overengineering for the user community we have currently. However, deployment monitoring is something we really need to do more and get right.

[xoriole]

InfluxDB and Graphana are indeed good choices.

1.Custom client-side code to prepare anonymized statistics
2.Dedicated server with custom API as an entry point
3.InfluxDB for storing anonymized data
4.Grafana for displaying beautiful graphs

I have done some work on 1 and 2. I'm extending https://release.tribler.org/docs to receive anonymized data from the client. That can be the entry point to further processing using InfluxDB and visualizing on Graphana.

[kozlovsky]

We probably can use InfluxDB Jenkins plugin to put deployment statistics into the InfluxDB:
https://wiki.jenkins.io/display/JENKINS//InfluxDB+Plugin

[synctext]

Change of plans:-)
By 25 September aim to have plots in Jenkins. The PopularityCommunity is crawled and health statistics are refreshed every few minutes or half an hour. After this test project we determine what we need and roadmap. Could be a fix of the PopularityCommunity code plus algorithm as next step, deploy, monitor, etc.

Our current methodology:

• undocumented algorithm
• exclusively rely on unit tests
• end-to-end test manually if the desired feature works
• no health monitoring of protocol deployment

Tribler is a bottomless pit of problems. (stolen quote)
Our work methodology should become relentlessly data-driven: there is direct evidence we need better crawling, no evidence of client monitoring beyond debug screen and crash reporting (might change; agile)

[devos50]

exclusively rely on unit tests

I think a key metric is the stability of our unit tests. Currently, unstable unit tests (both on devel and our release branches) are delaying the development process. Converting the test suite to pytest, which should make the debugging process of errors in the tests easier, is much more work than I anticipated.

My suggestion would be to continuously run all unit tests on a dedicated machine and include in the upcoming dashboard how stable they are (e.g., % of runs failing during the last day).

[synctext]

Related work: https://stats.goerli.net/