Airflow: Add celery stats to Airflow webui

[turbaszek]

Description

Show basic information about Celery workers in Airflow webui.

Use case / motivation

Currently, users using CeleryExecutor have to use Flower to access information about Celery workers. I would like to propose adding new view in Airflow webui that would show similar information.

This should be easily doable by using:

from airflow.executors.celery_executor import app

inspect = self.app.control.inspect(timeout=0.01, destination=None)
result = partial(getattr(inspect, "stats"))()

Note: this is how Flower gets the information.

Related Issues

N/A

[turbaszek]

@ashb @dimberman @kaxil @mik-laj @potiuk WDYT?

[ashb]

Might be useful - what info does that include?

Also there's so much more that flower does, and can do better because it uses Tornado, so we should draw a clear line in the code, and in the view say something like "for more details use flower"

[turbaszek]

[potiuk]

Cool!

[turbaszek]

I'm wondering if it would make sense to show more info, like number of tasks and so on. Maybe charts?

[potiuk]

I think if those are easy to retrieve, and show, it might make sense, but If those can be seen in Flower, I agree with @ashb that there's no reason for duplication. I think however (if it is not done already) that flower should become a first-class-citizen of Airflow in this case and it should have at least some support for RBAC authentication integrated with the Airflow RBAC. This might mean either an iframe (BRRRR!) or some other kind of proxying to flower interface and RBAC permission to control access there.

It would be nice to have to open only one firewall port (airflow UI) and one authentication/SSO integration to access flower interface.

[turbaszek]

We can start flower in the background and use its API to show data in Airflow. By more info I thought about showing Airflow-related information like, number of (failed, succeded, queued) tasks. I know users can use statsd and grafana, but having simple overview of the state without additional configuration would be helpful imho