Airflow: setproctitle is flaky to install and can be problematic in some environments

[jdavidheiser]

Apache Airflow version: 1.10.4

Kubernetes version (if you are using kubernetes) (use kubectl version): N/A

Environment: Mac OS / Linux

• Cloud provider or hardware configuration: N/A
• OS (e.g. from /etc/os-release): Mac OS 10.15.4
• Kernel (e.g. uname -a): darwin
• Install tools: pip/homebrew
• Others:
  What happened:
  I am running into issues with the installation of the python library setproctitle in environments we use to run unit tests.

What you expected to happen:
Airflow to be runnable across multiple platforms and environments with minimal tweaking

From some Google results, this package seems to be especially flaky on install, often getting messed up by things like cached wheels.

This seems to be used sparingly in Airflow, and I am wondering if it adds enough utility to be worth including a package that is flaky to install, and which is not very cross platform. From the setprocutils docs:

The procedure is hardly portable across different systems. PostgreSQL provides a good multi-platform implementation: this module is a Python wrapper around PostgreSQL code.

How to reproduce it:
This would be hard to repro outside of our specific testing environment, because it is an issue of caching Python packages.

However, I have a suggested fix: remove the dependence on setproctitle. This does not seem necessary, especially as Airflow migrates toward a distributed computing environment where named containers and pods are more useful than named processes.

Anything else we need to know:

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[mik-laj]

Airflow uses many processes that run in one container. The clear name of the process is very useful. What alternative solution do you suggest instead of deleting it? Every problem has many solutions and death is not a solution. It's an escape. We will manage together and we will definitely get out of this situation. We do not have to take such radical steps.

[ashb]

Closing this in favour of #8172 as it contains more information (i.e. it contains the actual error message.)