Beats: "make python-env" fails on current Homebrew

[beniwohli]

Since some time, python defaults to Python 3 when using Homebrew on OSX. Python 3 is also the default on Ubuntu 18.04.

make python-env defaults to use python, and when that points to a Python 3 binary, promptly fails, trying to install functools32:

https://github.com/elastic/beats/blob/a8da1868564c9851a8f23d35e563a09cab2df2e7/libbeat/tests/system/requirements.txt#L13

Collecting functools32==3.2.3.post2 (from -r ./_beats/libbeat/tests/system/requirements.txt (line 13))
Using cached https://files.pythonhosted.org/packages/c5/60/6ac26ad05857c601308d8fb9e87fa36d0ebf889423f47c3502ef034365db/functools32-3.2.3-2.tar.gz
    Complete output from command python setup.py egg_info:
    This backport is for Python 2.7 only.
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/fd/pwc3ll_s05v3cqbqsnv9r77r0000gn/T/pip-install-ns9id84h/functools32/

Even after fixing that (and enum34), the tests and other Python scripts won't run on Python 3 (see https://github.com/elastic/beats/issues/5950).

As a workaround, one can create the virtualenv manually using Python 2

virtualenv -p python2 build/python-env

[ruflin]

@elastic/beats Time to go to python 3?

[kesslerm]

As a workaround, one can set VIRTUALENV_PARAMS="-p python2.7" before running make.

VIRTUALENV_PARAMS="-p python2.7" make

[urso]

Maybe we should set VIRTUALENV_PARAMS in the Makefile for the time being. Having to remember to set an env variable or setup a private virtualenv as workaround is not really user-friendly.

[amandahla]

I have set this variable and it works but it shows the warning:
"DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7."

At least, it seems that everything worked as expected for the "make create-metricset".

[amandahla]

Update: now I had to use "export PYTHON_EXE=/usr/bin/python2" too to avoid:

2019/08/13 10:57:31 Found Elastic Beats dir at /home/METRICBEATS/beats/src/github.com/elastic/beats
Generated fields.yml for metricbeat to /home/METRICBEATS/beats/src/github.com/elastic/beats/metricbeat/build/fields/fields.all.yml
    ERROR: Command errored out with exit status 1:
     command: /home/METRICBEATS/beats/src/github.com/elastic/beats/metricbeat/build/python-env/build/ve/linux/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-g_nfbvj1/functools32/setup.py'"'"'; __file__='"'"'/tmp/pip-install-g_nfbvj1/functools32/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
         cwd: /tmp/pip-install-g_nfbvj1/functools32/
    Complete output (1 lines):
    This backport is for Python 2.7 only.
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Using "VIRTUALENV_PARAMS", the "make python-env" was fine but for "make update" in metricbeat a "...beats/metricbeat/build/python-env/build/ve/" directory was created using python3.

So I check the dev-tools/mage/pytest.go source and then set the PYTHON_EXE variable for him.

[fearful-symmetry]

This should be fixed by #14798

[jsoriano]

14798 finished, we are now in the Python 3 age :slightly_smiling_face:, what should solve this problems.