Securedrop: versions of pip, setuptools, wheel used by dh_virtualenv during securedrop-app-code build are not pinned.

[zenmonkeykstop]

Description

While investigating #5108, it was noticed that setuptools and associated packages are installed by dh_virtualenv using the following command:

 Running command '/tmp/securedrop-app-code_1.3.0~rc1+xenial_amd64/debian/securedrop-app-code/opt/venvs/securedrop-app-code/bin/python' '/tmp/securedrop-app-code_1.3.0~rc1+xenial_amd64/debian/securedrop-app-code/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/pip' install --ignore-installed --no-user --prefix /tmp/pip-build-env-tzw3qhas/overlay --no-warn-script-location -v --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=18.5' wheel 'cffi>=1.8,!=1.11.3; python_implementation != '\"'\"'PyPy'\"'\"''

Package versions are not pinned and hashes are not required. (wheel in particular is currently at version 0.34.1, which has a downstream issue that's breaking make build-debs.)

Steps to Reproduce

• run make build-debs against current develop

Expected Behavior

• build process uses pinned and reviewed packages

Actual Behavior

• build process uses latest versions without checking hashes

[rmol]

Looks like we can specify the versions of some of these with environment variables.

[kushaldas]

The problem starts with https://github.com/spotify/dh-virtualenv/blob/master/dh_virtualenv/deployment.py#L138 and because we are not using venv, it uses virtualenv as in https://github.com/spotify/dh-virtualenv/blob/master/dh_virtualenv/deployment.py#L145 and then it tries to get latest versions of pip and setuptools and somehow that screwed up the build environment.

The simpler solution is to have python3-venv installed and also pass --builtin-venv to dh-virtualenv and use a newer build container for the same. I will create a PR with the changes in some time. I tested my theory locally.

[pradyunsg]

Another option might be to pass --no-download to virtualenv, which makes it use the bundled wheels instead of fetching the latest.

[kushaldas]

Another option might be to pass --no-download to virtualenv, which makes it use the bundled wheels instead of fetching the latest.

Testing this right now.

[kushaldas]

Another option might be to pass --no-download to virtualenv, which makes it use the bundled wheels instead of fetching the latest.

Testing this right now.

Problem, in Xenial we have dh-virtualenv 0.11-1 version, and that has no --extra-virtualenv-arg "--no-download", thus we can not pass extra arguments while creating the virtualenv.

The docs are available at https://dh-virtualenv.readthedocs.io/en/0.11/usage.html

[kushaldas]

Maybe there is a better way to pass the variable using export DH_VIRTUALENV_ARGUMENTS=--no-site-packages example to the container.

[emkll]

Thanks for the investigation. I've pushed a builder image update with python3-venv.

As a workaround to provide more assurances that we are not installing unpinned dependencies at build-time, could we, instead pass some pip args (e.g.: --no-deps) through --extra-pip-arg [1] ?

[1] https://dh-virtualenv.readthedocs.io/en/0.11/usage.html?highlight=extra-pip#cmdoption--extra-pip-arg

[kushaldas]

Yes, if you want please add that in a new commit here.

[rmol]

Using dh-virtualenv --builtin-venv does fix the wheel bug, but results in pip, pkg_resources, and setuptools being installed in the virtualenv from /usr/share/python-wheels, which is populated from the python-pip-whl Debian package. (See /usr/lib/python3.x/ensurepip/__init__.py, which is called from /usr/lib/python3.x/venv/__init__.py.)

It's better than the current situation because we're using Debian packages instead of whatever's on PyPI that day, but they're still not specified or hashed, so we don't really know what we're getting when cutting any SD release. :slightly_frowning_face:

I think we're going to want to pin pip, pkg_resources, and setuptools in the requirements files, and we'll probably need https://github.com/freedomofpress/securedrop/issues/4686 as part of this.

[rmol]

I think this has been addressed in #5484.