Azure-functions-core-tools: distlib dependency is throwing an error during publish

[asavaritayal]

_From @asavaritayal on September 13, 2018 23:16_

I get the following error on publishing an app with the following dependencies -

> func azure functionapp publish astaycustomerchurnapp --force

There was an error restoring dependencies.ERROR: ['pip', 'download', '-r', '/Users/asavaritayal/customer-churn-prediction/requirements.txt', '--dest', '/var/folders/3v/hjcbc73d5t373f24vgrh8ny40000gn/T/azureworkersp1rpvrl'] failed with exit code 1

Here is my requirements.txt :

azure-functions==1.0.0a4
azure-functions-worker==1.0.0a4
distlib==0.2.8.dev0
grpcio==1.14.2
grpcio-tools==1.14.2
numpy==1.15.1
pandas==0.23.4
protobuf==3.6.1
python-dateutil==2.7.3
pytz==2018.5
scikit-learn==0.18.1
scipy==1.1.0
six==1.11.0

Everything works locally so not sure what the issue is.

_Copied from original issue: Azure/azure-functions-python-worker#197_

[asavaritayal]

@elprans this looks similar to the issue @priyaananthasankar was facing. Any idea what the issue is?

[ankitkumarr]

This requirements.txt is failing specifically because of distlib==0.2.8.dev0 on line 3. The version number is incorrect. @asavaritayal mentioned that she hadn't specifically added that package.

Investigating the code in PythonHelpers.cs ,
var exe = new Executable("pip", "install -U git+https://github.com/vsajip/distlib.git@15dba58a827f56195b0fa0afe80a8925a92e8bf5");
This installs distlib version specified by a commit id.

So, when someone executes pip freeze > requirements.txt, the installed package represented by distlib==0.2.8.dev0 gets added in the requirements.txt.

Trying, pip install distlib==0.2.8.dev0 will show that no such version of distlib exists (because that was installed by versioning a specific commit).

I am guessing we do need distlib's specified version to be there? @elprans would you know?

[elprans]

@ankitkumarr Yes, unfortunately there is a critical bug in all released versions of distlib, so we rely on an unreleased version. One possible fix would be to install distlib in a separate location, outside of the app virtualenv and configure PYTHONPATH accordingly when invoking packapp. This way the CLI dependencies would not be visible in pip freeze.

[ankitkumarr]

@elprans, thanks for the information. That makes sense. Your fix sounds good to me. Maybe we can wait for @ahmelsayed to pitch in.
The only other workaround I can think of is using pip install -e git+https://github.com/vsajip/distlib.git@15dba58a827f56195b0fa0afe80a8925a92e8bf5#egg=distlib,
this will make pip freeze have the correct entry -e git+https://github.com/vsajip/distlib.git@15dba58a827f56195b0fa0afe80a8925a92e8bf5#egg=distlib, which makes pip download -r requirements.txt work. However, I think we should try to not have the CLI dependencies show up when using pip freeze.
Or, if it's possible to package/bundle the required dependencies along with the packapp.py.

[asavaritayal]

@elprans can you make a PR for this fix?

[elprans]

@asavaritayal The fix needs to be made in the C# portion of the CLI code. I'd like to defer to @ahmelsayed on this.

[asavaritayal]

Quick update - Since @ahmelsayed is out of office, @ankitkumarr is going to take a stab at the PR.

[ankitkumarr]

I think maybe even simpler approach without modifying the environment variable would be to put packapp.py in a directory (perhaps packapp) and rename it to __main__.py. We could then download distlib in the same directory (packapp) under the directory distlib and expand it (such that distlib's __init__.py is directory under directory packapp/distlib).

The structure would be like-
.
|---- packapp/
            |---- __main__.py (packapp code)
            |---- distlib/
                        |---- __init__.py (distlib's code)
                        |---- .... (distlib's code)

With this configuration, we can simply run python packapp [args] and it should work!
I think ideally we should do this when we build the azure-functions-core-tools project. But for now, we can either change our pip command to achieve this or simply put it like this in the repo.

@elprans - would love to hear your thoughts and if you know the best change to the pip command for this?

[elprans]

@ankitkumarr Yes, vendoring distlib and wheel like that is an option. If you want the vendoring to be a build-time thing, use pip install --target <dir> distlib wheel

[ahmelsayed]

With @ankitkumarr change in #724, I'm still seeing the same error on publish.

ERROR: ['pip', 'download', '-r', '/tmp/f_app/requirements.txt', '--dest', '/tmp/azureworkerz366tkpj'] failed with exit code 1

[ankitkumarr]

@ahmelsayed did you create a new app and then use pip freeze? Or was it an old app that gave the same error? If it's an old app, you may have to remove distlib from requirements.txt manually.

[ahmelsayed]

Yes I did. I was using the cli built from your branch on that PR and did

python3 -m venv temp_venv
source temp_venv/bin/activate
func init temp --worker-runtime python
cd temp
func new 
pip freeze >! requirements.txt
func azure functionapp publish ....

and I'm getting

There was an error restoring dependencies.ERROR: ['pip', 'download', '-r', '/tmp/fofo/requirements.txt', '--dest', '/tmp/azureworker8n1stn5a'] failed with exit code 1

[ahmelsayed]

this is my requirements.txt

azure-functions==1.0.0a4
azure-functions-worker==1.0.0a4
grpcio==1.14.2
grpcio-tools==1.14.2
pkg-resources==0.0.0
protobuf==3.6.1
six==1.11.0

[ankitkumarr]

Looks like the problem is pkg-resources==0.0.0. This is not a valid package version. This seems separate from ditslib, but I will investigate where is this being added from.

[ahmelsayed]

ah sorry, it's an ubuntu bug on my machine. Seems to work after removing that.

[ankitkumarr]

Ok, looks like this is an Ubuntu bug where pip freeze adds pkg-resources==0.0.0 dependency when using a virtualenv. Until they fix it (if they haven't yet), we would either have to make users manually remove it, or packapp should check and remove that specifically from requirements.txt. @ahmelsayed what do think we should do about this?

source: - https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1635463

[ahmelsayed]

I don't think we have to worry about this one. I'll merge your PR

[ankitkumarr]

Closing the issue as this should be fixed by #724

[ahmelsayed]

I actually have a question, what does --target do in pip exactly? packApp path when the cli is installed is under /usr/bin somewhere and to change files there you'll need root.

[elprans]

@ahmelsayed: https://pip.pypa.io/en/stable/reference/pip_install/#cmdoption-t

[ahmelsayed]

@ankitkumarr based on that, I don't think the change you did will work. You can't change anything in the install dir of the cli if it's installed using a package manager since that would require you to sudo the command. Can you change the location for --target to be some temp folder instead?

[ankitkumarr]

@ahmelsayed Oh, I see. I think if we change the --target to some temp folder, the script can't import it directly. We would have to either change the PythonPath, or place the packapp in the same folder. Or change the script to somehow import it manually.

I think the best change would be if we can couple this module with core-tools so it gets installed when the user installs core-tools. Or when we build core-tools, could we download distlib and place it where it needs to be?