Azure-functions-core-tools: Blocked: publishing a Python Function always fails

Created on 25 Oct 2018 · 13Comments · Source: Azure/azure-functions-core-tools

This has happened on both Windows and Linux hosts. I am currently unable to deploy this Function (or is there a manual work-around?

Linux:

16.04.1-Ubuntu
Python 3.6.7 (default, Oct 21 2018, 04:56:05)
[GCC 5.4.0 20160609] on linux
pip 18.1 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)
func: 2.1.748
dotnet: 2.1.403

Windows:

Windows 10 Pro 1809 - 17763.104
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit (Intel)] on win32
pip 10.0.1 from c:\users\simon\appdata\local\programs\python\python36-32\lib\site-packages\pip (python 3.6)
func: 2.1.748
dotnet: 2.1.403

I can run and debug the Python function locally, but when I try to publish

func azure functionapp publish bioutils1 --force

I get the following experience.

Getting site publishing info...
pip download -r /home/simonw/bioutil-api-funcs/requirements.txt --dest /tmp/azureworker_m5f2yof
pip download --no-deps --only-binary :all: --platform manylinux1_x86_64 --python-version 36 --implementation cp --abi cp36m --dest /tmp/azureworkereti8hs90 azure_mgmt_notificationhubs==2.0.0
pip download --no-deps --only-binary :all: --platform manylinux1_x86_64 --python-version 36 --implementation cp --abi cp36m --dest  /tmp/azureworkereti8hs90 azure_mgmt_eventhub==2.1.0
pip download --no-deps --only-binary :all: --platform manylinux1_x86_64 --python-version 36 --implementation cp --abi cp36m --dest /tmp/azureworkereti8hs90 azure_mgmt_containerservice==4.2.2
pip download --no-deps --only-binary :all: --platform manylinux1_x86_64 --python-version 36 --implementation cp --abi cp36m --dest /tmp/azureworkereti8hs90 azure_mgmt_nspkg==3.0.2
pip download --no-deps --only-binary :all: --platform manylinux1_x86_64 --python-version 36 --implementation cp --abi cp36m --dest /tmp/azureworkereti8hs90 grpcio==1.14.2
pip download --no-deps --only-binary :all: --platform manylinux1_x86_64 --python-version 36 --implementation cp --abi cp36m --dest /tmp/azureworkereti8hs90 azure_nspkg==3.0.2
pip download --no-deps --only-binary :all: --platform manylinux1_x86_64 --python-version 36 --implementation cp --abi cp36m --dest /tmp/azureworkereti8hs90 azure_mgmt_trafficmanager==0.50.0
pip download --no-deps --only-binary :all: --platform manylinux1_x86_64 --python-version 36 --implementation cp --abi cp36m --dest /tmp/azureworkereti8hs90 azure_mgmt_iothubprovisioningservices==0.2.0
pip download --no-deps --only-binary :all: --platform manylinux1_x86_64 --python-version 36 --implementation cp --abi cp36m --dest /tmp/azureworkereti8hs90 cryptography==2.3.1
pip wheel --no-deps --no-binary :all: --wheel-dir /tmp/azureworkerhu7ffsu7 cryptography==2.3.1

There was an error restoring dependencies.ERROR: cannot install cryptography-2.3.1 dependency: binary dependencies without wheels are not supported.  Use the --build-native-deps option to try building the binary dependencies using a Docker container.

I also have the same outcome when I run this process using --build-native-deps in Docker.

Source

sjwaight

All 13 comments

Hi @sjwaight,

Do you get the exact same error when using --build-native-deps? If not, could you share your console output for the error you get then?

And can you please also share your requirements.txt?

Thanks!

ankitkumarr on 25 Oct 2018

To make sure it wasn't something I did in my own code I went back and followed the guide to create a Python Function on Linux (16.04.1-Ubuntu).

There seems to be a lot of packages being imported on 'func new' that seem entirely unnecessary (in my scenario I will use the storage 0.36.0 library and not much else).

This is the default requirements.txt generated by 'func new' > HTTPTrigger and which I can't deploy:

adal==1.2.0
asn1crypto==0.24.0
astroid==2.0.4
azure==4.0.0
azure-applicationinsights==0.1.0
azure-batch==4.1.3
azure-common==1.1.16
azure-cosmosdb-nspkg==2.0.2
azure-cosmosdb-table==1.0.5
azure-datalake-store==0.0.34
azure-eventgrid==1.2.0
azure-eventhub==1.1.1
azure-functions==1.0.0a5
azure-functions-worker==1.0.0a6
azure-graphrbac==0.40.0
azure-keyvault==1.1.0
azure-loganalytics==0.1.0
azure-mgmt==4.0.0
azure-mgmt-advisor==1.0.1
azure-mgmt-applicationinsights==0.1.1
azure-mgmt-authorization==0.50.0
azure-mgmt-batch==5.0.1
azure-mgmt-batchai==2.0.0
azure-mgmt-billing==0.2.0
azure-mgmt-cdn==3.0.0
azure-mgmt-cognitiveservices==3.0.0
azure-mgmt-commerce==1.0.1
azure-mgmt-compute==4.3.1
azure-mgmt-consumption==2.0.0
azure-mgmt-containerinstance==1.2.1
azure-mgmt-containerregistry==2.3.0
azure-mgmt-containerservice==4.2.2
azure-mgmt-cosmosdb==0.4.1
azure-mgmt-datafactory==0.6.0
azure-mgmt-datalake-analytics==0.6.0
azure-mgmt-datalake-nspkg==3.0.1
azure-mgmt-datalake-store==0.5.0
azure-mgmt-datamigration==1.0.0
azure-mgmt-devspaces==0.1.0
azure-mgmt-devtestlabs==2.2.0
azure-mgmt-dns==2.1.0
azure-mgmt-eventgrid==1.0.0
azure-mgmt-eventhub==2.1.0
azure-mgmt-hanaonazure==0.1.1
azure-mgmt-iotcentral==0.1.0
azure-mgmt-iothub==0.5.0
azure-mgmt-iothubprovisioningservices==0.2.0
azure-mgmt-keyvault==1.1.0
azure-mgmt-loganalytics==0.2.0
azure-mgmt-logic==3.0.0
azure-mgmt-machinelearningcompute==0.4.1
azure-mgmt-managementgroups==0.1.0
azure-mgmt-managementpartner==0.1.0
azure-mgmt-maps==0.1.0
azure-mgmt-marketplaceordering==0.1.0
azure-mgmt-media==1.0.0
azure-mgmt-monitor==0.5.2
azure-mgmt-msi==0.2.0
azure-mgmt-network==2.2.1
azure-mgmt-notificationhubs==2.0.0
azure-mgmt-nspkg==3.0.2
azure-mgmt-policyinsights==0.1.0
azure-mgmt-powerbiembedded==2.0.0
azure-mgmt-rdbms==1.4.1
azure-mgmt-recoveryservices==0.3.0
azure-mgmt-recoveryservicesbackup==0.3.0
azure-mgmt-redis==5.0.0
azure-mgmt-relay==0.1.0
azure-mgmt-reservations==0.2.1
azure-mgmt-resource==2.0.0
azure-mgmt-scheduler==2.0.0
azure-mgmt-search==2.0.0
azure-mgmt-servicebus==0.5.2
azure-mgmt-servicefabric==0.2.0
azure-mgmt-signalr==0.1.1
azure-mgmt-sql==0.9.1
azure-mgmt-storage==2.0.0
azure-mgmt-subscription==0.2.0
azure-mgmt-trafficmanager==0.50.0
azure-mgmt-web==0.35.0
azure-nspkg==3.0.2
azure-servicebus==0.21.1
azure-servicefabric==6.3.0.0
azure-servicemanagement-legacy==0.20.6
azure-storage==0.36.0
azure-storage-blob==1.3.1
azure-storage-common==1.3.0
azure-storage-file==1.3.1
azure-storage-nspkg==3.0.0
azure-storage-queue==1.3.0
certifi==2018.10.15
cffi==1.11.5
chardet==3.0.4
colorama==0.4.0
cryptography==2.3.1
entrypoints==0.2.3
grpcio==1.14.2
grpcio-tools==1.14.2
idna==2.7
isodate==0.6.0
isort==4.3.4
jeepney==0.4
keyring==15.1.0
lazy-object-proxy==1.3.1
mccabe==0.6.1
msrest==0.6.1
msrestazure==0.4.34
oauthlib==2.1.0
protobuf==3.6.1
ptvsd==4.1.4
pycparser==2.19
PyJWT==1.6.4
pylint==2.1.1
python-dateutil==2.7.3
pywin32-ctypes==0.2.0
requests==2.20.0
requests-oauthlib==1.0.0
SecretStorage==3.1.0
six==1.11.0
typed-ast==1.1.0
uamqp==1.0.3
urllib3==1.24
wrapt==1.10.11

Locally I still get the same issue:

There was an error restoring dependencies.ERROR: cannot install cryptography-2.3.1 dependency: binary dependencies without wheels are not supported. Use the --build-native-deps option to try building the binary dependencies using a Docker container.

It the Docker build `--build-native-deps' I now have the 'stack trace' with an error as attached.

stack-trace.txt

sjwaight on 26 Oct 2018

@sjwaight,
I suspect that you used your old venv to create the new function which is why you see an extended requirements.txt. When a new project is created using func init, we run a pip freeze > requirements.txt on your venv. So, please try to use a clean and fresh venv for every new project.

Aside from that, looks there was an error while packing with pyinstaller (--build-native-deps). I created a PR to fix that issue. Once it is in, please verify and let us know if you are still unable to successfully publish as well as run the function.

ankitkumarr on 27 Oct 2018

👍1

Just so I'm clear, you are saying that every time I create a new Function App (which may contain multiple Functions) I should do these steps (on Linux for example):

python3.6 -m venv env
source env/bin/activate

I'm not sure the current state of VS Code support either as it also seems to make some changes when you open the folder - says "this project wasn't initialised using VS Code" and prompts to initialise it - I assume this isn't breaking anything.

Regardless I will try the above suggestion.

sjwaight on 27 Oct 2018

Just so I'm clear, you are saying that every time I create a new Function App (which may contain multiple Functions) I should do these steps (on Linux for example):

Yes, that is correct. Also, your issue should be fixed in the master branch.

And yes, I don't think VS Code initializing should break anything.

Thanks for reporting your issue, please let us know if you still face any problems.

ankitkumarr on 29 Oct 2018

Please note this is still a problem. This Function runs fine locally and debugs no issues (on Windows). I'm trying to build / deploy on a Linux host in Azure as I'm unsure how big the deployment package will be and I don't have a large outbound pipe to deploy from so I'm cloning my repository and building there.

Requirements.txt:

azure-functions==1.0.0a5
azure-functions-worker==1.0.0a6
grpcio==1.14.2
grpcio-tools==1.14.2
protobuf==3.6.1
six==1.11.0
azure-storage==0.36.0
azure-cosmos==3.0.1
pathlib==1.0.1

If I try and deploy this on a Linux box I get:

```Getting site publishing info...
pip download -r /home/simonw/func/sss/createjob/requirements.txt --dest /tmp/azureworkerbi8udhsn
pip download --no-deps --only-binary :all: --platform manylinux1_x86_64 --python-version 36 --implementation cp --abi cp36m --dest /tmp/azureworkers_xii5e4 grpcio==1.14.2
pip download --no-deps --only-binary :all: --platform manylinux1_x86_64 --python-version 36 --implementation cp --abi cp36m --dest /tmp/azureworkers_xii5e4 azure_nspkg==3.0.2
pip download --no-deps --only-binary :all: --platform manylinux1_x86_64 --python-version 36 --implementation cp --abi cp36m --dest /tmp/azureworkers_xii5e4 cryptography==2.3.1
pip wheel --no-deps --no-binary :all: --wheel-dir /tmp/azureworkerxvjtxhma cryptography==2.3.1

There was an error restoring dependencies.ERROR: cannot install cryptography-2.3.1 dependency: binary dependencies without wheels are not supported. Use the --build-native-deps option to try building the binary dependencies using a Docker container.
```

I've ensured I have all the local bits required for the crypto library but it seems to fail no matter what I do. This seems unusual (note: I'm not a heavy duty Python dev).

Docker container also fails with this error:

ERROR: Hidden import '_cffi_backend.cpython-36m-x86_64-linux-gnu' not found.

sjwaight on 2 Nov 2018

👍1

That's odd!
I will look into the problem. Could you try using --build-native-deps and --no-bundler flags? I think that should solve this issue.

ankitkumarr on 2 Nov 2018

Can reproduce. Unable to deploy.
Build command ". ${config:azureFunctions.pythonVenv}/bin/activate && func pack --build-native-deps --no-bundler --additional-packages \"python3-dev libevent-dev unixodbc-dev build-essential libssl-dev libffi-dev python-dev \" "

requirements.txt

azure-functions-worker==1.0.0a4
azure-functions==1.0.0a4
cryptography
grpcio-tools==1.14.2
grpcio==1.14.2
pipreqs
protobuf==3.6.1
python-dateutil
pyyaml
six==1.11.0

Build output:

11982 INFO: Loading module hook "hook-cryptography.py"...
Traceback (most recent call last):
  File "/python_bundle_script.py", line 56, in <module>
    __main__()
  File "/python_bundle_script.py", line 53, in __main__
    run(["--paths=" + packages_location, "--distpath=.", *hidden_modules_commands, "--name=worker-bundle", entry_file])
  File "/root/.pyenv/versions/3.6.6/lib/python3.6/site-packages/PyInstaller/__main__.py", line 111, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "/root/.pyenv/versions/3.6.6/lib/python3.6/site-packages/PyInstaller/__main__.py", line 63, inrun_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "/root/.pyenv/versions/3.6.6/lib/python3.6/site-packages/PyInstaller/building/build_main.py", line 838, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "/root/.pyenv/versions/3.6.6/lib/python3.6/site-packages/PyInstaller/building/build_main.py", line 784, in build
    exec(text, spec_namespace)
  File "<string>", line 17, in <module>
  File "/root/.pyenv/versions/3.6.6/lib/python3.6/site-packages/PyInstaller/building/build_main.py", line 241, in __init__
    self.__postinit__()
  File "/root/.pyenv/versions/3.6.6/lib/python3.6/site-packages/PyInstaller/building/datastruct.py", line 158, in __postinit__
    self.assemble()
  File "/root/.pyenv/versions/3.6.6/lib/python3.6/site-packages/PyInstaller/building/build_main.py", line 500, in assemble
    module_hook.post_graph()
  File "/root/.pyenv/versions/3.6.6/lib/python3.6/site-packages/PyInstaller/building/imphook.py", line 410, in post_graph
    self._load_hook_module()
  File "/root/.pyenv/versions/3.6.6/lib/python3.6/site-packages/PyInstaller/building/imphook.py", line 377, in _load_hook_module
    self.hook_module_name, self.hook_filename)
  File "/root/.pyenv/versions/3.6.6/lib/python3.6/site-packages/PyInstaller/compat.py", line 736, in importlib_load_source
    return mod_loader.load_module()
  File "<frozen importlib._bootstrap_external>", line 399, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 823, in load_module
  File "<frozen importlib._bootstrap_external>", line 682, in load_module
  File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 684, in _load
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/root/.pyenv/versions/3.6.6/lib/python3.6/site-packages/PyInstaller/hooks/hook-cryptography.py", line 23, in <module>
    datas = copy_metadata('cryptography')
  File "/root/.pyenv/versions/3.6.6/lib/python3.6/site-packages/PyInstaller/utils/hooks/__init__.py",line 859, in copy_metadata
    dist = pkg_resources.get_distribution(package_name)
  File "/root/.pyenv/versions/3.6.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 472,in get_distribution
    dist = get_provider(dist)
  File "/root/.pyenv/versions/3.6.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 344,in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/root/.pyenv/versions/3.6.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 892,in require
    needed = self.resolve(parse_requirements(requirements))
  File "/root/.pyenv/versions/3.6.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 778,in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'cryptography' distribution was not found and is required by the application

monty-dev on 5 Nov 2018

@monteledwards I think the change with --no-bundler hasn't been released yet. I made some fixes recently, if possible could you try building from the master branch and testing it? If not, a new release should be out soon and this issue should be fixed.

ankitkumarr on 6 Nov 2018

👍1

OK, so I cloned master and built on Linux - invoking the resulting /tmp/cli/func binary with --build-native-deps and --no-bundler published successfully to Azure. My Function fails on invocation in Azure, but that may simply be due to missing environment variables / bugs in my code which I can now fix.

I did have to pip install distlib to start off with but this was without passing any flags and trying to publish from local which still failed anyhow. The local build still failed on the Cosmos SDK this time.

Perhaps Docker publishing should just become the standard method for these Functions types because it seems to be far more reliable. Some documentation around what artefact gets push to Azure would also help given it's hard to know the size which is a problem on small internet connections.

Thanks for the help.

sjwaight on 8 Nov 2018

@sjwaight, Thanks for the update. You also shouldn't need --no-bundler anymore.

You had to do pip install distlib because you were building from the source code. You wouldn't have to do that in any published version.

The Docker publishing is definitely more reliable right now, but we do realize that it is an additional dependency on users. So unless necessary, we try to publish without it. Thanks for the input on that. I think that's something we could potentially do.

ankitkumarr on 8 Nov 2018

Is it possible to do using VS Code, the document talks only of deploying using CLI