Azure-functions-core-tools: Enable Python 3.7 support

[peteroden]

As of the 1.0.0a6 release of the azure-functions-python-worker, Python 3.7 is supported however the CLI throws an error unless Python 3.6 is used. Rather than just adding a value for 3.7 , /src/Azure.Functions.Cli/Helpers/PythonHelpers.cs should probably be refactored to have a hash table of the valid versions or something. @ahmelsayed

[asavaritayal]

The issue is currently blocked on https://github.com/Azure/azure-functions-docker/issues/57

[ankitkumarr]

\cc @asavaritayal, @maiqbal11 @ahmelsayed

The way things are right now, we do not support multiple versions of Python. The update to version 3.7 could be a breaking change.
Might be a good idea to push this with the change needed in core-tools for this.

[ElliotWood]

Are there any known workarounds? Rolling back to 3.6 is currently not an option for the env.

[asavaritayal]

Blocked on version pooling /cc @pragnagopa

[mebibou]

Could we have a roadmap for this? will this be fixed soon? locking down a very specific version is locking out most potential users

[prakashmehrotra]

I am just starting with this and do not understand why tools cannot work with minimum required versions (3.6+) and require a specific version of Python. This is very annoying.

[smpeters]

Could we have a roadmap for this? will this be fixed soon? locking down a very specific version is locking out most potential users

Yep, I planned a project to build a function using Python to learn Azure. Instead, my choice became AWS or Node. Python 3.7 was released almost a year ago. Downgrading my laptop isn't really an option.

[ankitkumarr]

In Azure, Function Apps only run on Python 3.6 right now. We are working on fixing that on the platform. That's why func enforces you to use Python 3.6, to be sure you get a consistent experience locally and in a function app.

You can have Python 3.6 and Python 3.7 installed at the same time locally, and then you can create a virtualenv with Python 3.6 as explained here. This way you wouldn't have to downgrade your Python version.

That said, if this has been a big concern, we could relax the Python 3.6 local enforcement, and display a warning saying that you are using Python 3.7, but your published app will run on Python 3.6. @asavaritayal, what do you think?

[dirkesquire]

As it says that this issue is blocked on Azure/azure-functions-docker#57 and work was removed from the Feb sprint it seems that docker is a bit limited, if it takes over a year to get it to be compatible with Python 3.7. Is it just python it is so inflexible with, or other languages as well?

Azure has always had this problem of being too tightly coupled to particular python versions. Clearly the way this works currently requires too much work to maintain.

My suggestion I've made else where, is that we work with the existing pyenv tool to solve this, instead of some manual hardcoding, bespoke deployment file, or azure setting. Pyenv works by looking for a local .pyenv file to specify the python version, and if not there it uses the system version of python. The work that may be needed is to integrate the azure-functions-docker to respect and work with pyenv. Such as scanning for the .pyenv file and invoking the pyenv command line. Setting versions or downloading new versions of python, is a single command line in pyenv. This will make python lovers happy as it is a familar tool and approach.

pyenv install 3.7.0      # Download new version of python
pyenv local 3.7.0        # Tell current project to use this version (and creates a local .pyenv file)

[mswilson4040]

Any further update on when Python 3.7 will be supported for Azure Functions?

[ariciputi]

Hi,
I'm also interested in having Python 3.7 available as runtime for Azure Functions. I'm on a project at work where reverting from 3.7 to 3.6 may cost too much to take Functions into account.

I don't want to sound harsh here, but I also want to stress that Python 3.8 is expected to be released in 3 months from now and that from Python 3.9 onward the Python's release cycle is expected to be shortened to 9 months. Given the fact the present issue has been around for almost 10 months now (and that one of your major competitors is supporting v3.7 since last November), I would urge Azure Functions' team to find either a better way to support the Python runtime or just declare it as second citizen openly.

[ankitkumarr]

I understand your frustration. Right now, our platform constraints us to only support one specific version of Python. When we last surveyed, there were several packages that did not have support for Python 3.7. So we went ahead with Python 3.6.
I can assure you that we have plans for adding support to use multiple versions of Python. This will enable us very quickly adapt and support new versions, and not make our users wait so long.

I can't speak for the timeline specifically. @anirudhgarg may have a better idea.

[mswilson4040]

Thanks, @ankitkumarr. It'd be great to have some sort of timeline for this.

[gbmarc1]

@ankitkumarr. Is there any timeline for this? My organisation needs 3.7 for deployment purposes.

[dirkesquire]

As far as I can see the hold up is caused by Docker.

I also want to draw attention the handy tool that is pyenv which allows a system to easily install new python versions on the command line. So by checking in a .python-version file (that pyenv automatically creates), this is a fantastic way to control the python version that we are targeting.

After that it's a simple matter for a system to read the .python-version and run
pyenv install <version>

[memark]

@ankitkumarr
Any further update on when Python 3.7 will be supported for Azure Functions?

Python 3.6 is now only available as source code, making the step even greater for people wanting to start out with Azure Functions.

https://www.python.org/downloads/release/python-369/

Python 3.6 has now entered the security fix phase of its life cycle. Only security-related issues are accepted and addressed during this phase. We plan to provide security fixes for Python 3.6 as needed through 2021, five years following its initial release. Security fix releases are produced periodically as needed and only provided in source code form; binary installers are not provided.

[ariciputi]

Just saying...

[ahmedelnably]

We are still working on the support for Python 3.7, it should be released by/before the end of this calendar year

[memark]

@ahmedelnably, that's great! And you mean "3.7 and newer", right?

As many have pointed out, 3.8 is just around the corner and 3.7 will soon be what 3.6 is today.

https://www.python.org/dev/peps/pep-0537/

3.7.8 schedule (last bugfix release)
Last binary releases
3.7.8 candidate 1: 2020-06-15 (expected)
3.7.8 final: 2020-06-27 (expected)

[ahmedelnably]

Support is for 3.7 - will let @kulkarnisonia16 comment on 3.8 plans

[dirkesquire]

Any comment on the suggestions made above to use pyenv to make docker and azure-functions a bit more flexible in the future with regards to python version? @kulkarnisonia16
I use this myself everyday and it is such a handy and easy tool which I think will ultimately save the maintenance team a lot of time 😃

[ahmelsayed]

we used to use pyenv, but I couldn't find out how to reduce the image size. pyenv builds python from source, which requires these dependencies: https://github.com/pyenv/pyenv/wiki/Common-build-problems

This Dockerfile

FROM debian:stretch

RUN apt-get update && \
    apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \
    libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
    xz-utils tk-dev libffi-dev liblzma-dev python-openssl git bash

RUN curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash

ENV PYENV_ROOT=/root/.pyenv \
    PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

RUN pyenv install 3.6.9 && \
    pyenv global 3.6.9

results in a 969 MB image. python:3.6-stretch is also 936 MB, but python:3.6-slim-stretch is 151 MB, which is the base image we use. Additional packages are tracked differently so that we know what packages we're bringing in.

[kummerer94]

With this update [1] saying that Azure Functions now supports python 3.7, what is the status of this and could you maybe also provide some documentation to where I can let Functions know that I want to use 3.7?

Thank you!

[1] https://azure.microsoft.com/en-us/updates/python-3-7-support-and-remote-build-are-now-generally-available/

[ahmelsayed]

I think this can be closed, right @ankitkumarr ?

[ankitkumarr]

Yes! As stated above by @kummerer94, Python 3.7 should now be supported. I am going to close this issue

To create an Azure Function in Azure for Python 3.7, you can use Azure CLI (even from cloud shell) something like --
az functionapp create --resource-group myResourceGroup --os-type Linux --consumption-plan-location westeurope --runtime python --runtime-version 3.7 --name <APP_NAME> --storage-account <STORAGE_NAME>

And the new version of azure-functions-core-tools should work fine for Python 3.7 local development as well as deploying to a 3.7 function app.

@kulkarnisonia16, is the --runtime-version flag (way to create Python 3.7 app) documented somewhere?

[memark]

Great work guys! 👍

Support is for 3.7 - will let @kulkarnisonia16 comment on 3.8 plans

Did @kulkarnisonia16 have a comment? Do you guys want us to create a separate issue for supporting 3.8, or will you be working on upcoming versions from now on?

[kummerer94]

@ankitkumarr Thank you for the quick response! I found that this is documented in the Azure CLI documentation (https://docs.microsoft.com/en-us/cli/azure/functionapp?view=azure-cli-latest#az-functionapp-create).

What I originally tried to do was configure this through the Azure Portal UI. I was not able to find the setting but I did not search thorougly. Thank you again for your help!

[ankitkumarr]

@kulkarnisonia16 is unavailable this week from what I know.

@kummerer94, thanks for linking that. We also need to update documentation in the create Python function app flows.
I believe Azure Portal has not yet been updated to allow creating function apps with versions other than the defaults.

[jongio]

Getting this today @ankitkumarr

Python 3.6.x is required, and used in Azure Functions. You are using Python version Python 3.7.5. Please install Python 3.6, and use a virtual environment to switch to Python 3.6.

Azure Functions Core Tools (3.0.1740 Commit hash: 0b841ce061d005a05af8bb57a341989788266298)
Function Runtime Version: 3.0.12785.0

VS Code:

Version: 1.41.0-insider (user setup)
Commit: 3b17b7943842e84912a5507111a6cd790db18a46
Date: 2019-11-18T16:15:22.989Z
Electron: 6.1.4
Chrome: 76.0.3809.146
Node.js: 12.4.0
V8: 7.6.303.31-electron.0
OS: Windows_NT x64 10.0.18363

[ahmelsayed]

we need to merge the v2 changes into the v3 branch.

[dxkaufman]

I am seeing that my azure function is using 3.6 when in Azure. I created the service through the portal and just picked the Python stack. I was never given a choice of what version. @ankitkumarr - is there any way to change the python version of an existing function app?

I'm getting this error. I think it's complaining about cffi because it wasn't in my requirements.txt file when I deployed it, but I know it was installed with something else, I think when I ran it locally.

2019-11-26T05:30:28.180144717Z Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.KFAccess ---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException: Result: Failure
2019-11-26T05:30:28.180157217Z Exception: ModuleNotFoundError: No module named '_cffi_backend'
2019-11-26T05:30:28.180162117Z Stack:   File "/azure-functions-host/workers/python/3.6/LINUX/X64/azure_functions_worker/dispatcher.py", line 242, in _handle__function_load_request
2019-11-26T05:30:28.180166517Z     func_request.metadata.entry_point)
2019-11-26T05:30:28.180170417Z   File "/azure-functions-host/workers/python/3.6/LINUX/X64/azure_functions_worker/loader.py", line 66, in load_function
2019-11-26T05:30:28.180174817Z     mod = importlib.import_module(fullmodname)
2019-11-26T05:30:28.180178817Z   File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
2019-11-26T05:30:28.180197117Z     return _bootstrap._gcd_import(name[level:], package, level)
2019-11-26T05:30:28.180202817Z   File "/home/site/wwwroot/KFAccess/KF.py", line 10, in <module>
2019-11-26T05:30:28.180206717Z     from SharedCode import FunctionApp
2019-11-26T05:30:28.180210417Z   File "/home/site/wwwroot/SharedCode/FunctionApp.py", line 2, in <module>
2019-11-26T05:30:28.180214217Z     import azure.storage.blob as azureblob
2019-11-26T05:30:28.180217817Z   File "/home/site/wwwroot/.python_packages/lib/site-packages/azure/storage/blob/__init__.py", line 6, in <module>
2019-11-26T05:30:28.180221617Z     from .appendblobservice import AppendBlobService
2019-11-26T05:30:28.180225417Z   File "/home/site/wwwroot/.python_packages/lib/site-packages/azure/storage/blob/appendblobservice.py", line 30, in <module>
2019-11-26T05:30:28.180229217Z     from ._deserialization import (
2019-11-26T05:30:28.185266522Z   File "/home/site/wwwroot/.python_packages/lib/site-packages/azure/storage/blob/_deserialization.py", line 40, in <module>
2019-11-26T05:30:28.185280822Z     from ._encryption import _decrypt_blob
2019-11-26T05:30:28.185445122Z   File "/home/site/wwwroot/.python_packages/lib/site-packages/azure/storage/blob/_encryption.py", line 13, in <module>
2019-11-26T05:30:28.186310923Z     from cryptography.hazmat.primitives.padding import PKCS7
2019-11-26T05:30:28.186336323Z   File "/home/site/wwwroot/.python_packages/lib/site-packages/cryptography/hazmat/primitives/padding.py", line 13, in <module>
2019-11-26T05:30:28.186341523Z     from cryptography.hazmat.bindings._padding import lib

[ahmedelnably]

Closing as we already merged v3