Openwhisk: Python action snippets' Dockerfile to use Python 3

[iainhouston]

This is a feature / upgrade request

It would be preferable to change the Openwhisk python snippet Dockerfile to FROM python:3.5.2-alpine rather than FROM python:2.7.12-alpine

For a number of reasons including the trivial example issue below.

Rationale

Whilst it may make a lot of sense to Apple, for example, to have OS X default to python 2.7 because OS X / MacOS has a big investment in python 2 system programs, it makes better sense really to have (new) OpenWhisk Python snippets written in python 3.

Environment details:

• Bluemix, Openwhisk wsk action create pyAction pyProgram.py
• (Docker 1.12)
  

  Steps to reproduce the issue:

1. Use the python with statement (available since 2.6 - thus available to Openwhisk python snippets in python 2.7)
   The advantages being that opening and closing of the connection (or file or whatever) is done by the with statement and need not be managed by the programmer
2. Use it with python 2.7 libraries (like smtplib) that have not been upgraded to python 3 and thus have no attribute __exit__
   

   Provide the expected results and outputs:

If it were python 3 this would work without error

from smtplib import SMTP
with SMTP("domain.org") as smtp:     
    smtp.noop()

Provide the actual results and outputs:

In python 2.7 this is generated:

AttributeError: SMTP instance has no attribute '__exit__'

[rabbah]

We can do this the way we support multiple node or swift version, namely, as python (default to latest), python:2.7 and python:3.5.

[iainhouston]

Would it be possible for wsk action to look at a version.txt file for python 3.5 or python 2.7 if present?

[csantanapr]

@rabbah Is the request to support python 3 via the action kind property?

This enhancement will be in in the context of kind for python versions
wsk action create --kind python:3 myaction mysnippet.py
or
wsk action create --kind python:2 myaction mysnippet.py2

[sjfink]

I'd prefer to simply drop python 2.7 rather than support both.

[csantanapr]

I'd prefer to simply drop python 2.7 rather than support both.

Works for me, I was not sure why the runtime got implemented with 2 instead of 3

User can always choose any python version, and any packages they need via docker actions.

[iainhouston]

User can always choose any python version, and any packages they need via docker actions.

My feelings too. Python snippets should be Python 3 and so should Docker actions in the sense that the _blackbox_ uses openwhisk/dockerskeleton to do its thing for actions in any language.

Note that if you write a Docker action in Python today you have to install python3 into the Alpine Linux and pip3 your Python packages... which is an unecessary complexity and opportunity for error and may introduce conflicts with the whisk's own python 2 stuff .

[cclauss]

OK... We are now tantalizingly close on this one. Recent PRs have done much to get the codebase to be compatible with both Python 2 and Python 3. Note: this repo is ~90% Scala and only 1.5% Python but it does use Python is a few key areas.

We created a single Docker image that contained both Python 2 and Python 3 but that proved to be too big for daily use. Therefore the sense is that we should transition to a world where OpenWhisk majors in (defaults to) Python 3 and minors in Python 2.

We would make dockerskeleton into a Python 3 container.

For those who use the wsk syntax:

wsk action create myaction mysnippet.py  # use dockerskeleton-based container
wsk action create --kind python:3 myaction mysnippet.py  # use dockerskeleton-based container
wsk action create --kind python:2 myaction mysnippet.py  # use python:2.7.12-alpine-based container

For those who do _not_ use the CLI, OpenWhisk will just assume their code is Python 3 to be deployed on the Python 3-based dockerskeleton container. It is not considered Pythonic to change file suffixes to .py2 and .py3 and reading file shebang lines is probably not worth the effort.

TODOs (in rough order of execution):

• [x] Finish making the code compatible with both Python 2 and Python 3
  
  
  
  • [x] https://github.com/openwhisk/openwhisk/pull/1987 (done)
  
  
  • [x] https://github.com/openwhisk/openwhisk/pull/2006 (done)
  
  
  • [x] https://github.com/openwhisk/openwhisk/pull/1973 (closed in 2006)
  
  
  • [x] https://github.com/openwhisk/openwhisk/pull/2007 (done)
  
  
• [ ] Move dockerskeleton from Python 2.7.12 to 2.7.13 and update pip packages and docs
  
  
  
  • [ ] https://github.com/openwhisk/openwhisk/pull/1903 (ready for review)
  
  
• [ ] Create a legacy Python docker image which matches the current dockerskeleton
  
  
  
  • core/python2Action is now added to 1903
  
  
• [ ] Move dockerskeleton to Python 3.6 and update docs (go live)
• [ ] Modify wsk to accept --kind python:2 and --kind python:3
• [ ] Monitor usage and decide when to shut down legacy Python support

Also to consider: https://github.com/openwhisk/openwhisk/pull/1940 (venv vs. virtualenv), https://github.com/openwhisk/openwhisk/pull/1867, and https://github.com/openwhisk/openwhisk/pull/1510

Are there any Python 3-only PyPI modules that we should consider adding to the new dockerskeleton to provide folks an incentive to switch? I would vote for aiohttp (like requests on asyncio).

[cclauss]

@rabbah @mrutkows Thanks to the merging of recent PRs, flake8 now finds no Python 3 syntax errors in this repo. #1903 now adds a new core/python2Action with appropriate Dockerfile and build.gradle to allow us to continue to run Python 2 actions after the Python 3 upgrade.

Now that #2007 is merged, we have two choices:

1. Review and merge 1903 which would be minor a Python 2.7.12 to 2.7.13 (plus modules) upgrade that also creates the new core/python2Action --or--
2. Go the "big bang" approach and upgrade core/actionProxy and core/pythonAction to Python 3.

Thoughts?

• [ ] Modify wsk to accept --kind python:2 and --kind python:3
  What is required to make that happen?

[iainhouston]

Without understanding all the implications (I hadn't realised how much work would be involved for 1.5% of Openwhisk code!) of going for the "Big Bang", it would seem to me preferable to upgrade core/actionProxy and core/pythonAction to Python 3.

Need for a --kind python:n option? Do we have any idea what impact this would have - i.e. can we quantify the number of applications' Openwhisk Containers in use that would require a --kind python:n option? i.e. be adversely impacted by things going over to Python 3

[iainhouston]

BTW it seems quite common for people to

import asyncio 
import aiohttp 
import requests 

So if these libraries were available by default it would give a good range of functionality to applications deployed to an Openwhisk Container

[cclauss]

Will make that happen.

Asyncio is built into Python 3.
aiohttp is awesome!
requests is essential (Don't leave home without it)

[cclauss]

Update on our status for moving to Python 3... We have decided to go big bang (direct from Python 2.7.12 to 3.6.0) which makes it harder but we are still making progress. The checklist (in rough order of execution) now looks like this:

• [x] #2015 Fixes those Python files that the flake8 linter missed because their filenames do not end in .py This also implements flake8 tests that halt the build on Python syntax errors (ready for review)
• [x] #1867 _Closed 11 of 13_ failing tests when running under Python 3
• [x] #2020 Is a minimal change attempt to highlight the two remaining failing tests. ~Help needed here.~ If we can get that one to pass all tests then we can go back to the original #1903
• [ ] #1903 Our go live on Python 3 PR

Others Python (non-blocking) issues I am still monitoring include: #1940 and #1510

[cclauss]

Update on the journey getting Open Whisk to major on Python 3 and minor on Python 2...

~#1867~ solved most of the problems and ~#2044~ demonstrated the solutions to the remaining ones.

The checklist (in rough order of execution) now looks like this:

• [x] #2015 (approved)
• [x] #2047 Python 3 syntax errors in a pull request will fast fail the Travis build in the first minute.
• [ ] PR TBD to establish kind: Python (defaults to Python 3), kind: Python:3, kind: Python:2
• [ ] #1903 Our go live on Python 3 PR

[cclauss]

Update on the journey getting Open Whisk to major on Python 3 and minor on Python 2...

• ~#2015~ and ~#2047~ (via ~#2052~) are now all checked in (thanks @rabbah)
  
  
  
  • This means that there are no more Python 3 syntax errors in the codebase and that Travis will now fast fail any pull request that contains Python 3 syntax errors.
  
  

The checklist (in rough order of execution) now looks like this:

• [x] #2053 Remove 'kinds' from CLI (pg2/1265)
• [x] #1903 Our go live on Python 3 PR -- This touches more than a dozen files so please review it carefully

[rabbah]

Closed by #1903 and commit https://github.com/openwhisk/openwhisk/commit/3a738e63b5bf4e0b042b3f8e5e5c5c063959d4c5.