Build-image: Add support for Python 3.8

[Pieparker]

Python 3.8 has been around since last year and it'd be nice to it supported in our (Xenial) build image.

[Pieparker]

Requested by:
https://community.netlify.com/t/python-3-8-and-ignore-pipfile/15063/7 - community

[erans]

Python 3.8 has been around since Oct/2019 and its time to support it. I know the build image now support Python 2.7 (which is not supported any more) up until Python 3.7.

However, we don't really need Python at all :-) I think what we need is a way to skip the Pipfile detection because our repo has the OpenAPI files that we want to use Netlify to publish our API documentation using redoc-cli. Since the actual server code uses these files they sit right next to the server, which happens to be written in Python and have a Pipfile with Python 3.8 on it - that's why our build fails.

So I think there might be a case for an env flag that disables Pipfile detection altogether.

[gwk]

@erans just a suggestion: it might be worth opening a separate issue regarding your need for a flag.

@Pieparker I have been waiting for python 3.8 support for many months, and I'm starting to think that 3.9 will be out before Netlify even looks into it. I want to be your customer, and I've contacted multiple people about this problem, but I'm beginning to think I should just go elsewhere. Is there any company update on support or interest for modern Python?

[fool]

@erans there is a separate feature request for that, which you might be interested in: https://github.com/netlify/build-image/issues/398

[DeadWisdom]

I'm writing a tutorial for Flask developers to build static sites and push to Netlify. It will be seen by many.

It would be awesome if some of the basic build ops tools are available by default simply because it's easier for learners. Python, Ruby, Node, etc. We should have the latest versions.

Please make this more than low-priority.

[fool]

@gwk Sorry to report that we can't just slam everything that is requested into the build image. For transparency's sake, let me explain some reasons why we don't implement everything that is requested, even if it makes obvious sense, and even if a PR is provided:

• the testing and rollout of changes to our build image take substantial staff and wall clock time - this is:
  
  
  
  • do our best not to break anything else as a side effect
  
  
  • do a lot of manual testing/confirmation of results (we have automated what we can, but there are a lot of "do the logs look meaningfully different when the build result is the same? What edge cases do we need to consider?" type concerns that aren't automatable)
  
  
  • allow the changes to bake in in testing for awhile
  
  
  • monitor the results once it goes live - there are frequently additional edge cases we didn't consider, and our builds log many millions of lines a day to sift through looking for them and differentiate them from user error.
  
  
• anything we add to the image is basically impossible to ever remove without breaking a lot of folks' builds, and worse..
• everything we add to the image increases its size, which slows down EVERY BUILD across our system. Python installations are not tiny, and cramming more in there does help a few people. But most people do not use python, so having them all wait an additional 2 seconds with every build, for a tool that a handful (percentagewise, python use in builds is far under 1% of our builds as far as I can tell) of folks will benefit from - well, the math doesn't look good - good of the few does not outweigh good of everyone.

So - while this is a "not this moment" answer - the build team are considering a better way to handle dependencies like these. For instance: python should instead work like node does! You pick a version, any version - we install it for you. But, of course, we don't want to wait for it to download from the internet over and over again, so we need a version cache that can be shared across builds (and accounts) securely.

The team is working on these longer term goals, but "just upgrade" isn't "just" on our side. It's got a lot of pluses and minuses and a ton of effort. We will get there eventually, and we will follow up here when we do.

[erans]

I see and understand. My specific concern was that the image build detects
our Pipfile and tries to install it. It fails because we use Python 3.8.
We don't even need Python to build the API documentation part that we use
so having a flag to skip this check is all that we need :-)

On Fri, Aug 14, 2020 at 11:30 AM chris (fool) mccraw <
[email protected]> wrote:

@gwk https://github.com/gwk Sorry to report that we can't just slam
everything that is requested into the build image. For transparency's sake,
let me explain some reasons why we don't implement everything that is
requested, even if it makes obvious sense, and even if a PR is provided:

• the testing and rollout of changes to our build image take
  substantial staff and wall clock time - this is:
  
  
  
  • do our best not to break anything else as a side effect
  
  
  • do a lot of manual testing/confirmation of results (we have
    
    automated what we can, but there are a lot of "do the logs look
    
    meaningfully different when the build result is the same? What edge cases
    
    do we need to consider?" type concerns that aren't automatable)
  
  
  • allow the changes to bake in in testing for awhile
  
  
  • monitor the results once it goes live - there are frequently
    
    additional edge cases we didn't consider, and our builds log many millions
    
    of lines a day to sift through looking for them and differentiate them from
    
    user error.
  
  
• anything we add to the image is basically impossible to ever remove
  without breaking a lot of folks' builds, and worse..
• everything we add to the image increases its size, which slows down
  EVERY BUILD across our system. Python installations are not tiny, and
  cramming more in there does help a few people. But most people do not use
  python, so having them all wait an additional 2 seconds with every build,
  for a tool that a handful (percentagewise, python use in builds is far
  under 1% of our builds as far as I can tell) of folks will benefit from -
  well, the math doesn't look good - good of the few does not outweigh good
  of everyone.

So - while this is a "not this moment" answer - the build team are
considering a better way to handle dependencies like these. For instance:
python should instead work like node does! You pick a version, any version

• we install it for you. But, of course, we don't want to wait for it to
  download from the internet over and over again, so we need a version cache
  that can be shared across builds (and accounts) securely.

The team is working on these longer term goals, but "just upgrade" isn't
"just" on our side. It's got a lot of pluses and minuses and a ton of
effort. We will get there eventually, and we will follow up here when we do.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/netlify/build-image/issues/431#issuecomment-674206641,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAACUE672OZXKZOCFXQHWKLSAV7DDANCNFSM4NH5PA7Q
.

[fool]

Yup @erans - and that's why I suggested you follow that other feature request since it's the more relevant one - you want to disable python, not have us waste time installing unused dependencies ;)

[gwk]

@fool thank you for the detailed explanation. This kind of communication helps a great deal. I completely agree that ideally any python version you want could be installed on the fly. I just hope that Netlify recognizes its chicken-or-egg customer problem here, and that it is getting worse as the versions tick by. You have only 1% of your customers using python in part because from the perspective of a python developer, it's a severely limited environment. Thanks again!

[Stanzilla]

Any ETA?

[wlach]

The Ubuntu 20.04 beta image includes Python 3.8 🎉 https://github.com/netlify/build-image/blob/focal/CHANGELOG.md#400-2021-04-29

[ingride]

As @wlach mentioned, our newest release V4.0.0 has Python 3.8. At this time, we want to encourage you to upgrade to it if you require 3.8.

Let us know if you encounter any issues and we will be happy to help.