Aws-cli: Please support python-rsa 4.0

[jsteel44]

This has been available (stable release) for a month now. Please can you test and bump your requirements? Thank you

[justnance]

@jsteel44 - Thank you for reaching out. We will take a look at this request. I am marking this as a feature request pending further review and 👍 .

[Musikolo]

python-rsa 4.0 was released over two months ago. There are rolling release distros such as Arch Linux that no longer supports python-rsa 3.x. So, having aws-cli requiring python-rsa 3.x is kind of a problem. For instance, stsauth tool, which depends on both python-rsa and aws-cli, cannot be compiled because aws-cli uses version 3.x. More details at:

• https://bugs.archlinux.org/task/60311
• https://github.com/cshamrick/stsauth/issues/12

Is there any chance to get python-rsa 4.0 supported any time soon?

That would be really nice...

Thank you!

[johnnicely]

Also requesting an update to python-rsa 4.0 - there doesn't seem any information about an EOL for python-rsa 3.x, but I assume it's not actively supported. The README for the project also indicates that 4.0 drops support for several modules because they are insecure, so it would be really nice to get an update for this.

[hawkeyej]

Yes, please support the latest RSA package.

[justnance]

All - Thank you for the feedback. There is an formal internal ticket for this feature request however there is no eta on the complete yet.

[jamesls]

Hi everyone, thanks for the feature request. The main issue we're running into is that we still support python2.6 and the latest major version of python-rsa drops support for python2.6: https://github.com/sybrenstuvel/python-rsa/blob/master/CHANGELOG.txt#L14

While I think we will eventually drop support for python2.6, we'll have to come up with a something in the interim. We could use rsa 4.x for >= python2.7, and 3.x for python2.6. I would be a little hesitant about using multiple major versions but from what I can tell, the APIs we're using don't change from 3.x to 4.x, so I think this is our best option for now.

[nikolaik]

What is the reasoning behind keeping support for python2.6, are any versions of linux still supporting python 2.6?

[zt-initech]

Nikolai Røed Kristiansen: RHEL6 IIRC

>

[nikolaik]

Nikolai Røed Kristiansen: RHEL6 IIRC

You are right https://access.redhat.com/support/policy/updates/errata#Life_Cycle_Dates

[hawkeyej]

There is no reason to restrict all users to the lowest common denominator. If someone wants to install the AWS CLI on a modern system, there is no reason that they should be held back by other old OS's.

[tysonclugg]

Nikolai Røed Kristiansen: RHEL6 IIRC

You are right https://access.redhat.com/support/policy/updates/errata#Life_Cycle_Dates

Doesn't that mean that RedHat are paid by subscribers to provide support, including patches? I don't see how AWS CLI needs to support old versions of Python provided by $RANDOM_OLD_DISTRO - let RedHat (and others) earn their subscription fees.

Right now, there are large number of projects that are unable to apply security updates because AWS CLI supports an unsupported version of Python.

Please, drop unsupported versions of Python, and remove the maximum version limits from all requirements so that we may apply security updates in a timely fashion - without waiting for a new version of AWS CLI with updated requirements to be released.

[glaubitz]

Doesn't that mean that RedHat are paid by subscribers to provide support, including patches? I don't see how AWS CLI needs to support old versions of Python provided by $RANDOM_OLD_DISTRO - let RedHat (and others) earn their subscription fees.

RHEL (and SLE) customers still want to use the latest versions of boto and aws-cli so upstream still needs to make sure these packages work fine on these distributions.

Right now, there are large number of projects that are unable to apply security updates because AWS CLI supports an unsupported version of Python.

Define large number of projects? I don't think any serious project or company is running their stuff on a rolling release distribution. I would argue this problem affects some end users running Arch or Tumbleweed only.

Please, drop unsupported versions of Python, and remove the maximum version limits from all requirements so that we may apply security updates in a timely fashion - without waiting for a new version of AWS CLI with updated requirements to be released.

I'm sorry, but enterprise customers weigh way more than a handful of users of rolling release distributions. It's the customers of enterprise distributions or enterprise customers in general who are directly or indirectly paying for the development of this software. Not catering for the needs of these customers would be a rather bad idea.

[tysonclugg]

RHEL (and SLE) customers still want to use the latest versions of boto and aws-cli so upstream still needs to make sure these packages work fine on these distributions.

Any such customers can use to the latest versions of boto/aws-cli by upgrading to a supported version of Python - RHEL 6 Knowledgebase article #92933 details how to do this.

Right now, there are large number of projects that are unable to apply security updates because AWS CLI supports an unsupported version of Python.

Define large number of projects? I don't think any serious project or company is running their stuff on a rolling release distribution. I would argue this problem affects some end users running Arch or Tumbleweed only.

This issue affects all projects using Python 2.7 or newer - ie: the vast majority.

Please, drop unsupported versions of Python, and remove the maximum version limits from all requirements so that we may apply security updates in a timely fashion - without waiting for a new version of AWS CLI with updated requirements to be released.

I'm sorry, but enterprise customers weigh way more than a handful of users of rolling release distributions. It's the customers of enterprise distributions or enterprise customers in general who are directly or indirectly paying for the development of this software. Not catering for the needs of these customers would be a rather bad idea.

The "enterprise" customers that haven't upgraded from Python 2.6 which has been unsupported for over 5 years (since October 2013) are somehow more important than the diligent users who have kept their projects updated? Really???

[tysonclugg]

How many developers are using Python 2.6?

According to the Python Developers Survey 2018 conducted by the Python Software Foundation and JetBrains, Python 2.6 is used by 5% of the Python 2 developers, who in turn make up 16% of the Python developer community (compared with Python 3).

Put simply: 0.8% of Python developers were still using Python 2.6 as surveyed in the fall of 2018.

The last version Python 2.6 was released on June 3rd 2011, and became unsupported in October 2013. Surely it's time to let it go...

[sie-dlanz]

Would it be possible to lockdown a version as the "last py2.6" supported release of awscli, and move on with newer libraries for those on py2.7/3.6/3.7? This is also blocking the PyYAML upgrade, I believe. See https://github.com/aws/aws-cli/issues/3828 for reference.

[tysonclugg]

Would it be possible to lockdown a version as the "last py2.6" supported release of awscli, and move on with newer libraries for those on py2.7/3.6/3.7? This is also blocking the PyYAML upgrade, I believe. See #3828 for reference.

I can see this working if the "last py2.6" supported version removes the maximum version limits from all requirements - then nobody (especially 2.6 users) will remain stuck on outdated packages.

[nethershaw]

Hi, there. I am (or rather, I represent) one of those enterprise customers using enterprise Linux distributions as @glaubitz mentioned. In Amazon EC2. To the tune of... lots. Let's just say _lots._

I'd like to personally dispel this harmful myth that we so-called enterprise folk are so brittle or so fastened to the broken mast of outdated software that we cannot handle life cycle transitions in major dependencies without making the binary -- and false -- choice between something like RHEL and something like, say, Gentoo Linux. There are numerous and quite sane levels of risk between those two extremes that we, as professionals, are willing to accept. Stagnation breeds vulnerability and technical debt on its own, after all, and there is a point of inflection where those risks outweigh the entirely different ones you supposedly avoid by standing still.

But I digress. That is a philosophical point and perhaps worthy of debate -- elsewhere. Suffice it to say that, even as an enterprise customer, _we have no Earthly business using Python 2.6 and we are more than adequately provisioned with the means to categorically abandon it._

It is simply untrue that, in the case of Python 2.7, enterprise users have no recourse for support. Even in the oft-cited example of RHEL, Red Hat offers Software Collections providing completely supported package releases of Python 2.7. There are corresponding examples for other similar distributions. Amazon Linux, AWS' own preferred distribution for cloud computing platforms, has included Python 2.7 as its default solution since 2015.

We've all had just shy of six years to change trains, now. The vast majority of us already have done so. Others have capably cited the evidence. If AWS has some reason to continue dawdling in the implementation here, it surely isn't because we enterprise customers have gone soft with the threat assessments we love so dearly.

[glaubitz]

@nethershaw Amazon wouldn't support Python 2.6 if there wasn't a business case for it.

I'm always surprised that random drive-by commenters think they know the AWS customer base better than Amazon themselves. It's solely up to Amazon to decide when they drop Python 2.6 support. It's a pointless discussion as you won't be able to change decisions that are dictated by management.

Developers are always interested in dropping support for old platforms. But in the end, they're not the ones making the ultimate decisions.

[nethershaw]

@glaubitz Thank you for your delightfully ad hominem response. Of course, nothing you've said is informative, and you have gone off topic by opining about management decisions that have nothing to do with the technical facts of this issue. If it's solely up to Amazon to decide, and it has nothing at all to do with the enterprise customers from your first post, then what in the world are you doing here other than telling people their discussions are pointless?

The point of my comment was to speak as a career professional and _member_ of that customer base, to add another voice with some authority to describe what _our_ actual needs are, and to redress your claim that _we_ cannot tolerate dropping Python 2.6. But we'd better not tell them that because the decisions are dictated by management? Please. Someone is doing a drive-by, and it's not me.

The valuable contributions to this discussion are the ones making concrete demonstrations of fact that the community has already largely abandoned Python 2.6. We are here to let Amazon know that doing the same on their end (a) is a necessary step to take and (b) won't break us. Whether and when they decide to do so, as they inevitably must, will be informed by the people who use their tools. Otherwise this entire repository and especially its issue tracker needn't exist.

[johnnicely]

The valuable contributions to this discussion are the ones making concrete demonstrations of fact that the community has already largely abandoned Python 2.6. We are here to let Amazon know that doing the same on their end (a) is a necessary step to take and (b) won't break us. Whether and when they decide to do so, as they inevitably must, will be informed by the people who use their tools. Otherwise this entire repository and especially its issue tracker needn't exist.

+1 - Really don't need more holy wars in my inbox.

[glaubitz]

@nethershaw At which point exactly have I attacked you personally so my comment fulfills the definition of "ad hominem"? I haven't done that. I have solely explained that people are arguing against decisions that are most likely driven by management and management is not going to listen to comments on a GitHub tracker.

If you are a professional member, you should well aware know and understand how large companies work. Business decisions are driven by management, not by developers and not by people in GitHub comments.

Whether Python 2.6 is necessary or not is not solely a technical decision. By starting these arguments you are implying that the maintainers are making uninformed decisions as they don't understand that no one in the community is no longer using this Python version. I mean, seriously, are you not realizing how rude your argument is? Do you really think the maintainers are too stupid to understand the situation that they need random people on GitHub explaining them which fundamental design decisions to make?

Amazon has probably millions of cloud customers. Whether there are 10, 100 or 1000 different people arguing here whether Python 2.6 support should be dropped or not is still completely irrelevant. If there is one Fortune 500 company which is their customer and needs it, they are going to support it. That's how it works.

@johnnicely The "holy war" was started by people who think they needed to hijack this thread to discuss Amazon business decisions, not me.

[nikolaik]

For reference, here are the download stats for awscli per python version. And the last 299 days days from the pypi bigquery dataset:

  REGEXP_EXTRACT(details.python, r"^([^\.]+\.[^\.]+)") as python_version,
  COUNT(*) as downloads,
FROM
  TABLE_DATE_RANGE(
    [the-psf:pypi.downloads],
    DATE_ADD(CURRENT_TIMESTAMP(), -300, "day"),
    DATE_ADD(CURRENT_TIMESTAMP(), -1, "day")
  )
WHERE
  file.project = 'awscli'
GROUP BY
  python_version,
ORDER BY
  downloads DESC
LIMIT 100

Looks like python 2.6 pypi clients still accounts for a pretty large proportion:

Row | python_version | downloads
-- | -- | --
1 | 2.7 | 221267827
2 | 3.6 | 24700549
3 | 2.6 | 10581111
4 | 3.5 | 6489991
5 | 3.7 | 3657717
6 | 3.4 | 1908679
7 | null | 1485949
8 | 3.3 | 3378
9 | 3.8 | 3096
10 | 3.2 | 86
11 | 3.1 | 2

[tysonclugg]

Looks like python 2.6 pypi clients still accounts for a pretty large proportion:

Row python_version downloads
1 2.7 221267827
2 3.6 24700549
3 2.6 10581111
4 3.5 6489991
5 3.7 3657717
6 3.4 1908679
7 null 1485949
8 3.3 3378
9 3.8 3096
10 3.2 86
11 3.1 2

How many of those downloads are caused by automated test suites, web crawlers, etc; as opposed to production installs? The user-agent details aren't included in that query, so it's not the full picture.

Meanwhile...

On 6 February 2019 @jamesls wrote:

While I think we will eventually drop support for python2.6, we'll have to come up with a something in the interim. We could use rsa 4.x for >= python2.7, and 3.x for python2.6. I would be a little hesitant about using multiple major versions but from what I can tell, the APIs we're using don't change from 3.x to 4.x, so I think this is our best option for now.

On 26 April 2019 I suggested that we remove the maximum version limits from all requirements in order to fulfil the interim needs mentioned by @jamesls above. Can this be done ASAP and a new release be made? Doing so would resolve the following issues:

• [ ] #4042
• [ ] #3828
• [ ] #4015

[coredumperror]

Bump.

Please do something so that I'm not stuck on an ancient build of python-rsa. Thank you.

[Jamie5]

As python2.6 is not supported by awscli anymore, and this was the rationale of why rsa couldn't be upgraded, what is now the reason to not upgrade?

@jamesls

[lballabio]

FYI: as of today, SafetyDB and the related safety tool started reporting two vulnerabilities in rsa < 4.3.

I obviously don't know what factors entered so far your decision to not upgrade, but please add to them that you're now explicitly requiring an insecure package.

[muriloviana]

Same thing here, unsafe package warning because awscli is using rsa < 4.3.

[jgzurano]

Same here running pipenv check

Checking installed package safety…
38414: rsa <4.3 resolved (3.4.2 installed)!
Rsa 4.3 includes two security fixes:
- Choose blinding factor relatively prime to N.
- Reject cyphertexts (when decrypting) and signatures (when verifying) that have  been modified by prepending zero bytes. This resolves CVE-2020-13757.

[matthewdeanmartin]

This is breaking build scripts all across the python world. For big orgs, having an insecure package in the mix is a deal breaker. But there is no alternative to awscli. Too many packages are no longer supported by python 2.6, so it is very short sighted to expect your customers to stick with python 2.6 just because six of them can't upgrade to 3.x

[stealthycoin]

The upper constraint was relaxed to include <=4.5 here https://github.com/aws/aws-cli/pull/5355 and this has been released in CLI v1.18.97.