Aws-cli: Problems using AWS cli when behind proxy

[steevmi1]

I'm having a problem when I try to run the AWS cli on my OS X laptop when its behind the corporate proxies. When I'm using Python 2.7 (2.7.8, installed via macports), when I run the AWS cli I get an error that the proxy hostname doesn't match either of the region's service endpoints. If I turn on --debug, it looks like boto's failing to make the connection

2014-08-04 13:34:29,548 - MainThread - botocore.endpoint - DEBUG - Sending http request:
2014-08-04 13:34:29,548 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): ec2.us-west-2.amazonaws.com
2014-08-04 13:34:29,821 - MainThread - botocore.endpoint - DEBUG - Exception received when sending HTTP request.
Traceback (most recent call last):
File "/Users/steevmi1/Library/Python/2.7/lib/python/site-packages/botocore/endpoint.py", line 130, in _get_response
proxies=self.proxies)
File "/Users/steevmi1/Library/Python/2.7/lib/python/site-packages/botocore/vendored/requests/sessions.py", line 464, in send
r = adapter.send(request, **kwargs)
File "/Users/steevmi1/Library/Python/2.7/lib/python/site-packages/botocore/vendored/requests/adapters.py", line 363, in send
raise SSLError(e)
SSLError: hostname '' doesn't match either of 'us-west-2.ec2.amazonaws.com', 'ec2.us-west-2.amazonaws.com'
2014-08-04 13:34:29,822 - MainThread - botocore.hooks - DEBUG - Event needs-retry.ec2.DescribeRegions: calling handler

If I install python 2.6, and then install the AWS cli against that version, then everything works without any problems.

I saw an older issue with the AWS cli (issue 72) that looks sort of like what I'm seeing, but it looks like that issue was closed out a year or so ago.

[davidski]

It looks like your proxy is intercepting and decrypting SSL communications (returning a cert for 'usca-proxy01.na.novartis.net'). You can tell aws-cli to ignore the cert and continue with the --no-verify-ssl flag, which is how I work with this scenario in my corporate environment.

[steevmi1]

@davidski But when I use awscli that's been built against Python 2.6, I don't see this issue.

8:15 steevmi1@PHUSCA-L25172[~] % aws --version
aws-cli/1.3.24 Python/2.6.9 Darwin/13.3.0
8:16 steevmi1@PHUSCA-L25172[~] % aws ec2 describe-regions
REGIONS ec2.eu-west-1.amazonaws.com eu-west-1
REGIONS ec2.sa-east-1.amazonaws.com sa-east-1
REGIONS ec2.us-east-1.amazonaws.com us-east-1
REGIONS ec2.ap-northeast-1.amazonaws.com ap-northeast-1
REGIONS ec2.us-west-2.amazonaws.com us-west-2
REGIONS ec2.us-west-1.amazonaws.com us-west-1
REGIONS ec2.ap-southeast-1.amazonaws.com ap-southeast-1
REGIONS ec2.ap-southeast-2.amazonaws.com ap-southeast-2

I would think if the proxy is doing something non-standard that I'd see the problem across the board, no?

[davidski]

It's not that your proxy is doing something non-standard per say, it's just decoding the SSL traffic and behaving as a man-in-the-middle attack on the conversation between you and AWS. On the general internet, that would indeed be a Bad Thing(tm) but it's not uncommon for corporate environments to inspect SSL traffic to provide deep packet inspection, filtering, and logging.

I can't comment on the pytthon2.6 discrepancy, but suspect that one of the libraries from py2.6 does not perform SSL verification and that those versions of aws-cli are arguably broken (or at least less secure).

[steevmi1]

It looks like boto doesn't have these issues, though, so it's something that's going on in the CLI?

8:29 steevmi1@PHUSCA-L25172[~] % python2.7
Python 2.7.8 (default, Jul 13 2014, 17:11:32)
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

from boto import ec2;
regions = ec2.regions();
regions
[RegionInfo:us-east-1, RegionInfo:cn-north-1, RegionInfo:ap-northeast-1, RegionInfo:eu-west-1, RegionInfo:ap-southeast-1, RegionInfo:ap-southeast-2, RegionInfo:us-west-2, RegionInfo:us-gov-west-1, RegionInfo:us-west-1, RegionInfo:sa-east-1]
regions[0]
RegionInfo:us-east-1
use = regions[0]
conn = use.connect()
conn.get_all_instances()
[Reservation:r-a97f74d8]

[davidski]

I'll quibble over the label of issue. Boto may not verify SSL certificates by default, but you can search the code here on Github for verify_ssl and see that the CLI does use that option. That strengthens the out of the box experience for users, while --no-verify-ssl allows folks in the still unusual (but not rare) case of MITM proxies to function.

As for whether that's the best solution, I have to defer to the dev team. In my environment, I've set --no-ssl-verify and moved on.

[jamesls]

The only SSL difference in py26 vs. py27 that I'm aware of is the opposite case. That is, in certain situations py26 _can't_ validate certain types of SSL certs that have a subjectAltName, whereas py27 can. But that means you get SSL cert validation failures when they're actually ok certs. I'm not aware of any scenario where we aren't validating certs by default.

Is it possible that perhaps in py26 and in boto you aren't pointing at the proxy? Can you confirm that in python2.6/boto requests are being sent to the proxy?

But, as @davidski points out, if you are going through a proxy that is decrypting SSL traffic like that, you'll have to use the --no-verify-ssl argument.

[jamesls]

Closing, I've not been able repro the issue where we're not validating SSL certs by default on python2.6.

And the --no-verify-ssl is the arg you'll need to use if you don't want to validate SSL certs, as in the case of special proxies that are intercepting SSL traffic.