Amazon-ssm-agent: build 722 broke SSM

[AdilHindistan]

I reviewed the the latest release notes about 722 and did not notice a glaring one but change from 714 to 722 broke SSM in our RHEL images. Agent starts but complains:

systemd[1]: Started amazon-ssm-agent.
amazon-ssm-agent[8553]: 2019-10-18 20:54:25 INFO Entering SSM Agent hibernate - RequestError: send request failed
amazon-ssm-agent[8553]: caused by: Post https://ssm.us-east-1.amazonaws.com/: proxyconnect tcp: tls: first record does not look like a TLS handshake
amazon-ssm-agent[8553]: 2019-10-18 20:54:25 INFO Agent is in hibernate mode. Reducing logging. Logging will be reduced to one log per backoff period

https goes thru our proxy to connect to that endpoint. It works fine if I downgrade to 714.

Any ideas what change might have caused this?

[LiangChiAmzn]

Hello,

Thank you for reaching out to us. Could you help us to narrow down the issue by providing the following:

• What version of RHEL you are using?
• Are they EC2 instances or on-premises hosts?
• If those are EC2 instances, would you give us the AMI id so we can reproduce?

[richard-scott]

It's still broken on SSM 760 too!

We are using Ubuntu 16 host (on-premises) with a pfSense based Squid Proxy doing SSL. We use a self signed SSL cert for the Squid proxy and this works fine for other https sites.

[shihuazhang]

Hi @AdilHindistan @richard-scott

Please check your proxy to see if it supports https

We have upgraded the golang from 1.9 to 1.12 for build 722.

https://golang.org/doc/go1.10#net/http

An HTTP proxy can now be specified as an "https://" URL, meaning that the client connects to the proxy over HTTPS before issuing a standard, proxied HTTP request. Prior to version 1.10, plain HTTP was sent even when we specify https in our proxy variables.

Instead
HTTPS_PROXY=https://xxx.xxx.xxx.xxx:xxxx

do
HTTPS_PROXY=http://xxx.xxx.xxx.xxx:xxxx

As a walkaround.

[shihuazhang]

I'm closing this issue, please feel free to reopen it if you need additional help.

[richard-scott]

Why are you closing it? It's still broken.

We can't use this:
HTTPS_PROXY=http://xxx.xxx.xxx.xxx:xxxx
as our proxy ONLY works on https://.

[shihuazhang]

@richard-scott
Could you please confirm that your proxy listens on the https?
What's the result for the

curl -vI https://hostname:port

openssl s_client -connect https://hostname:port

Could you also provide amazon-ssm-agent.log to help us diagnose the issue?

[nonspecialist]

The approach that I've taken which works around this is to provision VPC endpoints for ssm and ssmmessages services. You then don't have to go through on-prem MITM proxies. Not sure if that'll work for this case, but it's worth considering

[shihuazhang]

@nonspecialist Thanks for your comment.

Do you mind help us to understand the issue you been having? Please run the following commands, it will help with diagnosing the issue.

curl -vI https://hostname:port
openssl s_client -connect https://hostname:port

[AdilHindistan]

Hi @AdilHindistan @richard-scott

Please check your proxy to see if it supports https

We have upgraded the golang from 1.9 to 1.12 for build 722.

https://golang.org/doc/go1.10#net/http

An HTTP proxy can now be specified as an "https://" URL, meaning that the client connects to the proxy over HTTPS before issuing a standard, proxied HTTP request. Prior to version 1.10, plain HTTP was sent even when we specify https in our proxy variables.

Instead
HTTPS_PROXY=https://xxx.xxx.xxx.xxx:xxxx

do
HTTPS_PROXY=http://xxx.xxx.xxx.xxx:xxxx

As a walkaround.

That worked as a workaround, thank you!

[ericgreveson]

Hi,

I'm having the same issue. We are using an HTTPS proxy which I believe is probably terminating the HTTPS connection to the remote server and then using its own local (corporate) certificate from the proxy to our server. I.e. HTTPS_PROXY=https://username:[email protected]:8080 (username, password and proxy host are slightly different in reality but this gives you an idea).

@shihuazhang: To give the output of the commands you asked about earlier in the thread:

$ curl -vI https://ssm.eu-west-1.amazonaws.com/
*   Trying 10.10.2.16...
* TCP_NODELAY set
* Connected to proxy.intranet (10.10.2.16) port 8080 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

and

$ openssl s_client -connect https://ssm.eu-west-1.amazonaws.com/
140547117523392:error:2008F002:BIO routines:BIO_lookup_ex:system lib:../crypto/bio/b_addr.c:704:Servname not supported for ai_socktype
connect:errno=0

Hope this helps...

[ericgreveson]

Actually I have now solved the issue (for me at least)! There were two things:
1) The proxy server is indeed converting HTTPS to HTTP. So I changed my env var to HTTPS_PROXY=http://username:[email protected]:8080 (rather than the previous https://... one) and this allowed amazon_ssm_agent to work properly on the command line. Things broke following a Ubuntu 16.04 -> Ubuntu 18.04 upgrade, so it's likely that the older libs didn't care too much that the proxy was actually returning HTTP rather than HTTPS (despite the incorrect configuration of HTTPS_PROXY) but the newer libs are more strict.
2) Even after this, though, the system service for amazon_ssm_agent (running on systemd) still didn't work. I eventually figured out the reason for this: the proxy password has a % character in it, and when using systemctl edit amazon-ssm-agent as per https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-proxy-with-ssm-agent.html, the Environment= line doesn't work with % characters. Systemd requires these to be escaped as %%. So I did that and now everything is working fine again.

[chefren]

The build is broken: https://travis-ci.org/aws/amazon-ssm-agent

Go version 1.12.0 is not in the releases, should use 1.12 or 1.12.x (ref)

https://golang.org/dl/