Amazon-ecs-agent: How to customize the CA certificates within ecs-agent (ssl inspection use case)

[chrisdag]

Summary

Need to install custom CA certificate into ecs-agent image so that it will function in the presence of an edge firewall device that is performing SSL inspection duties.

Description

I'm working on an ECS-on-EC2 project for a highly security conscious organization that operates with a "firewall all the things!" requirement. The use of edge firewalls in our VPC extends to include communication with AWS API endpoints AND it includes the use of firewalls that perform SSL decryption and inspection.

Honestly I can't blame them given how HTTPS is used to hide malware and data exfiltration these days. But ...

Because SSL is being intercepted and inspected we need to install an additional CA certificate into all of our clients and endpoints so that we can avoid "SSL verification" or "Unable to verify TLS session" type fatal errors

I'm trying to figure out if there is any method suitable for a docker novice that I can use to wedge a new CA certificate into ecs-agent. Basically just looking for tips, advice and guidance on any practical methods. Or blunt advice that I should just give up and call it a day.

Or maybe I should be making a feature request for a new ENV variable? Something like ECS_CA_PATH or similar ?

Expected Behavior

ecs-agent behaves as expected. It does not like SSL interception and generates the expected error messages.

Observed Behavior

Behaves as expected. I'm looking to alter behavior, not fix a real issue.

Environment Details

VPC comprised of 100% private facing subnets. No internet access by default. All outbound traffic, including traffic to AWS API endpoints passes through edge firewall devices that perform SSL decryption and inspection.

We know how to configure Linux OS and Python/boto etc. to work in this restricted environment but are having trouble figuring out if we can even customize ecs-agent to function within this type of setting.

Supporting Log Snippets

[aaithal]

Hi @chrisdag, thank you so much for that detailed issue description!

With the existing tooling/customizations available for the ECS agent, it'd be very hard for you to provide additional certs/cert stores to be packaged with the ECS agent. To unblock yourself immediately, you'd have to rebuild the ECS agent image from source and hook in to the Dockerfile to provide additional cert bundles.

Another option here could be do use a proxy and do additional cert validations there. More details about setting up a proxy for ECS agent can be found here.

However, this is certainly something which can be made customizable in the future. We can mark this as a feature request, where the ECS agent can consume additional sources of certs from config (and/or host volume mounts) and bootstrap with those. We're also be open to contributions in this realm :)

Thanks,
Anirudh

[samuelkarp]

You may also be able to mount your own CA certificate store over the one included in the image. We store the CA certificates in /etc/ssl/certs/ca-certificates.crt inside the image. However, there's no provision for adding a mount like this using ecs-init, so you'd instead need to run the agent with docker run.

[chrisdag]

Really appreciate the replies. Next step is to try to build from source while swapping in an amended set of CA certificates.

[chrisdag]

Ok from reading the source code it looks like the Makefile runs a minimal ubuntu container for the purpose of downloading and pruning the latest ca-certificates.crt file (I'm reading misc/certs/Dockerfile).

Once it pulls the latest CA certs into the container there is a COPY command that moves the ca-certificates.crt file out of the container and into misc/certs/ca-certificates.crt on the build host where it seems to be grabbed in a later step to be inserted/copied into the minimal scratch container used by ecs-agent

If I understand things correctly the I just need to edit the top level Makefile to "cat my-additional-CA-certs.crt >> misc/certs/ca-certificates.crt" so that the Ubuntu-derived CA file is updated with my custom additions, after it is refreshed but BEFORE any follow-on build commands occur. Then, later on in the build process the CA file in misc/certs/ is copied or inserted into the ecs-agent container where it lives at /etc/ssl/certs/ca-certificates.crt

[samuelkarp]

That could work. Another thing that could work is:

• Construct your own ca-certificates.crt with whatever contents you want
• Write a really simple Dockerfile like this:

FROM amazon/amazon-ecs-agent:v1.16.2
COPY ca-certificates.crt /etc/ssl/certs/ca-certificates.crt

I think that would likely be sufficient. You'd then be entirely replacing our CA certificate store with yours, but that also lets you control exactly which certificates are present. I also think this is a bit cleaner since you don't then need to modify any of our source files and git rebase every time we release a new agent.

[chrisdag]

Just wanted to say thanks for the feedback and hints here. As a novice in this space I found the Makefile in the source code repo more "approachable" to start with so I just made this type of change to the master Makefile:

````

We need to bundle certificates with our scratch-based container

certs: misc/certs/ca-certificates.crt
misc/certs/ca-certificates.crt:
docker build -t "amazon/amazon-ecs-agent-cert-source:make" misc/certs/
docker run "amazon/amazon-ecs-agent-cert-source:make" cat /etc/ssl/certs/ca-certificates.crt > misc/certs/ca-certificates.crt
echo "Attempting CompanyX CA certificate insert (copy step)..."
cp -f ../../CompanyX-ca/CompanyXIntCA-KEEP.cer misc/certs/
cp -f ../../CompanyX-ca/CompanyXRootCA-KEEP.cer misc/certs/
echo "Attempting CompanyX CA certificate insert (append step)..."
cat misc/certs/CompanyXCA-KEEP.cer >> misc/certs/ca-certificates.crt
cat misc/certs/CompanyXRootCA-KEEP.cer >> misc/certs/ca-certificates.crt
echo "Done with CompanyX CA cert tweaks."
````
Essentially I'm just appending to the file before a process later on in the build copies it into the actual container at path /etc/ssl/certs/ca-certificates.crt

Also managed to take the ecs-optimized AMI and customize it to disable the amazon-ecs-agent init process and replace it by the 'docker run' commands as your docs clearly showed. Worked great.

Can't tell you how nice it is to see ECS cluster status page showing both ACTIVE and "agent connected: true" !

It was also fun to see the full build happen via the make comment. In future revisions as I get better at docker work we'll replace this method with the one that Samuelkarp mentioned -- a simple Dockerfile that just executes a copy command to move the altered file into the prebuilt container.

[aaithal]

Awesome. I'm closing this for now. Please let us know if you need help with anything else.

Thanks,
Anirudh