Aws-sam-cli: sam deploy fails if Docker is not running

[wnkz]

Description:

Running sam deploy fails if Docker is not running

Steps to reproduce:

• Stop Docker daemon
• Run sam deploy

Observed result:

4 nested exceptions:

ConnectionRefusedError: [Errno 61] Connection refused
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionRefusedError(61, 'Connection refused'))
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(61, 'Connection refused'))
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', ConnectionRefusedError(61, 'Connection refused'))

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: macOS 10.15.7
2. sam --version: SAM CLI, version 1.13.1

Add --debug flag to command you are running

I tracked it down to here but there could be others.

https://github.com/aws/aws-sam-cli/blob/d2917102ef56ac05b9973f96c716612f9638bb62/samcli/commands/package/package_context.py#L103

This is a rare problem on local environment but I noticed it in a Jenkins environment where Docker is not available and this blocks new deployments. I am guessing this is related to support for Lambda Container Images.

[prenx4x]

I was unable to reproduce this using zip based sample app. Is this a lambda image application or zip based? does this occur when using sam deploy --guided as well?

[prenx4x]

so i was able to reproduce this with a hello world lambda image app. after doing sam build, I killed docker and then sam deploy-guided which resulted in this error.

[wnkz]

so i was able to reproduce this with a hello world lambda image app. after doing sam build, I killed docker and then sam deploy-guided which resulted in this error.

I'd have to do some more testing but I was having this error with a zip based Lambda function using a layer (AWS::Serverless::LayerVersion).

[jfuss]

@wnkz I am not seeing this reproduce with a zip based function and a Layer within the template. Can you share the template or one that reproduces the issue you are seeing?

[wnkz]

@jfuss I can't share the original template as-is. I'll try to reproduce with the minimum resources and let you know.

[wnkz]

Ok @jfuss I could not reproduce the problem on a minimal project but I finally found were the problem came from (was a tricky one).
On this project we are using Pipenv. I figured out that on the CI server at this time, Pipfile.lock referenced (among other things)

        "aws-sam-cli": {
            "hashes": [
                "sha256:340a56070d34011dd632a9c1f97a87b3dcdb23bdae4ac96aedb41239dcea4b68",
                "sha256:6f91af6f3f6a6c4eece26401dd7e91ee0167e7b83831c42ae207e582cf8ceb76"
            ],
            "index": "pypi",
            "version": "==1.13.1"
        },
        "docker": {
            "extras": [
                "ssh"
            ],
            "hashes": [
                "sha256:317e95a48c32de8c1aac92a48066a5b73e218ed096e03758bcdd799a7130a1a1",
                "sha256:cffc771d4ea1389fc66bc95cb72d304aa41d1a1563482a9a000fba3a84ed5071"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
            "version": "==4.4.0"
        },

This does not seem possible as aws-sam-cli requires docker~=4.2.0 but yeah .. I don't know how the .lock ended up like this.
Anyway, if you forcefully install docker==4.4.0 after aws-sam-cli ; you get the error I mention.

I guess the conclusion could be be careful if you're thinking about upgrading docker requirement.

Sorry for the trouble and thanks for the help.

[prenx4x]

Thanks @wnkz for letting us know the root cause. I will go ahead and close this ticket.

[wnkz]

@prenx4x FYI, I'm pretty sure this has been caused by https://github.com/aws/aws-sam-cli/commit/387f7319bd333b832bf3d88916d79314c633138b and that Pipenv never updated that dependency ; I don't know if it's a Pipenv bug but it might happen to others using Pipenv. Checking docker version might help if other users are reporting this.