Aws-sam-cli: symlinks in zip are not retained on the docker image created when using Layers

[innovia]

Description

When adding a package that contains symlinks these symlinks are flattened on the docker image

Briefly describe the bug you are facing.
I am taking a folder with symlinks in it and zipping it using the --symlinks option of the zip
when I unpack the zip file locally the symlinks are there but when I run the sam invoke command there are failures to find these file.

when debugging the produced container using
docker run -it --rm --entrypoint=/bin/bash samcli/lambda:provided-11bcc8dc4810532c82a432b30 -i the files are flatten into names with no symlinks.

Steps to reproduce

create a folder with a symlink to a file
package the files using zip -yr some.zip folder_name
run sam local invoke and call the symlink file.
Provide steps to replicate.

git clone [email protected]:gkrizek/bash-lambda-layer.git
cd bash-lambda-layer
zip -yr layer.zip bootstrap bin/ lib/ libexec/ share/
aws lambda publish-layer-version --layer-name myLayer --zip-file fileb://layer.zip

template.yaml:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  Sample SAM Template for sam-app
Globals:
  Function:
    Timeout: 300

Resources:
  Bash:
    Type: AWS::Serverless::Function
    Properties:
        CodeUri: .
        Handler: index.handler
        Runtime: provided
        Layers:
        - arn:aws:lambda:us-west-2:744348701589:layer:bash:4

index.sh

handler () {
    set -e
    git clone [email protected]:hypnoglow/helm-s3.git
}

Observed result

symlinks are flattened into files

ls -la
total 204
drwx------ 3 sbx_user1051  495   4096 Dec 18 20:33 .
drwxr-xr-x 1 root         root   4096 Dec 18 20:33 ..
drwx------ 2 sbx_user1051  495   4096 Dec 18 20:33 fipscheck
-rwx------ 1 sbx_user1051  495     17 Dec 18 20:33 libedit.so.0
-rwx------ 1 sbx_user1051  495 178816 Dec 18 20:33 libedit.so.0.0.27
-rwx------ 1 sbx_user1051  495     21 Dec 18 20:33 libfipscheck.so.1
-rwx------ 1 sbx_user1051  495   6960 Dec 18 20:33 libfipscheck.so.1.1.0
bash-4.2# cat libfipscheck.so.1
libfipscheck.so.1.1.0bash-4.2# ln -nfs libfipscheck.so.1.1.0 libfipscheck.so.1

command output with debug flag

Cloning into '/tmp/.helm/plugins/helm-s3'...
/opt/bin/ssh: error while loading shared libraries: /opt/lib/libfipscheck.so.1: file too short
fatal: Could not read from remote repository.

Please provide command output with `--debug` flag set.
```console
2018-12-19 10:52:45 Using SAM Template at /Users/ami/ltx/devops/lambda/kubernetes_deploy/template.yaml
2018-12-19 10:52:45 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2018-12-19 10:52:45 Changing event name from before-call.apigateway to before-call.api-gateway
2018-12-19 10:52:45 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2018-12-19 10:52:45 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2018-12-19 10:52:45 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2018-12-19 10:52:45 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2018-12-19 10:52:45 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2018-12-19 10:52:45 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2018-12-19 10:52:45 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2018-12-19 10:52:45 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2018-12-19 10:52:45 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2018-12-19 10:52:45 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2018-12-19 10:52:45 Changing event name from before-call.apigateway to before-call.api-gateway
2018-12-19 10:52:45 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2018-12-19 10:52:45 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2018-12-19 10:52:45 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2018-12-19 10:52:45 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2018-12-19 10:52:45 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2018-12-19 10:52:45 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2018-12-19 10:52:45 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2018-12-19 10:52:45 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2018-12-19 10:52:45 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2018-12-19 10:52:45 local invoke command is called
2018-12-19 10:52:45 Looking for credentials via: env
2018-12-19 10:52:45 Looking for credentials via: assume-role
2018-12-19 10:52:45 Looking for credentials via: shared-credentials-file
2018-12-19 10:52:45 Found credentials in shared credentials file: ~/.aws/credentials
2018-12-19 10:52:45 Loading JSON file: /Users/ami/ltx/virtual_envs/sam/lib/python2.7/site-packages/botocore/data/endpoints.json
2018-12-19 10:52:45 Event choose-service-name: calling handler <function handle_service_name_alias at 0x1039457d0>
2018-12-19 10:52:45 Loading JSON file: /Users/ami/ltx/virtual_envs/sam/lib/python2.7/site-packages/botocore/data/serverlessrepo/2017-09-08/service-2.json
2018-12-19 10:52:45 Event creating-client-class.serverlessapplicationrepository: calling handler <function add_generate_presigned_url at 0x1038f97d0>
2018-12-19 10:52:45 Setting serverlessrepo timeout as (60, 60)
2018-12-19 10:52:45 Loading JSON file: /Users/ami/ltx/virtual_envs/sam/lib/python2.7/site-packages/botocore/data/_retry.json
2018-12-19 10:52:45 Registering retry handlers for service: serverlessrepo
2018-12-19 10:52:45 No Parameters detected in the template
2018-12-19 10:52:45 1 resources found in the template
2018-12-19 10:52:45 Found Serverless function with name='Bash' and CodeUri='./mycode'
2018-12-19 10:52:45 Trying paths: ['/Users/ami/.docker/config.json', '/Users/ami/.dockercfg']
2018-12-19 10:52:45 Found file at path: /Users/ami/.docker/config.json
2018-12-19 10:52:45 Found 'auths' section
2018-12-19 10:52:45 Auth data for https://1234567812.dkr.ecr.us-west-2.amazonaws.com is absent. Client might be using a credentials store instead.
2018-12-19 10:52:45 Auth data for 1234567812.dkr.ecr.us-west-2.amazonaws.com is absent. Client might be using a credentials store instead.
2018-12-19 10:52:45 Auth data for 588736812464.dkr.ecr.us-west-2.amazonaws.com is absent. Client might be using a credentials store instead.
2018-12-19 10:52:45 Found 'credsStore' section
2018-12-19 10:52:45 http://localhost:None "GET /v1.35/_ping HTTP/1.1" 200 2
2018-12-19 10:52:45 Event choose-service-name: calling handler <function handle_service_name_alias at 0x1039457d0>
2018-12-19 10:52:45 Loading JSON file: /Users/ami/ltx/virtual_envs/sam/lib/python2.7/site-packages/botocore/data/lambda/2015-03-31/service-2.json
2018-12-19 10:52:45 Event creating-client-class.lambda: calling handler <function add_generate_presigned_url at 0x1038f97d0>
2018-12-19 10:52:45 Setting lambda timeout as (60, 60)
2018-12-19 10:52:45 Registering retry handlers for service: lambda
2018-12-19 10:52:45 Trying paths: ['/Users/ami/.docker/config.json', '/Users/ami/.dockercfg']
2018-12-19 10:52:45 Found file at path: /Users/ami/.docker/config.json
2018-12-19 10:52:45 Found 'auths' section
2018-12-19 10:52:45 Auth data for https://1234567812.dkr.ecr.us-west-2.amazonaws.com is absent. Client might be using a credentials store instead.
2018-12-19 10:52:45 Auth data for 1234567812.dkr.ecr.us-west-2.amazonaws.com is absent. Client might be using a credentials store instead.
2018-12-19 10:52:45 Auth data for 588736812464.dkr.ecr.us-west-2.amazonaws.com is absent. Client might be using a credentials store instead.
2018-12-19 10:52:45 Found 'credsStore' section
2018-12-19 10:52:45 Found one Lambda function with name 'Bash'
2018-12-19 10:52:45 Invoking index.handler (provided)
2018-12-19 10:52:45 No environment variables found for function 'Bash'
2018-12-19 10:52:45 Environment variables overrides data is standard format
2018-12-19 10:52:45 Loading AWS credentials from session with profile 'default'
2018-12-19 10:52:45 Resolving code path. Cwd=/Users/ami/ltx/devops/lambda/kubernetes_deploy, CodeUri=./mycode
2018-12-19 10:52:45 Resolved absolute path to code is /Users/ami/ltx/devops/lambda/kubernetes_deploy/mycode
2018-12-19 10:52:45 Code /Users/ami/ltx/devops/lambda/kubernetes_deploy/mycode is not a zip/jar file
2018-12-19 10:52:45 arn:aws:lambda:us-west-2:744348701589:layer:bash:4 is already cached. Skipping download
2018-12-19 10:52:45 http://localhost:None "GET /v1.35/images/samcli/lambda:provided-2a230b32cca171ea74c975d71/json HTTP/1.1" 200 None
2018-12-19 10:52:45 Trying paths: ['/Users/ami/.docker/config.json', '/Users/ami/.dockercfg']
2018-12-19 10:52:45 Found file at path: /Users/ami/.docker/config.json
2018-12-19 10:52:45 Found 'auths' section
2018-12-19 10:52:45 Auth data for https://1234567812.dkr.ecr.us-west-2.amazonaws.com is absent. Client might be using a credentials store instead.
2018-12-19 10:52:45 Auth data for 1234567812.dkr.ecr.us-west-2.amazonaws.com is absent. Client might be using a credentials store instead.
2018-12-19 10:52:45 Auth data for 588736812464.dkr.ecr.us-west-2.amazonaws.com is absent. Client might be using a credentials store instead.
2018-12-19 10:52:45 Found 'credsStore' section
2018-12-19 10:52:45 http://localhost:None "GET /v1.35/images/samcli/lambda:provided-2a230b32cca171ea74c975d71/json HTTP/1.1" 200 None
2018-12-19 10:52:45 Requested to skip pulling images ...

2018-12-19 10:52:45 Mounting /Users/ami/ltx/devops/lambda/kubernetes_deploy/mycode as /var/task:ro inside runtime container
2018-12-19 10:52:45 http://localhost:None "POST /v1.35/containers/create HTTP/1.1" 201 90
2018-12-19 10:52:45 http://localhost:None "GET /v1.35/containers/db9cd95fd9d8945622ffd8659e58c48c569cd1016aec570976d2293c88cac725/json HTTP/1.1" 200 None
2018-12-19 10:52:46 http://localhost:None "GET /v1.35/containers/db9cd95fd9d8945622ffd8659e58c48c569cd1016aec570976d2293c88cac725/json HTTP/1.1" 200 None
2018-12-19 10:52:46 http://localhost:None "POST /v1.35/containers/db9cd95fd9d8945622ffd8659e58c48c569cd1016aec570976d2293c88cac725/start HTTP/1.1" 204 0
2018-12-19 10:52:46 Starting a timer for 300 seconds for function 'Bash'
2018-12-19 10:52:46 http://localhost:None "GET /v1.35/containers/db9cd95fd9d8945622ffd8659e58c48c569cd1016aec570976d2293c88cac725/json HTTP/1.1" 200 None
2018-12-19 10:52:46 http://localhost:None "POST /containers/db9cd95fd9d8945622ffd8659e58c48c569cd1016aec570976d2293c88cac725/attach?stream=1&stdin=0&logs=1&stderr=1&stdout=1 HTTP/1.1" 101 0
START RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72 Version: $LATEST
Cloning into '/tmp/.helm/plugins/helm-s3'...
/opt/bin/ssh: error while loading shared libraries: /opt/lib/libfipscheck.so.1: file too short
fatal: Could not read from remote repository.

Please make sure you have the correct access rights

Expected result

symlinks files should be preserved

total 368
drwxr-xr-x   7 ami   wheel     224 Dec 18 17:31 .
drwxrwxrwt  24 root  wheel     768 Dec 19 10:56 ..
drwxr-xr-x   6 ami   wheel     192 Dec 18 17:31 fipscheck
lrwxr-xr-x   1 ami   wheel      17 Dec 18 23:26 libedit.so.0 -> libedit.so.0.0.27
-rwxr-xr-x   1 ami   wheel  178816 Dec 18 17:31 libedit.so.0.0.27
lrwxr-xr-x   1 ami   wheel      21 Dec 18 23:26 libfipscheck.so.1 -> libfipscheck.so.1.1.0
-rwxr-xr-x   1 ami   wheel    6960 Dec 18 17:31 libfipscheck.so.1.1.0

Describe what you expected.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS:
2. sam --version:
   SAM CLI, version 0.9.0
   Add --debug flag to command you are running

Note: this does not happen with real lambda

[mnapoli]

+1, just wanted to say this is a pretty big problem for us to build PHP runtimes (see https://github.com/mnapoli/bref/issues/146 for more details).

[ispyinternet]

Any update on this from SAM team? Also Unable to use this layer locally https://github.com/lambci/git-lambda-layer

[metaskills]

Here here, ran into this issue today too with Ruby runtime and a Layer with libvips. https://github.com/customink/ruby-vips-lambda

[mnapoli]

Maybe the core team can point us to the piece of code that does the unzip of the layers? If it's a matter of looking up with flag or option to use I could maybe try to contribute a fix.

[jfuss]

@mnapoli We download and unzip layers here but the main unzipping logic lives here.

The other part this could be going wrong is when we tar up the contents and build the image. This happens within the lambda_image.py file with the actual tar'ing code here.

I haven't had the chance to dig into which part (or both) are causing trouble here. Happy to help lead someone or answer questions around the code.

[metaskills]

Maybe related to this? https://stackoverflow.com/questions/35782941/archiving-symlinks-with-python-zipfile and maybe require this? http://infozip.sourceforge.net/UnZip.html or using the CLI zip of the Docker host?

[jfuss]

@metaskills SAM CLI doesn't do the zipping. It seems like the zip information is already there but we aren't respecting it?

[metaskills]

Yup, that's what I'm thinking cause Python zip does not have the capability to do so. From my limited research. If that is true, how would you suggest a proposed change fix that? Seems the options (and maybe more) are:

1. Shell out to the lower OS for the unzip. Can we assume that is the Docker container?
2. Add some Python package for zip features that respects symlinks.

[marcomagdy]

This is affecting C++ Lambda functions as well, which relies heavily on symlinks in the zip file.

I see that the label "stage/waiting-for-release" has been removed and the related PR has been reverted. Is there still a plan to fix this problem?

[jfuss]

@bubba-h57 Are you still willing to send a new PR with what we talked about here?

[bubba-h57]

I apologize, I'm struggling with some PTSD issues and once the episode
passes I can focus again. I would love to do it, I just don't know how long
this one will lat and that makes me unreliable. Again, my apologies.

On Fri, Aug 30, 2019, 4:07 PM Jacob Fuss [e-mail] wrote:

@bubba-h57 https://github.com/bubba-h57 Are you still willing to send a
new PR with what we talked about here
https://github.com/awslabs/aws-sam-cli/pull/1315#issuecomment-520499875?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/awslabs/aws-sam-cli/issues/878?email_source=notifications&email_token=AAETL3VZCN4IMIXWDJCI4KLQHF4XBA5CNFSM4GLHGAGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5SUWIQ#issuecomment-526732066,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAETL3XDB2EGODV47QXBYJDQHF4XBANCNFSM4GLHGAGA
.

[boulabiar]

This is also affecting me for C++ lambda.

[jfuss]

@bubba-h57 No need to apologize, your health is way more important!

[jfuss]

Since we just merged removing Py2.7 support #1416 and we reverted the initial fix for this due to Py2.7 issues: https://github.com/awslabs/aws-sam-cli/pull/1140#issuecomment-519154547. We should be able to just add this back in. Going to work on a PR based on the previous PR.

[jfuss]

PR: #1482

[hburrows]

+1 on getting this fixed. I've built an "imagemagick" image processing layer that is impacted by this because imagemagick uses lots of symlinks. Not being able to run and develop locally adds significant friction.

[c2tarun]

Closing this issue as it was fixed in Release 0.48.0