Aws-sam-cli: Compression packs symlinks as copies

[garthk]

Description:

sam package compresses symlinks as an additional copy of the file to which they resolve. Given a traditional shared library layout with

libz.so → libz.so.1
libz.so.1 → libz.so.1.2.11
libz.so.1.2.11

… sam package will pack the library three times, not once with two symlinks.

Steps to reproduce the issue:

make in a directory with .env, template.yml, and Makefile:

.env: replace the values with your own profile, region, and S3 bucket name:

AWS_PROFILE=default
AWS_REGION=ap-southeast-2
BUCKET=your-bucket-name-I-do-not-want-to-know

template.yml:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Reproducing a SAM CLI compression wart

Globals:
  Function:
    Runtime: nodejs8.10

Resources:
  DummyFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: Dummy
      CodeUri: ./build
      Handler: lambda.dummy
  ```

`Makefile`:
```make
# Import AWS_PROFILE, AWS_REGION, BUCKET, and S3_PREFIX  from ./.env:
-include .env

# Set defaults, with none of which you'll likely be happy:
AWS_PROFILE ?= default
AWS_REGION ?= ap-southeast-2
BUCKET ?= your-bucket-name-here
S3_PREFIX ?= lambdas

# Targets
EXPECTED := expected.zip
OBSERVED := observed.zip

# Satisfy `aws` and `sam`:
export AWS_PROFILE
export AWS_REGION

# These targets don't produce files:
.PHONY: comparison clean

# This target is our default:
comparison: $(EXPECTED) $(OBSERVED)

# This cleans up:
clean:
    rm -rf build $(OBSERVED) $(EXPECTED)

# Construct the ZIP file we wish SAM had made:
$(EXPECTED): build
    ( cd build ; zip -9y ../$(EXPECTED) * )
    unzip -l $(EXPECTED)

# Download and list the contents of the ZIP file `sam package` uploaded:
$(OBSERVED): packaged.yml
    aws s3 cp `grep CodeUri packaged.yml  | awk '{ print $$2 }'` $(OBSERVED)
    unzip -l $(OBSERVED)

# Zip the `build` directory, upload it to `BUCKET`, and prepare `packaged.yml`:
packaged.yml: build template.yml
    sam package \
        --debug \
        --template-file template.yml \
        --s3-bucket $(BUCKET) \
        --s3-prefix $(S3_PREFIX) \
        --output-template-file packaged.yml

# Prepare a `build` directory with a random 10MB shared library with
# traditional symlinks:
build: Makefile
    rm -rf build
    mkdir -p build
    echo 'module.exports.dummy = function() { return { pass: true }; };' > build/lambda.js
    dd if=/dev/random of=build/libz.so.1.2.11 bs=65536 count=160
    ln -s libz.so.1.2.11 build/libz.so.1
    ln -s libz.so.1 build/libz.so

Observed result:

Archive:  observed.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
       62  06-13-2018 14:02   lambda.js
 10485760  06-13-2018 14:02   libz.so.1
 10485760  06-13-2018 14:02   libz.so
 10485760  06-13-2018 14:02   libz.so.1.2.11
---------                     -------
 31457342                     4 files

Expected result:

Archive:  expected.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
       62  06-13-2018 14:02   lambda.js
        9  06-13-2018 14:02   libz.so
       14  06-13-2018 14:02   libz.so.1
 10485760  06-13-2018 14:02   libz.so.1.2.11
---------                     -------
 10485845                     4 files

Also, I expect compression at least as good as zip -9.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc) macOS 10.13.5

Output of sam --version: SAM CLI, version 0.3.0

Optional Debug logs: I gave --debug but couldn't find the output

[mhart]

This is also an issue with sam package creating layers that contain symlinks.

Given:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Resources:
  DependenciesLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      ContentUri: dependencies/

If the dependencies directory has symlinks (which is very common for runtimes and binaries), then these won't be zipped correctly (ie, in the same way that zip -yr would) – this can lead to bloat and outright failure if the symlink is an absolute one (eg, it links to /opt/something)

You can manually create the zip yourself, but if you refer to it in the ContentUri (eg, ContentUri: dependencies.zip), then sam local invoke no longer appears to work (it doesn't unpack the zip). Should I file a separate bug for that?