Aws-cdk: Tools assume `pip` is installed globally

[SoManyHs]

@aws-cdk/aws-s3-deployment: > @aws-cdk/[email protected] build /mnt/c/Users/stankoa/Documents/GitHub/aws-cdk/packages/@aws-cdk/aws-s3-deployment
@aws-cdk/aws-s3-deployment: > cdk-build
@aws-cdk/aws-s3-deployment: ++ dirname lambda/build.sh
@aws-cdk/aws-s3-deployment: + cd lambda
@aws-cdk/aws-s3-deployment: + bundle_out=/mnt/c/Users/stankoa/Documents/GitHub/aws-cdk/packages/@aws-cdk/aws-s3-deployment/lambda/bundle.zip
@aws-cdk/aws-s3-deployment: + rm -f /mnt/c/Users/stankoa/Documents/GitHub/aws-cdk/packages/@aws-cdk/aws-s3-deployment/lambda/bundle.zip
@aws-cdk/aws-s3-deployment: ++ mktemp -d
@aws-cdk/aws-s3-deployment: + staging=/tmp/tmp.NF0bIo9ers
@aws-cdk/aws-s3-deployment: ++ mktemp -d
@aws-cdk/aws-s3-deployment: + piptemp=/tmp/tmp.Emoe4mSNtS
@aws-cdk/aws-s3-deployment: + trap 'rm -rf /tmp/tmp.NF0bIo9ers /tmp/tmp.Emoe4mSNtS' EXIT
@aws-cdk/aws-s3-deployment: + echo 'staging lambda bundle at /tmp/tmp.NF0bIo9ers...'
@aws-cdk/aws-s3-deployment: + rsync -av src/ /tmp/tmp.NF0bIo9ers
@aws-cdk/aws-s3-deployment: staging lambda bundle at /tmp/tmp.NF0bIo9ers...
@aws-cdk/aws-s3-deployment: sending incremental file list
@aws-cdk/aws-s3-deployment: ./
@aws-cdk/aws-s3-deployment: aws
@aws-cdk/aws-s3-deployment: index.py
@aws-cdk/aws-s3-deployment: requirements.txt
@aws-cdk/aws-s3-deployment: sent 6,965 bytes  received 76 bytes  14,082.00 bytes/sec
@aws-cdk/aws-s3-deployment: total size is 6,725  speedup is 0.96
@aws-cdk/aws-s3-deployment: + cd /tmp/tmp.NF0bIo9ers
@aws-cdk/aws-s3-deployment: + pip3 install --ignore-installed --prefix /tmp/tmp.Emoe4mSNtS -r /tmp/tmp.NF0bIo9ers/requirements.txt
@aws-cdk/aws-s3-deployment: lambda/build.sh: line 30: pip3: command not found
@aws-cdk/aws-s3-deployment: + rm -rf /tmp/tmp.NF0bIo9ers /tmp/tmp.Emoe4mSNtS
@aws-cdk/aws-s3-deployment: Error: '/bin/bash lambda/build.sh' exited with error code 127
@aws-cdk/aws-s3-deployment: Build failed. Total time (3.6s) | /bin/bash lambda/build.sh (3.0s)
@aws-cdk/aws-s3-deployment: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
@aws-cdk/aws-s3-deployment: npm ERR! code ELIFECYCLE
@aws-cdk/aws-s3-deployment: npm ERR! errno 1
@aws-cdk/aws-s3-deployment: npm ERR! @aws-cdk/[email protected] build: `cdk-build`
@aws-cdk/aws-s3-deployment: npm ERR! Exit status 1
@aws-cdk/aws-s3-deployment: npm ERR!
@aws-cdk/aws-s3-deployment: npm ERR! Failed at the @aws-cdk/[email protected] build script.
@aws-cdk/aws-s3-deployment: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
@aws-cdk/aws-s3-deployment: npm ERR! A complete log of this run can be found in:
@aws-cdk/aws-s3-deployment: npm ERR!     /root/.npm/_logs/2019-02-06T00_02_11_114Z-debug.log
lerna ERR! npm run build exited 1 in '@aws-cdk/aws-s3-deployment'
lerna WARN complete Waiting for 3 child processes to exit. CTRL-C to exit immediately.

real    22m5.358s
user    24m15.469s
sys     20m47.453s
❌  Last command failed. Scroll up to see errors in log (search for '!!!!!!!!').

Build fails in a new dev environment if pip is not previously installed, in this case due to
the s3-deployment build.

Is there a recommend way to install pip/virtualenv that could be added to either the contributing doc or the install script?

/c @allisaurus

[eladb]

We should move to docker-based builds so that people don't have to deal with this S*T

[SoManyHs]

That's fine, but I don't suppose in the meantime is there any guidance on where we might document the tools/external dependencies needed for s3-deployment (or other packages)? It looks like it's pip3 and zip.

[robertd]

@eladb Docker builds fail too. No code changes... just tried building the fresh repo.

 ~/code/robertd/aws-cdk  master ●  docker build -t aws-cdk .                       1 ↵  18:50:30
Sending build context to Docker daemon  1.204GB
Step 1/6 : FROM node:8.15-alpine
8.15-alpine: Pulling from library/node
169185f82c45: Already exists
556aee4b47a5: Pull complete
5f8e385168b1: Pull complete
Digest: sha256:32e73aad77719468ac82097642fde0f4747258a386f48e9bad25c91546d8e37d
Status: Downloaded newer image for node:8.15-alpine
 ---> cf491d6b25e2
Step 2/6 : WORKDIR /app
 ---> Running in f9ac84fd766a
Removing intermediate container f9ac84fd766a
 ---> 8fd5c81e5b45
Step 3/6 : RUN apk add --update     bash     git     rsync     zip     python3     python3-dev     py3-setuptools  && rm -rf /var/cache/apk/*
 ---> Running in b51099f7ad56
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
(1/27) Installing ncurses-terminfo-base (6.1_p20180818-r1)
(2/27) Installing ncurses-terminfo (6.1_p20180818-r1)
(3/27) Installing ncurses-libs (6.1_p20180818-r1)
(4/27) Installing readline (7.0.003-r0)
(5/27) Installing bash (4.4.19-r1)
Executing bash-4.4.19-r1.post-install
(6/27) Installing ca-certificates (20171114-r3)
(7/27) Installing nghttp2-libs (1.32.0-r0)
(8/27) Installing libssh2 (1.8.0-r3)
(9/27) Installing libcurl (7.61.1-r1)
(10/27) Installing expat (2.2.5-r0)
(11/27) Installing pcre2 (10.31-r0)
(12/27) Installing git (2.18.1-r0)
(13/27) Installing libbz2 (1.0.6-r6)
(14/27) Installing libffi (3.2.1-r4)
(15/27) Installing gdbm (1.13-r1)
(16/27) Installing xz-libs (5.2.4-r0)
(17/27) Installing sqlite-libs (3.25.3-r0)
(18/27) Installing python3 (3.6.6-r0)
(19/27) Installing py3-setuptools (39.1.0-r0)
(20/27) Installing pkgconf (1.5.3-r0)
(21/27) Installing python3-dev (3.6.6-r0)
(22/27) Installing libattr (2.4.47-r7)
(23/27) Installing libacl (2.2.52-r5)
(24/27) Installing popt (1.16-r7)
(25/27) Installing rsync (3.1.3-r1)
(26/27) Installing unzip (6.0-r4)
(27/27) Installing zip (3.0-r6)
Executing busybox-1.28.4-r3.trigger
Executing ca-certificates-20171114-r3.trigger
OK: 101 MiB in 42 packages
Removing intermediate container b51099f7ad56
 ---> 3c747a22ee4c
Step 4/6 : COPY . .
 ---> 0ceebb7395bd
Step 5/6 : RUN ./build.sh  && ./link-all.sh
 ---> Running in a2dc52e10666
git-secrets scan ok
lerna notice cli v3.13.1
lerna success found 110 packages
=============================================================================================
building...
lerna notice cli v3.13.1
lerna info Executing command in 110 packages: "npm run build"
dns_validated_certificate_handler: > [email protected] build /app/packages/@aws-cdk/aws-certificatemanager/lambda-packages/dns_validated_certificate_handler
dns_validated_certificate_handler: > echo No build
pkglint: > [email protected] build /app/tools/pkglint
pkglint: > tsc && tslint -p . && chmod +x bin/pkglint
awslint: > [email protected] build /app/tools/awslint
awslint: > tsc && chmod +x bin/awslint
dns_validated_certificate_handler: No build
cdk-build-tools: > [email protected] build /app/tools/cdk-build-tools
cdk-build-tools: > tsc && tslint -p . && chmod +x bin/cdk-build && chmod +x bin/cdk-test && chmod +x bin/cdk-watch && chmod +x bin/cdk-awslint && pkglint
simple-resource-bundler: > [email protected] build /app/packages/simple-resource-bundler
simple-resource-bundler: > cdk-build
@aws-cdk/cfnspec: > @aws-cdk/[email protected] build /app/packages/@aws-cdk/cfnspec
@aws-cdk/cfnspec: > cdk-build && node build-tools/build

...

@aws-cdk/aws-s3-deployment: + zip -qr /app/packages/@aws-cdk/aws-s3-deployment/lambda/bundle.zip .
@aws-cdk/aws-s3-deployment: + echo 'bundle: /app/packages/@aws-cdk/aws-s3-deployment/lambda/bundle.zip'
@aws-cdk/aws-s3-deployment: + rm -rf /tmp/tmp.epDFia /tmp/tmp.lCkFmB
@aws-cdk/aws-s3-deployment: bundle: /app/packages/@aws-cdk/aws-s3-deployment/lambda/bundle.zip
@aws-cdk/aws-cloudfront: > @aws-cdk/[email protected] build /app/packages/@aws-cdk/aws-cloudfront
@aws-cdk/aws-cloudfront: > cdk-build
@aws-cdk/aws-ecs: > @aws-cdk/[email protected] build /app/packages/@aws-cdk/aws-ecs
@aws-cdk/aws-ecs: > cdk-build
@aws-cdk/aws-codebuild: > @aws-cdk/[email protected] build /app/packages/@aws-cdk/aws-codebuild
@aws-cdk/aws-codebuild: > cdk-build
@aws-cdk/aws-ecs: sh: cdk-build: not found
@aws-cdk/aws-ecs: npm ERR! file sh
@aws-cdk/aws-ecs: npm ERR! code ELIFECYCLE
@aws-cdk/aws-ecs: npm ERR! errno ENOENT
@aws-cdk/aws-ecs: npm ERR! syscall spawn
@aws-cdk/aws-ecs: npm ERR! @aws-cdk/[email protected] build: `cdk-build`
@aws-cdk/aws-ecs: npm ERR! spawn ENOENT
@aws-cdk/aws-ecs: npm ERR!
@aws-cdk/aws-ecs: npm ERR! Failed at the @aws-cdk/[email protected] build script.
@aws-cdk/aws-ecs: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
@aws-cdk/aws-ecs: npm ERR! A complete log of this run can be found in:
@aws-cdk/aws-ecs: npm ERR!     /root/.npm/_logs/2019-03-05T02_04_00_649Z-debug.log
lerna ERR! npm run build exited 1 in '@aws-cdk/aws-ecs'
lerna WARN complete Waiting for 2 child processes to exit. CTRL-C to exit immediately.

real    5m34.993s
user    14m36.590s
sys 2m22.930s
❌  Last command failed. Scroll up to see errors in log (search for '!!!!!!!!').
The command '/bin/sh -c ./build.sh  && ./link-all.sh' returned a non-zero code: 1

[eladb]

I believe this is resolved