Aws-cdk: [core] allow asset bundling on docker remote host

[alekitto]

Core AssetStaging is now exposing a bundling option key allowing assets to be built before being uploaded through a docker image.
Unfortunately the docker command is now using volume mount flags to mount input and output folder, making it impossible to execute if docker is set to run on a remote host or in a docker-in-docker env.

Use Case

I encountered this problem trying to build a couple of lambdas (and a custom resource provider) written in typescript on a Gitlab CI instance with docker-in-docker or executing docker commands via docker-machine.

Proposed Solution

My proposal is to create two temporary volumes on target docker env, one for inputs and one for outputs.
Then the first volume can be filled running a busybox image as helper and calling docker cp to fill the volume.
Once the cp has finished the helper is stopped and the build command is invoked.
After that, if exit code is 0, another helper is started and the docker cp is used to copy outputs back to the cdk env.

• [x] :wave: I may be able to implement this feature request
• [ ] :warning: This feature might incur a breaking change

---

This is a :rocket: Feature Request

[rinfield]

I'm also struggling this problem with using aws-lambda-nodejs in a VSCode Remote Container environment.
(/var/run/docker.sock sharing)

Project workspace is already mounted...

I wish the good old non-docker implementation comes back for compatibility.

[eladb]

Copy @jogold

The plot thickens... (which is a good thing!)

[jogold]

@alekitto thanks for opening this issue.

Can you detail the exact problem you're seeing?

making it impossible to execute if docker is set to run on a remote host or in a docker-in-docker env.

It currently runs without issues in CodeBuild which is a docker-in-docker setup. Can you elaborate on the Gitlab CI setup maybe?

Is your solution something along those lines?

$ docker volume create asset-input
$ docker volume create asset-output
$ docker run -v asset-input:/asset-input -v asset-output:/asset-output --name helper busybox
$ docker cp <asset source> helper:/asset-input
$ docker run --rm -v asset-input:/asset-input -v asset-output/asset-output <user command>
$ docker cp helper:/asset-output <staged bundling dir>
$ docker rm helper
$ docker volume rm asset-input asset-output

For the @aws-lamda-nodejs.NodejsFunction we currently mount the projet root as the asset source, for a large repo (or even monorepo) this represents lots of files, how does this affect the docker cp?

[alekitto]

Can you detail the exact problem you're seeing?

I'm trying to build compile a lambda from typescript down to js to be executed on node_10.x runtime.
A docker:dind service container is run and correcly responds to tcp://docker:2375, then the build starts.
When executing cdk diff, I can see that another node container has been pulled and run by cdk with the arguments set in code bundlingOptions.
For a coincidence, Gitlab CI mounts the build folder at the same path in all the containers, so the build container and the docker container shares the same cdk code at the same path.
The dependencies are installed correcly, the compilation executes successfully, then an error is thrown stating that Bundling did not produce any output. Inspecting the docker:dind container I noticed that the temporary path created by cdk to be mounted as asset-output is created on both containers, but only the one on the docker container is populated with the compiled script.
That's because the -v docker option mounts maps on the container a volume (or a path) created on the docker host, not the one calling the docker cli.

It currently runs without issues in CodeBuild which is a docker-in-docker setup. Can you elaborate on the Gitlab CI setup maybe?

The build container is a node:14-buster image with the docker cli added. A service docker:dind image is run for the job, responding as docker hostname.

Is your solution something along those lines?

$ docker volume create asset-input
$ docker volume create asset-output
$ docker run -v asset-input:/asset-input -v asset-output:/asset-output --name helper busybox
$ docker cp <asset source> helper:/asset-input
$ docker run --rm -v asset-input:/asset-input -v asset-output/asset-output <user command>
$ docker cp helper:/asset-output <staged bundling dir>
$ docker rm helper
$ docker volume rm asset-input asset-output

Yes, with auto-generated id appended to volume names to avoid collisions.

For the @aws-lamda-nodejs.NodejsFunction we currently mount the projet root as the asset source, for a large repo (or even monorepo) this represents lots of files, how does this affect the docker cp?

IIRC the docker cp command builds a tar archive internally and streams it to the docker engine, but I don't know if the number of files can affect the performance significantly compared to the size of the files.

[alekitto]

Could be related to #8544?
Creating volumes for input and output can avoid osxfs performance issues on io intensive operations (skipping continuous syncs between macos filesystem and the virtual machine hosting docker).

[alekitto]

UPDATE: my current workaround (valid with docker executor on gitlab-runner only) is to configure gitlab-runner to share the /tmp path mounting it as a volume across all the containers in the same build job.
This way the build (cdk) container and the docker:dind container share the same /tmp, allowing cdk to find the output files.

[jogold]

UPDATE: my current workaround (valid with docker executor on gitlab-runner only) is to configure gitlab-runner to share the /tmp path mounting it as a volume across all the containers in the same build job.
This way the build (cdk) container and the docker:dind container share the same /tmp, allowing cdk to find the output files.

@alekitto This has been fixed in #8601 and released in v1.46.0, can you try with a version >= 1.46.0?

[alekitto]

@alekitto This has been fixed in #8601 and released in v1.46.0, can you try with a version >= 1.46.0?

I tried with 1.47 from local, but i cannot make it working when using docker-machine (it throws package.json not exists error), because the input files are on my computer and it tries to mount a non-existent path on the docker host.

[jogold]

it throws package.json not exists error

CDK or inside the container?

it tries to mount a non-existent path

which path?

[alekitto]

CDK or inside the container?

The bundling container, when executing npm install.

which path?

It tries to mount /home/alekitto/my-project-folder/cdk/lib/construct/kms/lambda which is the path of the code to be compiled on my local machine, currently non-existent in a newly created docker-machine where the docker engine is hosted.

[jogold]

currently non-existent in a newly created docker-machine where the docker engine is hosted.

Not sure I'm following here... you can detail this?

[alekitto]

Not sure I'm following here... you can detail this?

I'm not currently executing the docker engine on my local machine.
After docker-machine create --driver amazonec2 docker-aws a ec2 instance is provisioned to run the docker engine, exposing the docker tcp port (2376).
After provisioning is finished I run eval $(docker-machine env docker-aws) to be able to run docker command on the newly created host.

Then I try to run cdk diff which invokes the docker cli (now pointing to the remote docker host).

The command built by cdk is docker run --rm -v /home/alekitto/my-project-folder/cdk/lib/construct/kms/lambda:/asset-input -v /home/alekitto/my-project-folder/cdk/.cdk.staging:/asset-output node:14-buster sh -c "npm install && npm run build && cp app.js /asset-input"

The problem is that /home/alekitto/my-project-folder/cdk exists on my local machine, but not on the docker host.
When launching that command, the docker host is instructed to mount the specified path on the host machine into the container, but on the host machine that path is non-existent.
The docker engine then creates all the folders, which are obviously empty, and mounts them into the new container.
When the container tries to execute npm install the command exits with error package.json does not exist, because the file is not present on the docker host at the specified path.

Inspecting the docker machine via SSH I can see the all the folder structure, but no file is present, because nothing has been copied to the docker host from my local computer.