screwdrivercd/launcher:latest & Docker 18.03.0-ce, missing files

[ScottyFillups]

What happened:

When using Docker 18.03.0-ce, files are missing in the launcher image, screwdrivercd/launcher:latest:

$ docker run -it --entrypoint "/bin/sh" screwdrivercd/launcher:latest
/opt/sd # ls
bin

This causes issues when trying to run Screwdriver locally (eg. screwdriver-cd/in-a-box):

api_1    | 180404/230523.665, (1522883090290:fe5940f20e14:17:jflp8evb:10033) [request,server,error] data: Error: (HTTP code 400) unexpected - OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"/opt/sd/tini\": stat /opt/sd/tini: no such file or directory": unknown
api_1    |     at /usr/src/app/node_modules/docker-modem/lib/modem.js:254:17
api_1    |     at getCause (/usr/src/app/node_modules/docker-modem/lib/modem.js:284:7)
api_1    |     at Modem.buildPayload (/usr/src/app/node_modules/docker-modem/lib/modem.js:253:5)
api_1    |     at IncomingMessage.<anonymous> (/usr/src/app/node_modules/docker-modem/lib/modem.js:229:14)
api_1    |     at emitNone (events.js:111:20)
api_1    |     at IncomingMessage.emit (events.js:208:7)
api_1    |     at endReadableNT (_stream_readable.js:1064:12)
api_1    |     at _combinedTickCallback (internal/process/next_tick.js:138:11)
api_1    |     at process._tickDomainCallback (internal/process/next_tick.js:218:9)
api_1    | 180404/230450.290, [response,api,events] http://172.142.174.240:9001: post /v4/events {} 500 (33378ms)

What you expected to happen:

$ docker run -it --entrypoint "/bin/sh" screwdrivercd/launcher:latest
/opt/sd # ls
bin         emitter     launch      logservice  meta        sd-cmd      sd-step     tini

How to reproduce it:

1. Install Docker version 17.12.0-ce from https://docs.docker.com/docker-for-mac/release-notes/#docker-community-edition-18030-ce-mac60-2018-03-30
2. Run docker run -it --entrypoint "/bin/sh" screwdrivercd/launcher:latest
3. Run ls, note output
4. Uninstall 17.12.0-ce and install 18.03.0-ce
5. Repeat steps 1-3, compare output

Workarounds

Using Docker ^17.0.0-ce seems to work fine. Avoid 18.03.0-ce for now.

Related links

• Issue in docker/for-mac: https://github.com/docker/for-mac/issues/2772

[catto]

I had a quick look at source code of Docker.
Before 17.12, docker engine copies everything via tar.
https://github.com/docker/docker-ce/commit/f0113d4e5a76f38716e74d0611b775055da5fb1d#diff-1ed02cdca47babc3fea55881d757580eL412
After 18.02, docker engine uses containerd/continuity library which doesn't support a named pipe currently. In the for loop at the url below, it returns with error, not continue, if there is unsupported file such as a pipe, so copying is cancelled at emitter and other files will be omitted.
https://github.com/containerd/continuity/pull/101/files#diff-baa2046d08674df161187d3f68336ff6R91

Should we remove emitter pipe from launcher image and create emitter pipe just before each build?

[lgfausak]

I tested the removal of the emitter from launcher with the creation just before the build. That worked. It feels out of place to put the creation of the emitter into each executor. Perhaps the launcher or logger should create the emitter if it doesn't already exist? Or maybe create a startup script that encapsulates the mkfifo, launcher, logger actions. Anyway, just wanted to confirm that the approach suggested works.

[FenrirUnbound]

So I looked further into this, and my naive solution (provided by screwdriver-cd/launcher#177) won't fix this issue directly. In every executor strategy, the entrypoint is overridden so my change won't be seen in production.

In order to truly solve this, it needs to be applied on a per-executor basis; doing it in just the launcher won't fix it

References

• https://github.com/screwdriver-cd/executor-docker/blob/master/index.js#L173
• https://github.com/screwdriver-cd/executor-k8s/blob/master/config/pod.yaml.tim#L45
• https://github.com/screwdriver-cd/hyperctl-image/blob/master/scripts/hyper-pod-template.json#L12

[stjohnjohnson]

Is there a way to do this in the launcher or log reader code? As in create
if not exists.
On Mon, Jun 4, 2018 at 18:02 Tiffany K [e-mail] wrote:

Reopened #979 https://github.com/screwdriver-cd/screwdriver/issues/979.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/screwdriver-cd/screwdriver/issues/979#event-1662582350,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAl98eJr562b5ZRv3vDvPhU-1F81idKsks5t5diMgaJpZM4THoUr
.

[FenrirUnbound]

Is there a way to do this in the launcher or log reader code? As in create if not exists.

That part I'm not quite sure. This is the way I understand it...

The executors are pretty consistent when they start up:

1. Initiate the launcher process in the background
2. Initiate the logservervice in the background
3. Wait for background jobs to complete

Starting either the launcher or the logservice processes is a non-blocking operation; we initiate them and move on to the next. This leads me to believe that there'll be a race condition if we move this logic to either the launcher or logservice (or both).

a. If the launcher is responsible to create the fifos, but the logservice tries to read it before it's created, it'll error.
b. If the logservice is responsible to create the fifos, but the launcher tries to write to it before it's created, it'll error
c. If they're both responsible: the slower program could detect that the fifos file is missing before the faster program creates it, and when the slower program tries to create the fifos, the slower program errors out because the file already exists.

Option C might work out better, but at that point we're choosing to just push around work. IMHO, it's at the executor-level where the launcher and log-service are told where the emitter fifos exists so it should also be at the executor that's responsible for ensuring the fifos exists.

[catto]

I think the launcher has responsibility to create the fifos because it need to flush build logs to emitter first. logservice actually works triggered by flushed logs.
logservice can wait for emitter to be created and begin reading and sending logs once it is ready. So, I prefer kind of option B.

[stjohnjohnson]

There's another option that @FenrirUnbound and I talked about today. We can add a start.sh script into the launcher image that creates the emitter, starts the launcher, starts the log service, and waits for both to be complete. Then we just change all the executors to use that as the command. That way it solves the race condition AND makes updating it easier in the future.

[FenrirUnbound]

The awkward thing I noticed is that some of the executors (e.g. executor-k8s) template the pod-definition so we can't completely migrate all the commands into the script.

So, the idea I'm now at is:

0.) Change the Launcher's Docker entrypoint script to be the same as all the executor's entrypoint.
1.) Update the executor's entrypoint to use the launcher script.

I need help with updating the Launcher version so that upgrading to it is more obvious (e.g. a major or minor version bump). The executors will need a major version bump, but that can be done with a title prefix

[catto]

Make sense. I agree with that plan.
As an eventual plan, Director will also fix this issue, right?

[FenrirUnbound]

As an eventual plan, Director will also fix this issue, right?

Yes, but the Director requires the Auth server refactor.

[minz1027]

Works with docker 18 now.

➜  ~ docker run -it --entrypoint "/bin/sh" screwdrivercd/launcher:latest
/opt/sd # ls
bin                     launcher_entrypoint.sh  meta                    sd-cmd                  tini
launch                  logservice              run.sh                  sd-step
/opt/sd #