Actions-runner-controller: Issue with running `docker run` for integration testing

[chenrui333]

I have run into some issue with running docker run xx-image after moving some of the workflow into self-hosted runner. I could not figure it out, but feel it might be helpful to post in here and discuss.

The flow looks like something below:

docker run -it --rm \
    -v $(PWD):/local \
    xx-image:latest \
    /local/e2e.sh

The error log that i keep getting:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"/local/e2e.sh\": stat /local/e2e.sh: no such file or directory": unknown.

While it works before, I suspect it might be related to the dind docker image it used for the runner. Does the dind docker engine impact the volume mount??

Please shed some insight on fixing the issue and let me know if this makes sense. Thanks!!

[mumoshu]

@chenrui333 Ah yes, the dind container and the runner container needs explicit volume & volume mounts configured for that to work.

Could you configure spec.volumes and spec.volumeMounts appropriately in your runner/runnerreplicaset/runnerdeployment spec?

If you could share me a few examples of paths returned by $(PWD), I can guide you how you should configure volumes/volumeMounts.

FYI, the mapping between the runner spec and the runner pod volumes/mounts is implemented in https://github.com/summerwind/actions-runner-controller/blob/b8b4ef4b60a45fe647a19a6f56130eb3449a9a99/controllers/runner_controller.go#L416.

[chenrui333]

Let me check on my environment really quick, thanks for clarifying the mystery.

Below is the PWD environment variable

runner@github-action-runner-kbjn6-dtxg7:/$ echo $PWD
/

[chenrui333]

@mumoshu any advice or do you need more info from my side?

[mumoshu]

Ah, that's too bad. We can't share / between the runner container and the find container cuz it breaks both sides. We need to change either the runner side or your workflow to use another directory.

I thought it was /github/workspace in the managed GitHub Actions runner.

Would you mind running find -maxdepth 3 /github/workspace in your workflow to figure out what's in your workflow env currently?

Depending on that, we can perhaps change the runner container to create necessary directories and set proper workdir so that you can just mount $(PWD) to docker containers.

[chenrui333]

I dont think I have /github/workflow path there.

Here is the debug output:

Run find -maxdepth 3 /github/workspace
  find -maxdepth 3 /github/workspace
  shell: /bin/bash -e {0}
find: paths must precede expression: `/github/workspace'
##[error]Process completed with exit code 1.

[chenrui333]

Here are some env values that I found on the runner as well:

OLDPWD=/runner/bin
PWD=/runner
HOME=/home/runner

[chenrui333]

@mumoshu this issue might be related to this?
https://github.com/actions/runner/issues/406

[mumoshu]

@chenrui333 Thanks! That does help. I believe ./config.sh --work /home/github/work is the key - I believe that it worth trying to update our Dockerfile to use ./config.sh --work /home/github/work --once rather than ./config.sh --once which we have today.

[chenrui333]

Coo, let me do a quick check.

[chenrui333]

This is the current setup:

cd /runner
./config.sh --unattended --replace --name "${RUNNER_NAME}" --url "https://github.com/${RUNNER_REPO}" --token "${RUNNER_TOKEN}"

unset RUNNER_NAME RUNNER_REPO RUNNER_TOKEN
exec ./run.sh --once

So I should update ./config.sh to include work option?

[mumoshu]

@chenrui333 Ah yes you're absolutely correct! In my previous comment, I was confusing between config.sh and run.sh.

[chenrui333]

Still having the same error message:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"/local/e2e.sh\": stat /local/e2e.sh: no such file or directory": unknown.

[chenrui333]

In my test PR, I am still seeing the same thing

Run find -maxdepth 3 /github/workspace
  find -maxdepth 3 /github/workspace
  shell: /bin/bash -e {0}
find: paths must precede expression: `/github/workspace'
##[error]Process completed with exit code 1.

[summerwind]

@chenrui333 Let me confirm the details. Did you run the following docker command in GitHub Actions Workflow?

$ docker run -it --rm \
    -v $(PWD):/local \
    xx-image:latest \
    /local/e2e.sh

If so, what kind of output do you get if you run the following command in the same GitHub Actions workflow?

$ ls -al 
$ echo $PWD

[chenrui333]

@summerwind this is what I have got

drwxr-xr-x  4 runner root  145 Apr 18 18:48 .
drwxr-xr-x 61 runner root 4096 Apr 18 18:48 ..
-rw-r--r--  1 runner root    6 Apr 18 18:48 .python-version
-rw-r--r--  1 runner root 1631 Apr 18 18:48 BUILD
-rw-r--r--  1 runner root 1041 Apr 18 18:48 Dockerfile
-rw-r--r--  1 runner root 7340 Apr 18 18:48 Makefile
-rw-r--r--  1 runner root 2793 Apr 18 18:48 README.md
-rwxr-xr-x  1 runner root 1367 Apr 18 18:48 e2e.sh
drwxr-xr-x  3 runner root   47 Apr 18 18:48 infra
-rwxr-xr-x  1 runner root  986 Apr 18 18:48 run.sh
drwxr-xr-x  3 runner root   18 Apr 18 18:48 src
/runner/_work/xx/xx/modules/api-server

[summerwind]

This problem was reproduced in my environment.

The problem is that dockerd is running in a sidecar and the volume mount (the -v option) does not work as intended because files are not shared between containers.

Let me consider a good solution to solve this problem.

[mumoshu]

@summerwind Thanks for taking look into this!

I was wondering if this is where local persistent volumes shine.

[chenrui333]

Thanks @summerwind!

[summerwind]

I created PR #40 to try and figure out how to combine the runner and docker containers into one. Feedbacks are welcome!

[summerwind]

Reopen this due to accidentally closed.

[summerwind]

Fixed in #45 and v0.4.3 by sharing working directory between runner and docker container.
https://github.com/summerwind/actions-runner-controller/releases/tag/v0.4.3

[summerwind]

I'm going to close this, but feel free to reopen if this is still an issue.

[chenrui333]

Sure, sounds good!