Renovate: Bundler update fails

[vestimir]

What Renovate type are you using?

Self-hosted with docker in docker executor.

Describe the bug

Renovate is detecting the Gemfile in the “Extraction statistics” step, but failing to read it in the docker execution. However, the rest of the dependency updates are handlet right – packages.json, Dockerfile and .ruby-version.

Did you see anything helpful in debug logs?

DEBUG: latest commit (repository=example-group/example-project, branch=renovate/major-all-bundler)
       "branchName": "master",
       "latestCommitDate": "2020-04-11 14:48:43 +0300"
DEBUG: getBranchPr(renovate/major-all-bundler) (repository=example-group/example-project, branch=renovate/major-all-bundler)
DEBUG: branchExists=false (repository=example-group/example-project, branch=renovate/major-all-bundler)
DEBUG: Branch has 2 upgrade(s) (repository=example-group/example-project, branch=renovate/major-all-bundler)
DEBUG: recreateClosed is true (repository=example-group/example-project, branch=renovate/major-all-bundler)
DEBUG: Checking schedule(at any time, null) (repository=example-group/example-project, branch=renovate/major-all-bundler)
DEBUG: No schedule defined (repository=example-group/example-project, branch=renovate/major-all-bundler)
DEBUG: Branch needs creating (repository=example-group/example-project, branch=renovate/major-all-bundler)
DEBUG: Using parentBranch: undefined (repository=example-group/example-project, branch=renovate/major-all-bundler)
DEBUG: manager.getUpdatedPackageFiles() (repository=example-group/example-project, branch=renovate/major-all-bundler)
DEBUG: Updating packageFile content (repository=example-group/example-project, branch=renovate/major-all-bundler)
DEBUG: Updating packageFile content (repository=example-group/example-project, branch=renovate/major-all-bundler)
DEBUG: bundler.updateArtifacts(Gemfile) (repository=example-group/example-project, branch=renovate/major-all-bundler)
DEBUG: Found bundler version (repository=example-group/example-project, branch=renovate/major-all-bundler)
       "bundlerVersion": "2.1.4"
DEBUG: Using ruby version specified in .ruby-version (repository=example-group/example-project, branch=renovate/major-all-bundler)
DEBUG: Using docker to execute (repository=example-group/example-project, branch=renovate/major-all-bundler)
DEBUG: Found ruby version constraint - checking for a compatible renovate/ruby image to use (repository=example-group/example-project, branch=renovate/major-all-bundler)
       "constraint": "2.5.1"
DEBUG: getLabels(https://index.docker.io, renovate/ruby, latest) (repository=example-group/example-project, branch=renovate/major-all-bundler)
DEBUG: getManifestResponse(https://index.docker.io, renovate/ruby, latest) (repository=example-group/example-project, branch=renovate/major-all-bundler)
DEBUG: Found compatible ruby version (repository=example-group/example-project, branch=renovate/major-all-bundler)
       "constraint": "2.5.1",
       "version": "2.5.1"
DEBUG: Fetching Docker image: renovate/ruby:2.5.1 (repository=example-group/example-project, branch=renovate/major-all-bundler)
DEBUG: Finished fetching Docker image (repository=example-group/example-project, branch=renovate/major-all-bundler)
DEBUG: Executing command (repository=example-group/example-project, branch=renovate/major-all-bundler)
       "command": "docker run --rm --name=renovate_ruby --label=renovate_child -v \"/tmp/renovate/repos/gitlab/example-group/example-project\":\"/tmp/renovate/repos/gitlab/example-group/example-project\" -v \"/tmp/renovate/cache\":\"/tmp/renovate/cache\" -w \"/tmp/renovate/repos/gitlab/example-group/example-project\" renovate/ruby:2.5.1 bash -l -c \"ruby --version && gem install bundler -v 2.1.4 --no-document && bundle lock --update puma rails\""
 WARN: Gemfile.lock update failed due to unknown reason - skipping branch (repository=example-group/example-project, branch=renovate/major-all-bundler)
       "err": {
         "killed": false,
         "code": 10,
         "signal": null,
         "cmd": "docker run --rm --name=renovate_ruby --label=renovate_child -v \"/tmp/renovate/repos/gitlab/example-group/example-project\":\"/tmp/renovate/repos/gitlab/example-group/example-project\" -v \"/tmp/renovate/cache\":\"/tmp/renovate/cache\" -w \"/tmp/renovate/repos/gitlab/example-group/example-project\" renovate/ruby:2.5.1 bash -l -c \"ruby --version && gem install bundler -v 2.1.4 --no-document && bundle lock --update puma rails\"",
         "stdout": "ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux]\nSuccessfully installed bundler-2.1.4\n1 gem installed\n",
         "stderr": "Could not locate Gemfile\n",
         "message": "Command failed: docker run --rm --name=renovate_ruby --label=renovate_child -v \"/tmp/renovate/repos/gitlab/example-group/example-project\":\"/tmp/renovate/repos/gitlab/example-group/example-project\" -v \"/tmp/renovate/cache\":\"/tmp/renovate/cache\" -w \"/tmp/renovate/repos/gitlab/example-group/example-project\" renovate/ruby:2.5.1 bash -l -c \"ruby --version && gem install bundler -v 2.1.4 --no-document && bundle lock --update puma rails\"\nCould not locate Gemfile\n",
         "stack": "Error: Command failed: docker run --rm --name=renovate_ruby --label=renovate_child -v \"/tmp/renovate/repos/gitlab/example-group/example-project\":\"/tmp/renovate/repos/gitlab/example-group/example-project\" -v \"/tmp/renovate/cache\":\"/tmp/renovate/cache\" -w \"/tmp/renovate/repos/gitlab/example-group/example-project\" renovate/ruby:2.5.1 bash -l -c \"ruby --version && gem install bundler -v 2.1.4 --no-document && bundle lock --update puma rails\"\nCould not locate Gemfile\n\n    at ChildProcess.exithandler (child_process.js:303:12)\n    at ChildProcess.emit (events.js:311:20)\n    at ChildProcess.EventEmitter.emit (domain.js:482:12)\n    at maybeClose (internal/child_process.js:1021:16)\n    at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)"
       }

To Reproduce

Here's my .gitlab-yml:

image: docker:dind

services:
  - docker:dind

renovate:
  stage: build
  tags:
    - do-david-docker
  script:
    - docker run --tty
      -e GITLAB_TOKEN="$GITLAB_TOKEN"
      --rm
      -v /var/run/docker.sock:/var/run/docker.sock
      renovate/renovate
      --log-level=debug
      --binary-source=docker
      --platform gitlab
      --endpoint https://git.my-domain.com/api/v4/
      --token $GITLAB_TOKEN
      $(cat repositories.txt | xargs)
  only:
    - master

Additional context

This is a standard Rails + Webpacker application, so everything is located in the root folder of the project (Gemfile and package.json).

For some reason I couldn't figure out how to mount the config file, so I switched to setting the config with cli args.

[viceice]

You need to mount /tmp to renovate image.

You can mount the config.js to /usr/src/app/config.js

[vestimir]

I've updated the gitlab file like this:

image: docker:dind

services:
  - docker:dind

renovate:
  stage: build
  tags:
    - do-david-docker
  script:
    - docker run --tty
      -e GITLAB_TOKEN="$GITLAB_TOKEN"
      --rm
      -v /tmp:/tmp
      -v /var/run/docker.sock:/var/run/docker.sock
      renovate/renovate
      --log-level=debug
      --binary-source=docker
      --platform gitlab
      --endpoint https://git.my-domain.com/api/v4/
      --token $GITLAB_TOKEN
      $(cat repositories.txt | xargs)
  only:
    - master

Now the error is Fatal error: EACCES: permission denied, rmdir '/tmp/renovate/repos/gitlab/example-group/example-project'

Please note that the executor is privileged.

[viceice]

Renovate docker images running as user 1000, this needs write access to tmp

[vestimir]

I added chown 1000 /tmp right before the docker run in gitlab-ci, but it still reports the same error.

[sho918]

I'm trying to do something similar, I have been working well with this setting.
Run with TLS-enabled runner (I don't know if it's necessary).
https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#tls-enabled

image: docker:19.03-dind

stages:
  - renovate

renovate:
  stage: renovate
  services:
    - docker:19.03-dind
  tags:
    - tls
  script:
    - docker run --tty
      -e RENOVATE_PLATFORM=gitlab
      -e RENOVATE_ENDPOINT=$CI_API_V4_URL
      -e RENOVATE_TOKEN=$RENOVATE_TOKEN
      -e GITHUB_COM_TOKEN=$GITHUB_COM_TOKEN
      --rm
      -v /tmp:/tmp
      -v /var/run/docker.sock:/var/run/docker.sock
      -v $PWD/config.js:/usr/src/app/config.js
      renovate/renovate
      --log-level=debug
  only:
    - schedules

[rohansingh]

@vestimir

I added chown 1000 /tmp right before the docker run in gitlab-ci, but it still reports the same error.

That's unlikely to work unless your Docker image also has a user with ID 1000 (check /etc/passwd).

[rohansingh]

@viceice

You need to mount /tmp to renovate image.

Renovate docker images running as user 1000, this needs write access to tmp

This doesn't seem to be sufficient. For example, I've mounted /tmp from the host and it is world-writeable. However, renovate itself runs as root in my situation, and it creates a /tmp/renovate that is only writeable by root.

Is there a reason to be running as ubuntu in the Docker image rather than root? While running stuff as root is not great, it seems to be the convention in Docker images and it avoids issues like this.

And note that this isn't just a problem with Docker-in-Docker. Anyone running on a system where renovate isn't running as uid 1000 would have the same issue, right? Perhaps the fix is to have Renovate make /tmp/renovate world-writeable?

[viceice]

@rohansingh If you can't run renovate as user 1000, you should configure dockerUser to match, so the sidecontainer can read and write to the renovate path

[rohansingh]

@viceice Thanks, total oversight on my part 😄

[alexanderadam]

Is there a reason to be running as ubuntu in the Docker image rather than root?

Yes, it is a security concern.

While running stuff as root is not great, it seems to be the convention in Docker images and it avoids issues like this.

Well, even the folks who made Docker say in their best practices guide:

If a service _can_ run without privileges, use USER to change to a non-root user.

And nearly every service can run without privileges. The reason why so many images run as root within Docker are

1. Often people don't know about the risks
2. They don't even read the Docker guides
3. You have to actively use the USER command. Which you usually just do if you _know_ about it. Switching to an unprivileged user is also often connected with more work on the development side.

[rohansingh]

@alexanderadam Yes, I understand all of that. If you run as root in a container, then your entire system is potentially hosed if someone manages to break out of the container. Whereas if you run as any other user, then an adversary would need to break out and also need to somehow escalate privileges.

And yet, it doesn't change the fact the current default configuration breaks in any scenario where Renovate is not run with uid 1000 on the host. On any given host, perhaps uid 1000 has special privileges of some sort and it's not safe to run the Renovate container with that uid either.

In fact, the more I think about it, if I run renovate as a certain user, I expect it to use that same user for everything that it does.

Anyway, as @viceice pointed out, there is a config to override this so that works for me. But I do think it would be more ergonomic if, by default, Renovate always ran the container with its own uid and gid. That's effectively what happens if you don't use --binary-source=docker, right?

[viceice]

Yes, if you unset bindarySource renovate will run the tools as simple child-process with current rights.

But you are responsible to install all required tools in the renovate image (if you use the slim image) or you need to use the full image

[rohansingh]

Yes, if you unset bindarySource renovate will run the tools as simple child-process with current rights.

So why not do the same for the container if using the Docker binary source? At least by default, why not use the current process's uid/gid to launch the container?

That would ensure that this is never an issue regardless of whether using the slim image, your own custom image, or even running on bare metal but with a different uid.

[rarkins]

There are three components in play here:

• Renovate's container
• The sibling containers that Renovate spins up
• The host

When Renovate creates a sibling container, it thinks it's creating a child container and mapping its own file system, but actually the sibling container will map from the host.

Note: this describes the sibling/sidecar container mode, not the docker-in-docker mode which is not a supported deployment model.

[bilby91]

I've been running with GitLab this way without any issues:

image: renovate/renovate:19.188.0
variables:
  DOCKER_HOST: tcp://127.0.0.1:2375
  # Use overlay2 instead of vfs to improve performance
  DOCKER_DRIVER: overlay2
  # Shared directory between renovate container and spawned containers.
  # It's crutial to work on a folder under $CI_PROJECT_DIR in order
  # to have the same paths between host and DIND.
  RENOVATE_BASE_DIR_PATH: $CI_PROJECT_DIR/renovate
services:
  - docker:18.09.8-dind
before_script:
  # Configure ssh key in order to be able to push new branches
  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh
  - echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
  - chmod 600 ~/.ssh/id_rsa

  # Prepare renovate directory
  - mkdir $RENOVATE_BASE_DIR_PATH
renovate-run:on-schedule:
  only:
    - schedules
  script:
    - node /usr/src/app/dist/renovate.js --binary-source=docker --base-dir $RENOVATE_BASE_DIR_PATH $RENOVATE_EXTRA_FLAGS

Having said that, I got a new error recently when we added a new dependency that came from GitHub directly.

I'm getting this error now:

There was an error while trying to write to\n`/usr/local/ruby/2.7.1/lib/ruby/gems/2.7.0/cache/bundler/git`. It is likely that\nyou need to grant write permissions for that path.\n

I'm not sure if there is a why to tell bundler where to cache this gems.

[viceice]

@bilby91 please use renovate --binary-source=docker --base-dir $RENOVATE_BASE_DIR_PATH $RENOVATE_EXTRA_FLAGS as start script

[bilby91]

@viceice Thanks for the quick response. What do you mean by start script ?

Thanks

[viceice]

renovate-run:on-schedule:
  only:
    - schedules
  script:
    - renovate --binary-source=docker --base-dir $RENOVATE_BASE_DIR_PATH $RENOVATE_EXTRA_FLAGS

It also looks like you are running renovate as root so you need to configure dockerUser as root too. See comments above.

[bilby91]

@viceice Got it!

I've been running renovate on GitLab with my previous setup for some time without any issues, this seems to start only when I added a git dependency.

Also, when I run renovate on my mac (which doesn't run docker as root) I have the same issue.

[viceice]

@bilby91 You should use the slim image when you use docker, as it has all requirements to run.

Currently preparing some docs:
https://github.com/renovatebot/docker-renovate/blob/07e8b524ec7202a809bd62656852d3fe622656f6/docs/gitlab.md#renovate-slim-with-docker-in-docker

[bilby91]

@viceice you are saying that if I move to slim the problem will go away?

I'll give it a try then!

[viceice]

@bilby91 Please check working solutions above, but i you like to use dind we recommend to use the slim docker image

[bilby91]

@viceice Same issue using the slim version. I think this is something unrelated to how I run docker because this was previously working for me and it stopped working while staying on the same version of renovate.

[viceice]

@bilby91 can you share a complete sample so I can try it on my own gitlab instance

[bilby91]

@viceice

I just tried this modified version with the same problematic result:

image: renovate/renovate:19.239-slim
variables:
  DOCKER_HOST: tcp://127.0.0.1:2375
  # Use overlay2 instead of vfs to improve performance
  DOCKER_DRIVER: overlay2
  # Shared directory between renovate container and spawned containers.
  # It's crutial to work on a folder under $CI_PROJECT_DIR in order
  # to have the same paths between host and DIND.
  RENOVATE_BASE_DIR_PATH: $CI_PROJECT_DIR/renovate
services:
  - docker:18.09.8-dind
before_script:
  # Configure ssh key in order to be able to push new branches
  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh
  - echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
  - chmod 600 ~/.ssh/id_rsa

  # Prepare renovate directory
  - mkdir $RENOVATE_BASE_DIR_PATH
renovate-run:on-schedule:
  only:
    - schedules
  script:
    - node /usr/src/app/dist/renovate.js --binary-source=docker --base-dir $RENOVATE_BASE_DIR_PATH $RENOVATE_EXTRA_FLAGS

$RENOVATE_EXTRA_FLAGS it's just adding the --dry-run flag.

[viceice]

Can you also share the a minimal repo which fails?

[bilby91]

@viceice At the moment I don't have an easy way to make my repo sharable.

Having said that, I found an interesting thing. With renovate/ruby-2.6.4 I don't have this issue, with renovate/ruby-2.7.1 the problem shows up.

Has something changed in terms of how the image is built ?

I found that some things are installed in different places.

For 2.6, the bundler cache directory lives here /usr/local/lib/ruby/gems/2.6.0/cache/
For 2.7, the bundler cache directory lives here /usr/local/ruby/2.7.1/lib/ruby/gems/2.7.0/cache

This explains my problem.

[bilby91]

@viceice This looks suspicious, I'm not 100% sure yet if timing matches. I think it does.

https://github.com/renovatebot/docker-ruby/commit/3a12ef5a4c6fb2ae6067b67350bfd2a1565af367#diff-3254677a7917c6c01f55212f86c57fbf

[bilby91]

This line seems to have been removed:

chmod -R a+rw /usr/local

[viceice]

@bilby91 looks like your issue is not related to gitlab. Looks like bundler tries to write to global cache instead of user home. I'll check it.

[viceice]

issue resolved, need to wait for updated docker images currently rebuilding

[viceice]

for any other gitlab related hosting please check the new samples

[viceice]

@vestimir @bilby91 Please try again, the ruby images are fixed now

[bilby91]

@viceice Stared working again with the new image.

Thanks!