Docker: Another /var/jenkins_home/copy_reference_file.log: Permission denied

[alexlatchford]

Hey guys,

Been trying to fix this on my own for nearly 6 hours now and none of the workarounds seem to be working for me I've tried. A bit of background:

• I'm on AWS, running a CentOS 7 AMI
• CentOS has the default OS user (centos) as uid 1000. Not sure if this is the problem? http://stackoverflow.com/a/34709344/724251
• Using saltstack to deploy:
  
  
  
  • to mount an ext4 volume to /var/jenkins
  
  
  • to run useradd -u 1001 -d /var/jenkins jenkins. Suggested by https://github.com/jenkinsci/docker/issues/177#issuecomment-163681464
  
  
  • to run chown -R jenkins:jenkins /var/jenkins
  
  
  • to run chmod 755 /var/jenkins
  
  
  • to run docker run -p 8080:8080 -p 50000:50000 -v /var/jenkins:/var/jenkins_home -u 1001 -t jenkins:2.6.
  
  

When I do docker run I'm getting:

[root@jenkins var]# df -h
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/xvdb        50G   53M   47G   1% /var/jenkins
[root@jenkins var]# ls -l /var/
total 12
...
drwxr-xr-x.  2 jenkins jenkins 4096 May 28 01:41 jenkins
...
[root@jenkins var]# cat /etc/passwd
...
jenkins:x:1001:1001::/var/jenkins:/bin/bash
[root@jenkins var]# docker run -p 8080:8080 -p 50000:50000 -v /var/jenkins:/var/jenkins_home -u 1001 -t jenkins:2.6
/usr/local/bin/jenkins.sh: line 25: /var/jenkins_home/copy_reference_file.log: Permission denied

I've tried fixes I've found on previous stackoverflow/gh issues to no avail:

1. extending the Dockerfile to create my own custom image and added in:

USER root RUN chown -R jenkins:jenkins /var/jenkins_home USER jenkins

Suggested by https://github.com/jenkinsci/docker/pull/225#issuecomment-203986508

1. running chmod -R 777 /var/jenkins on the host. This is the weirdest one, have no clue why this isn't working after that! Suggested by https://github.com/jenkinsci/docker/issues/177#issuecomment-171345387
2. Starts fine if I drop the volume mount, (obviously I'd like the persistent EBS storage though).
3. Same error occurs if I create a folder on the local fs, eg. /var/jenkins2 (not externally mounted disk) either.

What I thought was going to be a super simple project has taken all day :/ Anyone know what I'm doing wrong?

Cheers, Alex

[carlossg]

Try

docker run -v /var/jenkins:/var/jenkins_home -u 1001 -ti --entrypoint ls jenkins -alF /var/jenkins_home
docker run -v /var/jenkins:/var/jenkins_home -u 1001 -ti --entrypoint touch jenkins /var/jenkins_home/test

[alexlatchford]

Hey Carlos,

No lucj, here's what I get:

[root@jenkins var]# docker run -v /var/jenkins:/var/jenkins_home -u 1001 -ti --entrypoint ls jenkins:2.6 -alF /var/jenkins_home
Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
ls: cannot open directory /var/jenkins_home: Permission denied
[root@jenkins var]# docker run -v /var/jenkins:/var/jenkins_home -u 1001 -ti --entrypoint touch jenkins:2.6 /var/jenkins_home/test
Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
touch: cannot touch ‘/var/jenkins_home/test’: Permission denied

Also tried:

[root@jenkins var]# docker run -v /var/jenkins:/var/jenkins_home -u 1001 -ti --entrypoint chown jenkins:2.6 -R jenkins:jenkins /var/jenkins_home
Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
chown: cannot read directory ‘/var/jenkins_home’: Permission denied
[root@jenkins var]# docker run -v /var/jenkins:/var/jenkins_home -u 1001 -ti --entrypoint ls jenkins:2.6 -l /var/
Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
total 12
...
drwxr-xr-x.  2 1001  1001 4096 May 28 01:41 jenkins_home
...

[bbriggs]

I'm running into a very similar issue. Jenkins runs as UID 1000 in the container, but UID 1000 is almost always already occupied by another user on the host.

In my opinion, the jenkins user in the container should be created with a non-conflicting UID (like 10000 or something?) so we don't run into issues where we have to map whatever user was lucky enough to land UID 1000 on the host to the container volumes or kludge permissions with a chmod 777

[carlossg]

@bbriggs your case is solved by passing -u 10000 to docker run

[carlossg]

@alexlatchford your issue must be selinux, you can mount any dir into the container, check http://www.projectatomic.io/blog/2015/06/using-volumes-with-docker-can-cause-problems-with-selinux/

[bbriggs]

That worked, but it just raised another issue. When using the SSH agent/Git repo provider, I got a notice that uid 10000 does not exist, so I ended up having to create a dummy account _inside_ the container with that UID anyways.

If not for that secondary problem, I'd say that the -u ${OTHER_UID} thing is a decent solution. Given that it caused more problems, can we please revist the idea of renumbering the jenkins user?

On Tue, Jun 07, 2016 at 11:42:27AM -0700, Carlos Sanchez wrote:

@bbriggs your case is solved by passing -u 10000 to docker run

---

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/jenkinsci/docker/issues/277#issuecomment-224375408

[sazulo]

I had the same problem when sharing directly an host directory so I switched to the following :

# Create new volume container, inheriting from jenkins container
docker create -v /var/jenkins_home --name jenkins-data jenkinsci/jenkins
# Starting jenkins container, using previously created volume container.
docker run -d --name jenkins --hostname jenkins -p 8080:8080 -p 50000:50000 --volumes-from jenkins-data jenkinsci/jenkins

And if I need access to the files , I just use docker inspect -f '{{ .Mounts }}' jenkins to see where the files are located.

[bbriggs]

@sazulo If you are doing backups on the volume container, how are you doing it?

[sazulo]

I didn't need backup for now, but I just did a little research and found this article

I did some testing and the following seems to do the trick ( adapt as you see fit ) :

Backup

# This will create a backup.tar file in the current directory of the host.
# You might want to change $(pwd) to a specific directory
docker run --rm \
           --volumes-from jenkins-data \
           -v $(pwd):/backup \
           -u root:root \
           jenkinsci/jenkins \
           tar cvf  /backup/backup.tar /var/jenkins_home

Restore

# First create a new empty datastore that will receive the backup
docker run \
          -v /var/jenkins_home \
          --name jenkins-data-2 \
          jenkinsci/jenkins \
          /bin/bash
# Then mount the new datastore and the host's backup directory. Change 
# $(pwd) to the same value as the backup command line if needed,
# and then just extract the backup tar .
docker run --rm \
          --volumes-from jenkins-data-2 \
          -v $(pwd):/backup \
          -u root:root \
          jenkinsci/jenkins \
          bash -c "cd /var/jenkins_home && tar -xvf /backup/backup.tar"

After this you can just create a new jenkins container and mount jenkins-data-2 to /var/jenkins_home.

Edit :

This will obviously copy everything in /var/jenkins_home, from configuration files to old builds and workspaces, and might take a lot of space. I suggest that you save only what you need

[bbriggs]

Thanks for that. For now, I just went with the solution I wrote out here: http://brenbriggs.com/2016/06/08/jenkins-on-docker-via-puppet.html

tl;dr:

• Create a user with a predictable UID
• Create data directory and set ownership to said user
• Run the container with -u ${predictable_uid} option
• Create dummy user inside container with a matching UID of user outside the container so that git/ssh will work.

Here is the container I created for just such a thing (It may still be building, automated builds are taking a long time today):

https://hub.docker.com/r/wusa/jenkins-puppet/

[gotgenes]

@bbriggs, this just bit me in the butt, too:

When using the SSH agent/Git repo provider, I got a notice that uid 10000 does not exist

Running the jenkins container as -u <uid_on_host_but_not_in_container> causes more issues than it solves. In my specific case, a job that calls git commit errors with the following:

   fatal: unable to look up current user in the passwd file: no such user

Several users of Jenkins Docker have requested the project change the default UID of the jenkins user. See discussions in #112, #153, #154, #165, #177. The request keeps getting dismissed with "run with -u <some_other_uid>". As described here and in the other issues, this is not a foolproof workaround.

Right now, users of Jenkins Docker are left to perform workarounds by following one of two methods:

1. Create a new Dockerfile where you add a user with the UID, solely so there is a user associated with that ID, build your custom image, and possibly have to upload it to a registry.
2. Create a custom build of Jenkins Docker image

docker build --build-arg uid=<uid_compatible_with_host_os> --build-arg gid=<gid_compatible_with_host_os> --build-arg JENKINS_VERSION=<jenkins_version> --build-arg JENKINS_SHA=<jenkins_sha> -t jenkins:<jenkins_version> https://github.com/jenkinsci/docker.git

and then possibly have to upload that to a registry.

Note that both methods require using a custom build of Jenkins Docker.

If the UID and GID was changed to unreserved values, these workarounds would no longer be necessary, and we would be able to use the stock Jenkins Docker images hosted on Docker Hub, right out of the box. Understanding the maintainers have rejected changing the UID and GID multiple times, I'd like to ask them to kindly reconsider this issue once again, with this new knowledge of why running a container with a different UID is insufficient. It is a small change of two values that will make a huge usability improvement.

More information on custom builds of Jenkins Docker (workaround 2):

For those of you who are looking to use workaround option 2 (custom build of Jenkins Docker image), most of the values for the arguments are obvious except for the value for JENKINS_SHA. I have found these values located in the corresponding subdirectory of your desired Jenkins version under https://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/. For example, the SHA1 checksum for Jenkins 2.2 is located at https://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/2.2/jenkins-war-2.2.war.sha1

Below is a complete example of a custom image build for Jenkins 2.2 using UID of 200 and GID of 200 for the jenkins user and group, respectively:

docker build --build-arg uid=200 --build-arg gid=200 --build-arg JENKINS_VERSION=2.2 --build-arg JENKINS_SHA=ac0b046f477620088fc8e9a40c510e484d975d63 -t jenkins:2.2 https://github.com/jenkinsci/docker.git

Edits: Removed aggressive language so as not to offend the Jenkins Docker maintainers.
Further edits: Advised how to find appropriate SHA1 checksum and showed a complete custom build command.
Further further edits: reduced UID and GID example values to within system UID/GID range. (See comment in #112.)

[bbriggs]

Great solutions, @gotgenes!

For the record, I went the first route by publishing my own version of the Jenkins container that used a UID friendly to me. It's no problem for me to consistently create a user without an offending UID because I'm automating the deployment with Puppet, but I also realize that may not be the case for everyone.

In a perfect world, the Jenkins container itself would have a good UID. Until such a time, I have to stick with the solution we discussed earlier.

[carlossg]

I'm closing this issue that is fixed by setting the correct selinux labels
http://www.projectatomic.io/blog/2015/06/using-volumes-with-docker-can-cause-problems-with-selinux/

For any other questions about the default UID see https://github.com/jenkinsci/docker/issues/112#issuecomment-227080794