Docker: question: use of volume makes docker build with this image as base hard work

[edsykes]

Hi,

I'm using this image as a base image for a custom docker build. The fact that a volume is being used means that docker build throws away file system changes to the jenkins user home directory. This makes using docker build to create a new image difficult. For example, I can't edit ~/.bashrc in a docker run command in my own dockerfile because it gets thrown away at the end of the build step.

I'm not sure what the alternative is, as removing the volume will make it harder for those who want to just you the built image and have their data saved between runs of the container. However I want to highlight it for discussion.

[jimzucker]

@edsykes take a look at the example I put on #208

[edsykes]

thanks for that. Useful to see your approach, and part of the approach I have taken is to add files in the same way.

However, one of the nice things about docker is the ability to take what you would run on the command line to setup a machine and drop it into a dockerfile without much fuss. As an example, i want to install nvm on the jenkins box. Normally this is done through a shell script like this:

RUN curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.31.1/install.sh | bash

which puts a folder .nvm in ~/. However, due to the volume, all file changes are discarded between this step and the next.

I think the source of the issue is the fact that the jenkins user's home directory is volume mapped. THis has a wide impact on not only jenkins but other tools. This means any tool installing into ~/ will suffer from having it's file changes discarded when part of a RUN step.

I'll carry on having a dig and a think and see if there is a way to have the volume mapped but do it away from ~/

[jimzucker]

@edsykes the issues is the in the Jenkins docker file there is Volume command:

VOLUME /var/jenkins_home

once this is done you cant modify the volume in subsequent images derived from this file, its all about how the union file system in docker works, i.e. this is a feature not a bug.

The solution I documented in #208 is based on best practices.

[edsykes]

The issue is partly as you mention, it's also that JENKINS_HOME and the jenkins user home directory are conflated, in my opinion.

I've created a pull request that removes that conflation and allows my use case:

https://github.com/jenkinsci/docker/pull/272

I'm going to be working with this version today to ensure that everything works as it should.

ed

[carlossg]

have you tried extending the image with

FROM jenkins
ENV HOME=/home/user
RUN curl...

[edsykes]

@carlossg I haven't tried that, but looking at this line from the Dockerfile I don't think that will change things:

useradd -d "$JENKINS_HOME" -u ${uid} -g ${gid} -m -s /bin/bash ${user}

since this is setting the jenkins user's home directory to the same directory as the jenkins app's operational directory. That's the bit that seems conflated to me.

In my pull request I removed the setting of the home directory:

useradd -u ${uid} -g ${gid} -m -s /bin/bash ${user}

which means you need to do a couple of more things to get the permissions right, see #272 for more details.

[carlossg]

You should try overriding HOME, IIRC tools will use that

[edsykes]

I'll try it now - I'm not sure which tools you mean, tools in general?

[edsykes]

I adapted your suggestion to this:

FROM jenkins

USER root
RUN mkdir /home/jenkins && chown jenkins /home/jenkins

USER jenkins
ENV HOME=/home/jenkins
RUN wget -qO- https://raw.githubusercontent.com/creationix/nvm/v0.31.1/install.sh | bash

however, since the home directory was really created in $JENKINS_HOME there are none of the usual configuration files like .bashrc - this means nvm fails to install with errors like this:

=> Profile not found. Tried (as defined in $PROFILE), ~/.bashrc, ~/.bash_profile, ~/.zshrc, and ~/.profile.

If i were to go and create these files, I would also need to tell the system that the jenkins user's home directory is no longer /var/jenkins_home but /home/jenkins. I could do that.

So the two options here are:

1. change the base image to separate out $JENKINS_HOME and the home directory for the jenkins user.
2. anyone who wants to use this image as a base for further images must rewire the home directory.

Is there a reason that the home directory is $JENKINS_HOME? Does jenkins write some files to ~/ that need to be included in the volume for preservation across building images?

If that is the case, then option 2 would interfere with that preservation too.

[seymen]

The fact that a volume is being used means that docker build throws away file system changes to the jenkins user home directory.

This indeed makes it very hard to work with this docker image.

I came up with another solution using the entrypoint technique that @jimzucker has in #208. It involves copying the job files to a temp folder that jenkins user has rights to (e.g. /usr/share/jenkins/jobs) and copying them back to JENKINS_HOME using jenkins_entrypoint script. It is a bit cumbersome but it works.

This approach removes the need for gosu and also enables you to revert the USER back to jenkins in Dockerfile after performing tasks as root - https://github.com/capitalone/Hygieia/blob/master/test-servers/jenkins/Dockerfile#L53

_Dockerfile_

FROM jenkins
...
# copy jenkins jobs
ADD jenkins/jobs /usr/share/jenkins/jobs

# override the entry point to use a custom script
ADD jenkins/jenkins-entrypoint.sh /usr/local/bin/jenkins-entrypoint.sh

USER root

# make custom entry point script executable
RUN chmod +x /usr/local/bin/jenkins-entrypoint.sh
...

USER jenkins

ENTRYPOINT ["/bin/tini", "--", "/usr/local/bin/jenkins-entrypoint.sh"]

_jenkins_entrypoint.sh_

#!/bin/bash

# we have to do this in an entrypoint becuase the jenkins image has defined $JENKINS_HOME as a volume
cp -R /usr/share/jenkins/jobs /var/jenkins_home/jobs

# call the original entry point
/bin/tini -- /usr/local/bin/jenkins.sh

[carlossg]

The volume is required, otherwise you won't be able to persist Jenkins data when running the container.

[edsykes]

I'm not sure I've really got across what the issue is, to me it's due to the jenkins user and the jenkins app sharing the same directory.

thoughts?

[gclayburg]

I ran into this same issue when trying to extend this Dockerfile. I tried a few different work-arounds, but what I ended up using was the /usr/share/jenkins/ref directory. The jenkins.sh launch script copies files to JENKINS_HOME. This way, I could just extend the Jenkins Dockerfile and still use the same jenkins.sh entrypoint. It isn't perfect since the jenkins.sh script doesn't quite get all the permissions right, but it generally works. The Dockerfile I use is here:

BuildFarm jenkins-data

[ndeloof]

I think it would make sense to split jenkins user home and JENKINS_HOME.
see https://github.com/jenkinsci/docker/pull/272

[realdadfish]

The volume / jenkins-ref approach is actually quite neat, but under-documented and the consequences of JENKINS_HOME being a volume should also be made more clear (is there any way to tell Docker to let writes to a not-mapped volume fail on build time?!)

However, I'm suffering a different issue: I really want to be able to tweak configuration files before Jenkins is starting up and I miss a "callback" for that in jenkins.sh. These tweaks for example would include adding paths / identities / credentials based on the run environment.

Right now for example I'm pre-packaging master.key, hudson.util.Secret in my image and that is not quite what I want (I'd rather dynamically add master.key on run time).

[ndeloof]

@tommyd3mdi what do you mean by "a not-mapped volume" ? Why would you like this make it fail ?

About tweaking, in derived Dockerfile, define your own entreypoint, to do some customization, then invoke the upstream entrypoint script. Other (better?) option is to rely on jenkins init.groovy scripts, which can address most customization needs.

[realdadfish]

@ndeloof I'm a newbie, so excuse me, but I thought saying VOLUME /foo/bar in a Dockerfile instructs Docker to bind whatever host directory that is given to docker run -v ... to /foo/bar at _run time_, so in my understanding anything that is saved their on _build time_ is subject to be unavailable / removed later on, no?

And about initialization, the upstream initialization script (https://github.com/jenkinsci/docker/blob/master/jenkins.sh) does the copying and directly afterwards starts Jenkins, so yes, I can probably copy that script and adapt it to my purposes. I thought though that it might be useful to have some extension point right after all configuration files are in place and _before_ Jenkins is started.

[ndeloof]

@tommyd3mdi VOLUME indeed instruct docker daemon that this folder is mounted from host (doesn't need to be explicitly set to a specific host folder, docker will create a volume).
Docker image can define initial volume content, but for flexibility and to assist people with upgrades, we preferred to rely on the ref mechanism and copy on startup.

For your need, just define another custom script as ENTRYPOINT to do whatever customization you need, then exec jenkins.sh. Or rely on jenkins init scripts like you would on a traditional jenkins deployment.

[realdadfish]

@ndeloof

Docker image can define initial volume content...

Really? Whatever I wrote directly to $JENKINS_HOME at _build time_ was lost at _run time_ and I don't see where the base Jenkins Dockerfile / scripts actually actively _remove_ things from $JENKINS_HOME before it started copying things from /usr/share/jenkins/ref.

So I resorted to create / add everything to /usr/share/jenkins/ref - even though copying bigger files like Android emulator images takes longer for the instance to come up and I'd much rather have seen these things moved than copied.

[ndeloof]

@tommyd3mdi because your customization comes AFTER the VOLUME definition. This is by intention. We rely on ref to customize jenkins_home so we can manage data migration on upgrades.

[Schnitzel]

I had the same issue, but with OpenShift, where you don't know the UserID of the user that will be passed into the container. Usually you give the root group access to the files, as the user will have the group root. But changing the permissions of files within an already defined volume does not work.

Luckily all the scripts are using the JENKINS_HOME environment variable, so you can just define another home and, put stuff in there and define it then as a volume:

This is how my Dockerfile looks like:

FROM jenkins:latest

USER root

# The original jenkins image defines a volume for /var/jenkins_home, which means we cannot change that volume, as every change you do to folder after it is defined as a volume will be discarded. Luckily there is an environment variable which allows to define another folder as the Jenkins Home
RUN mkdir /home/jenkins
ENV JENKINS_HOME=/home/jenkins
ENV COPY_REFERENCE_FILE_LOG $JENKINS_HOME/copy_reference_file.log

# Copying our slaves info
COPY nodes $JENKINS_HOME/nodes

# OpenShift Sync Plugin config which disables the sync
COPY io.fabric8.jenkins.openshiftsync.GlobalPluginConfiguration.xml $JENKINS_HOME/io.fabric8.jenkins.openshiftsync.GlobalPluginConfiguration.xml

# Changing permissions so we can run on openshift
RUN chgrp root $JENKINS_HOME
RUN chmod a+rw -R $JENKINS_HOME

# defining our new folder also as a volume
VOLUME /home/jenkins

[carlossg]

@Schnitzel so how do you keep your data when your container dies or you upgrade the image ?

[Schnitzel]

@carlossg
I still define a Volume at the end of the Dockerfile (see last line), which will create a volume for the container and keep the data around when i restart the container or update the image.

[carlossg]

ok, that didn't use to work in previous versions, everything copied to a volume was not available later on. I've tested it and COPYed files are always there, but files created during RUN are only available if the VOLUME entry is after the RUN, not before

Example

FROM busybox
VOLUME /vol
RUN mkdir -p /vol && touch /vol/test
COPY file /vol/file
CMD ["ls", "-alF", "/vol"]

output:
file

but

FROM busybox
RUN mkdir -p /vol && touch /vol/test
COPY file /vol/file
VOLUME /vol
CMD ["ls", "/vol"]

output:
file
test

Maybe it is a side effect that only works in local volumes

@ndeloof do you know any changes in docker?

[uschtwill]

Thanks @Schnitzel, nice work-around!

[ghost]

I had the same problem with the jenkins/ssh-slave .... I wanted to install RVM and ruby under it (why"? cause rvm handles installing all the ruby dependencies for me and I don't have to add those instructions to my Dockerfile). Ok so in my Dockerfile ....

FROM jenkins/ssh-slave

...

USER jenkins
ARG RUBYVERSION
ENV PATH /usr/local/rvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
RUN gpg2 --keyserver hkp://keys.gnupg.net --recv-keys D39DC0E3 \
&& /bin/bash -l -c "curl -L get.rvm.io | bash -s stable; /bin/bash -l -c \"rvm requirements\"; /bin/bash -l -c \"rvm install $RUBYVERSION\"; /bin/bash -l -c \"gem install bundler --no-ri --no-rdoc\"; mv /home/jenkins/.rvm /home/tmpj/.rvm; cp /home/jenkins/.b* /home/tmpj; cp /home/jenkins/.profile /home/tmpj"

USER root

RUN ls -asl /home/tmpj

RUN ls -aslR /home/tmpj/.rvm

RUN sed -i '2 i cp /home/tmpj/.bash* /home/jenkins/' /usr/local/bin/setup-sshd
RUN sed -i '2 i cp /home/tmpj/.profile /home/jenkins/.profile' /usr/local/bin/setup-sshd
RUN sed -i '2 i mv /home/tmpj/.rvm /home/jenkins/.rvm' /usr/local/bin/setup-sshd
RUN sed -i '2 i chown -R jenkins:jenkins /home/jenkins/.*' /usr/local/bin/setup-sshd

=====

now when I start up my container ... /usr/local/bin/setup-sshd is the command that the ENTRYPOINT ... so it kind of stinks but when the container starts up (everytime) it moves my .rvm stuff into place. A side effect is installing rvm/ruby is always the last thing I do in my Dockerfile ... otherwise I got another management nightmare on my hands. I couldn't get the "global rvm" install to work ...

[carlossg]

jenkins/ssh-slave no longer adds /home/jenkins as a volume https://github.com/jenkinsci/docker-slave/blob/master/Dockerfile#L42