Docker: Error: more than one instance sharing /var/home

[swkarr]

Environment and steps taken

Running Docker for Windows on Windows 10
Jenkins container pulled from an internal mirror of this site.
Startup description

1. Start a new jenkins container instance mapping JENKINS_HOME to a host folder
2. Sleep for 30 seconds to give time to create the JENKINS_HOME folder/files.
3. Copy the UpdateCenter plugin and configuration derived by a file difference comparison. over JENKINS_HOME. This maps to an offline plugin repository.
4. Restart Jenkins to enable the new functionality.
5. Logged into the Jenkins dashboard creating an admin account.
6. No other activity for the day.

Docker commands that you execute (Windows Power Shell)

docker run --name jenkinsCI -d -p 9095:8080 -p 50030:50030 --env JENKINS_SLAVE_AGENT_PORT=50030  -v /d/docker/jenkinsCI:/var/jenkins_home ArtifactoryDocker/jenkins:2.19.4
Start-Sleep -Seconds 30
Copy-Item -Path D:/docker/UpdateCenter/* -Destination D:/docker/jenkinsCI -Force -Recurse
docker restart jenkinsCI

Actual result

Works as expected, however, with no other changes I am seeing the multiple instance error by the next day.

Error

Jenkins detected that you appear to be running more than one instance of Jenkins that share the same home directory '/var/jenkins_home’. This greatly confuses Jenkins and you will likely experience strange behaviors, so please correct the situation.
This Jenkins:   1777109262 contextPath="" at 7@7bc7f2506196
Other Jenkins:  1777109262 contextPath="" at 7@7bc7f2506196

Expected outcome

Do not expect this error, especially since the information for This and Other Jenkins is the same.

Have you tried a non-dockerized Jenkins and get the expected outcome?

Not on this machine, but have directly instantiated multiple instances on a single Linux system user's account using unique "JENKINS_HOME" folders for each instance.

Output of docker version

PS D:\docker> docker --version
Docker version 1.12.2-rc1, build 45bed2c, experimental

Other relevant information

I have seen it with a single Jenkins container running although currently I have three Jenkins containers on this system. However, each is mapped to a different host folder based on the instance name. Using a Power Shell script to enforce this.

First seen using 2.19.1-alpine as is.
Current occurrence is built on 2.19.4 using a Docker file to add Maven and Gradle.

FROM ArtifactoryDocker/jenkins:2.19.4
MAINTAINER Shawn Karr <[email protected]>
ADD apache-maven-3.2.2-bin.tar.gz /etc
COPY settings.xml /etc/apache-maven-3.2.2/conf
ENV MAVEN_HOME=/etc/apache-maven-3.2.2
RUN export MAVEN_HOME
ADD gradle-3.1.tar.gz /etc
ENV GRADLE_HOME=/etc/gradle-3.1
RUN export GRADLE_HOME
ENV PATH=${PATH}:${MAVEN_HOME}/bin:${GRADLE_HOME}/bin
RUN export PATH
ENTRYPOINT ["/bin/tini", "--", "/usr/local/bin/jenkins.sh"]

[ianwhite]

I'm also experiencing this issue, Docker version 17.06.0-ce, build 02c1d87.

[nateridderman]

I also encountered this, with Docker 17.12.0-ce-win47 and the current LTS docker image.

[batmat]

FROM ArtifactoryDocker/jenkins:2.19.4

@ianwhite @nateridderman it looks like the wrong place to report. This repository is the source for jenkins/jenkins, jenkinsci:jenkins and jenkins AFAIK. ArtifactoryDocker/jenkins:2.19.4 is not coming from here.

cc @carlossg @rtyler do you agree? If so I guess this can be closed.

[seetharam757]

Any solution for this? Facing similar issue #720

[batmat]

Care to clarify what you are trying to achieve? If you are running two Jenkins instances on the same JENKINS_HOME, then this is not supported, and not an issue specific to the Jenkins official Docker image. Going to close this, if I misunderstood your intent, please feel free to reopen.

If that is that case, and you wish it to be supported to have many instances sharing the same JENKINS_HOME, please file (or watch) an issue at https://issues.jenkins-ci.org/secure/Dashboard.jspa on the JENKINS project.

Also, if so, you will likely be interested to read https://jenkins.io/blog/2018/08/31/shifting-gears/ which discusses the current plans for improving Jenkins in many areas like these.

Thanks and cheers!

[swkarr]

To clarify:

• ArtifacroryDocker is a corporate proxy of jenkinsci:jenkins
• While the system has a few jenkins images running, they have each been assigned their own unique JENKINS_HOME using docker -v.

[batmat]

@swkarr reopening to at least better understand. I think something is simply fishy with your setup.

FYI this double instance checking is simply done through a hidden file located under $JENKINS_HOME/.owner. So, if you see this warning this can mean two things:

• either you are actually running two Jenkins instance, and they will detect each other and issue this warning
• or you've copied a pre-existing $JENKINS_HOME from a running instance and you're using it for a new one (though as I understand the code at https://github.com/jenkinsci/jenkins/blob/5df63bc63ba352ac24da59729407f3439aa6a26d/core/src/main/java/hudson/util/DoubleLaunchChecker.java#L70 that should still mean many instances are updating this file regularly).

Also, what is that D:/docker/UpdateCenter/* directory? What is this supposed to contain?

[swkarr]

The JENKINS_HOME folder was created for the instance the first time the image was run.

D:/docker/UpdateCenter contains the following as identified by a diff of a base JENKINS_HOME and one manually configured for update-sites-manager:

• plugins/update-sites-manager.jpi
• hudson.model.UpdateCenter.xml configuration file with information needed to connect to a JUSEPPE instance.

This is due to corporate policy preventing direct internet downloads breaking Jenkins update-center.json capability within our environment and I was unable to find an easy way to redirect to an Artifactory proxy for plugin downloads.

Error showed up only after I logged into the instance's dashboard and then came back to the dashboard, that had not been closed, the next day. The error did not show immediately on dashboard login and a simple refresh when it is noticed removes the error message.

Last version tested was 2.60.3 of jenkins:lts-alpine. Will try to reproduce with the latest container version.

[swkarr]

Tested 5 instances of 3 versions including the base jenkins:jenkins:lts-alpine. The only configuration for lts-alpine version 2.121.3 was the inclusion and configuration of the update-sites-manager plugin.

All instances: opened the dashboard in Firefox. Without closing the dashboard tabs, left Firefox running. Then the next day, performed an F5 refresh for each Jenkins dashboard tab. Result was the indicated error.

Error
Jenkins detected that you appear to be running more than one instance of Jenkins that share the same home directory '/var/jenkins_home'. This greatly confuses Jenkins and you will likely experience stange behaviors, so please correct the situation.
This Jenkins: 560070034 contextPath="" at 7@53e4f6966b7e
Other Jenkins: 560070034 contextPath="" at 7@53e4f6966b7e
[Ignore this problem and keep using Jenkins anyway]

Each error page has a unique ID and "at" address but it appears that each instance is colliding with itself.

Further F5 refreshes does not clear the error page.

Clicking [Ignore] button returns to login page.
Able to successfully log into the instance.

[Coral-Liu]

I also met same issue as swkarr. Any solution to fix it?

[dellgreen]

I'm seeing this issue just running jenkins normally, once in a while it throws this error when the first person in the morning navigates to the jenkisn website after it has been idle all night. There are defintely not 2 instances running on my server

maybe lastmodifiedDate of .owner file should be relooked at or at a minimum a system property to disable this check in some way as this error blocks other automated systems talking to jenkins when it happens.

[ianfixes]

I'm noticing this problem in some Jenkins tabs that have been open for a few hours:

This Jenkins: | 1864319447 contextPath="" at 11@33d99e0b230e
-- | --
Other Jenkins: | 548107553 contextPath="" at 7@6d791b76b116

Strangely, the bottom of the page where this error occurs says Jenkins ver. 2.150.2. But when I refresh, I get the version from the docker image that I'm running: Jenkins ver. 2.190.2.

Am I commenting on the correct GitHub issue for this?

If so, how would I track down the rogue 2.150.2 process? And why does it start periodically but vanish immediately afterward?

[mjeco]

I'm also noticing this issue. Running Jenkins v2.222.1 in Kubernetes. Only one instance of Jenkins is ever running at the same time. What would cause the DoubleLaucherChecker to collide with itself? To me, this suggests that something outside of Jenkins modified the .owner file?
JENKINS_HOME is on an Azure Fileshare (CIFS) and mounted into the Pod as a Volume.

System property to disable this check would be useful.

[dellgreen]

As it just happened to you, are you able to confirm the timestamps are different in the log files with respects to the the logging statement in: https://github.com/jenkinsci/jenkins/blob/5df63bc63ba352ac24da59729407f3439aa6a26d/core/src/main/java/hudson/util/DoubleLaunchChecker.java#L110

[mjeco]

@dellgreen

Apr 10, 2020 1:23:46 AM hudson.util.DoubleLaunchChecker execute
SEVERE: Collision detected. timestamp=1586477926000, expected=1586477925000

[ianfixes]

As a follow-up to my previous comment, I can't say whether this happens all the time but on 2 different occasions I've received this error right around the stroke of midnight when I happened to have Jenkins open in a browser.

[mjeco]

Happens to my instance every day between 7PM and 8PM UTC time.

Apr 11, 2020 7:47:57 PM hudson.util.DoubleLaunchChecker execute
SEVERE: Collision detected. timestamp=1586629317000, expected=1586629316000

Apr 12, 2020 7:15:58 PM hudson.util.DoubleLaunchChecker execute
SEVERE: Collision detected. timestamp=1586714338000, expected=1586714337000

Apr 13, 2020 7:07:59 PM hudson.util.DoubleLaunchChecker execute
SEVERE: Collision detected. timestamp=1586799539000, expected=1586799538000

I can assure that the only process running in the pod is Jenkins.

PID   USER     TIME  COMMAND
    1 jenkins   0:10 /sbin/tini -- /usr/local/bin/jenkins.sh
    7 jenkins  10:49 java -Duser.home=/var/jenkins_home -Xmx4096m -Djava.util.l
 8017 jenkins   0:00 ps aux

[mjeco]

Ok, I think I have figured out what was happening in my case. If you look at the collision logs, the two timestamps always differ by around 1 second. My JENKINS_HOME is set up on a CIFS mount in Azure Fileshare. When Jenkins updates the .owner file it immediately records its last-modified timestamp. But I believe there is a small (1 second) delay until the CIFS mount records the actual file modification time. So the timestamp Jenkins has stored and the actual timestamp of the file is not the same, hence the error message.

To get around this issue you can symlink the .owner file to /dev/null (or any other file not in JENKINS_HOME). The symlink has its own last-modified timestamp which does not update when Jenkins writes to the file. This basically makes Jenkins check the same timestamp over and over again. Preventing the error from manifesting.

rm -f $JENKINS_HOME/.owner
ln -s /dev/null $JENKINS_HOME/.owner

[ianfixes]

Does this imply that Jenkins could resolve this by reading from the file immediately after writing it in order to determine the proper last-modified time? Or does that defeat the purpose?

[mjeco]

I think there should be some delay between writing to the file and reading the timestamp to cater for delays associated with NFS/CIFS mounts. If you don't care about this check just set the symlink as I mentioned above. Fixed the issue for me.