Docker: Pulled image jenkins/jenkins doesn't run on arm64 architecture

[shawnguo2]

Hi,

I can pull jenkins/jenkins successfully on arm64 architecture, but it fails to run as below.

$ uname -m
aarch64
$ docker pull jenkins/jenkins
Using default tag: latest
latest: Pulling from jenkins/jenkins
Digest: sha256:83d149bc799c1d50d146aecc75c558754594d3a2b69d0a81c4eb897ae27a58d0
Status: Image is up to date for jenkins/jenkins:latest
$ docker run --rm -p 8080:8080 -p 50000:50000 jenkins/jenkins
standard_init_linux.go:190: exec user process caused "exec format error"
$

[carlossg]

there is no docker image build for arm

[shawnguo2]

@carlossg thanks for confirming this. As the pull works on arm64 machine, I thought the arm64 build of jenkins/jenkins is available. I just did an arm64 build of jenkins/jenkins image and confirmed it works on arm64 architecture. I really want to get arm64 architecture supported by jenkins docker image. Is there anything I can do to help it happen? Thanks.

[carlossg]

ping @rtyler

[batmat]

@shawnguo2 I think the right way forward is to file an INFRA issue on https://issues.jenkins-ci.org/projects/INFRA/issues/INFRA. Cheers

[shawnguo2]

@batmat Thanks a lot for the hint. I just created an INFRA issue as below. Please let me know if I need to add more information into the issue. Thanks.

https://issues.jenkins-ci.org/projects/INFRA/issues/INFRA-1652

Update - the issue link is changed to:

https://issues.jenkins-ci.org/browse/JENKINS-52090

Thanks @lag-linaro for spotting it.

[oleg-nenashev]

So far is is a feature request for the scripts in this repository.
We build Docker images on Jenkins, they just get deployed to Dockerhub.
Once the build scripts are ready in this repo are ready, we can get the image. Is it right?
Or are there special infra requirements to build ARM images?

[carlossg]

I think you need to do the build in ARM, but not sure

[logi85]

hi, I also wanted to try jenkins on aarch64 ( rock64 in my case ).
So I changed the Dockerfile and created a build-script (build).

Steps to use:

1. clone https://github.com/logi85/docker/
2. execute ./build
3. use docker-aarch64:latest instead of jenkins/jenkins:lts to launch

For the jenkinsci-developers:
Please have a look on
https://github.com/logi85/docker/commit/e5ee848b0698fc8435f3db074f980e8c33d6958f .
I just changed the base image to aarch-jdk.
Perhaps you could add a aarch64-branch or a seperate repository, however with this change you can build an aarch64-image.

It is working for me, just finished setup and now I see "Willkommen bei Jenkins!" ;-)

[lag-linaro]

Oddly the link to the Jenkins Issue for this provided by Shawn isn't working.

For reference, it's here: https://issues.jenkins-ci.org/browse/JENKINS-52090

[shawnguo2]

@lag-linaro Yes, the issue was moved by @oleg-nenashev and I forgot to update the page here.

[lag-linaro]

Why isn't this project part of Docker's Official Images repo?

This issue would just go away if it was, since Docker's builders create and publish images for all architectures tagged as supported, and since Jenkins is based on OpenJDK it should 'just work'.

[carlossg]

because the official images repo is manually updated and this is currently fully automated for each release

[lag-linaro]

Okay, understood.

So the only way to move this forward is for you guys to fully automate ARM64 builds too, right?

Is there anything we can do to expedite that process?

[khasim]

Hi - any update on ARM64 builds ?

[lag-linaro]

So an INFRA was raised over 2 months ago now, but hasn't gained any attention.

Is there anything we can do to expedite support for Aarch64?

This project literally 'just works'.

[carlossg]

Unless someone can come up with a way to build without arm server it requires infrastructure that is not yet available

[oleg-nenashev]

@lag-linaro Would Linaro be interested to contribute/sponsor some infrastructure for that? Or maybe build the image on ci.linaro.org? If yes, we could find a way to host it IMHO.

If it is something you may consider, I propose to start a thread in the Jenkins Platform SIG we have a meeting at 2PM UTC today, and I am happy to sync-up on this topic if you want.

P.S: Soon I also plan to create an Embedded SIG. We had a discussion about something like that with Linaro folks at the last FOSDEM, so maybe we could use such venues to coordinate the activities

[oleg-nenashev]

As proposed in the Ticket, we could start from offering the build from the Jenkins Experimental repo: https://hub.docker.com/r/jenkins/jenkins-experimental/ or equivalent. This is built on DockerHub, and it will have much less quality requirements than building official Docker images in Trusted CI

[lag-linaro]

@oleg-nenashev thanks for taking this on, I really appreciate it.

With regards to infrastructure, it would be preferred if you were to use a publicly available service such as CodeFresh, WorksOnARM or Shippable.

CodeFresh Arm Beta signup: https://goo.gl/forms/aDhlk56jZcblYokj1

http://docs.shippable.com/platform/tutorial/workflow/run-ci-builds-on-arm/

If you need raw infrastructure for some reason the WorksOnArm project will supply full servers if you want to deal with metal: https://github.com/worksonarm/cluster/

I can personally vouch for CodeFresh, as I've set some builds up there recently. The platform is very easy to use and initial set-up is trivial, especially if you have a Dockerfile.

[oleg-nenashev]

We have a related discussion in Platform SIG about multi-arch images: https://groups.google.com/forum/#!topic/jenkins-platform-sig/6-X_8J0Ii8s . I propose to have a call next week and to discuss this multi-architecture Docker images (see the timeframe voting link).

Personally I find WorksOnARM really interesting from the Jenkins project PoV. We have recently reached Quality Outreach program, and I would be interested to join such program for Arm platforms.

[oleg-nenashev]

https://github.com/jenkinsci/docker/pull/719 actually addresses it, right?

[khasim]

@oleg-nenashev, yes the two commits seem to address it.

[carlossg]

related to #646

[vielmetti]

Happy to help with this, and as well bringing this to the attention of @tianon regarding official Docker image support.

[lag-linaro]

Good news. I am able to build this for ARM64 with no issues.

I have taken the liberty to push the build into CodeFresh CI.

Anyone can pull the resultant build from any ARM64 platform using:

docker pull laglinaro/jenkinsdocker:master

Please could you add arm64v8 as a supported platform, so that the following will 'just work':

docker pull jenkins/jenkins

[oleg-nenashev]

@lag-linaro There is pending https://github.com/jenkinsci/docker/pull/719 which enables building platforms with QEMU. I believe we could add the Arm flow to it as well. WDYT?

[vielmetti]

@oleg-nenashev -- adding arm64 to #719 sounds like it would be a win.

[lag-linaro]

@oleg-nenashev go for it. Works for me.

[lag-linaro]

@oleg-nenashev could I trouble you for an update please?

[oleg-nenashev]

@lag-linaro Please see https://docs.google.com/document/d/1FARi55vDjsdzi6Nj9ZB9e1wh2dU8nyWK6mq_cge0ceg/edit#heading=h.948e191k7cc6

[lag-linaro]

@oleg-nenashev thanks for the information.

So the idea is that these images will be pushed to https://hub.docker.com/r/jenkins4eval/jenkins-multiarch/?

Any idea when that will happen?

And how long will they live there until they are rotated into the master branch?

[jmreicha]

@lag-linaro yes images will be pushed to the repo you linked. There is a PR https://github.com/jenkinsci/docker/pull/750 to turn on auto builds, once it is merged the multi-arch builds should start showing up there.

Once multi-arch has been turned on, the new workflow will need to be tested and vetted for bugs and issues, with the hope that eventually it will be the process for all builds but that may take a while.

[lag-linaro]

Do you know when you still start building !x86_64 images natively?

There shouldn't be any need for separate software infrastructure for this. I would have thought that most (all?) architectures will build just fine with the default Dockerfile you use for x86_64.

[darcyrush]

It looks like #719 and #750 recently got merged. I'm looking in jenkins4eval/jenkins-multiarch and jenkins4eval/jenkins-multiarch-qemu but not seeing any tags for either. jenkins/jenkins-experimental has some builds but they are for other projects and over 4 months old.

The hacking docs says builds should be pushed to jenkins4eval/jenkins/ but this comment implies credentials may still be needed for the images to be pushed.

Is this the case or am I misunderstanding the purpose of these PRs? I want to pull a jenkins image to run on my Raspberry Pi.

[oleg-nenashev]

It is still in progress, we need to figure out some issues with the
deployment flow

On Sat, Mar 9, 2019 at 5:42 PM darcyrush [e-mail] wrote:

It looks like #719 https://github.com/jenkinsci/docker/pull/719 and #750
https://github.com/jenkinsci/docker/pull/750 recently got merged. I'm
looking in jenkins4eval/jenkins-multiarch and
jenkins4eval/jenkins-multiarch-qemu but not seeing any tags for either.
jenkins/jenkins-experimental has some builds but they are for other
projects and over 4 months old.

The hacking docs
https://github.com/jmreicha/docker/blob/8fac4d781d96e5991943120a977132471eda81e7/HACKING.adoc#multiarch-support
says builds should be pushed to jenkins4eval/jenkins/
https://hub.docker.com/r/jenkins4eval/jenkins/ but this comment
https://github.com/jenkinsci/docker/pull/750#issuecomment-468999876
implies credentials may still be needed for the images to be pushed.

Is this the case or am I misunderstanding the purpose of these PRs? I want
to pull a jenkins image to run on my Raspberry Pi.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jenkinsci/docker/issues/686#issuecomment-471199008,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC3IoPC5XE13LTFlU-9OpgGl_DLsz2Jaks5vU-RxgaJpZM4UcTx0
.

[lag-linaro]

Is this likely to be delivered any time soon?

[slide]

The permissions are fixed now, so the publishing will hopefully start working. There are some other issues with some agents that are causing build failures, but that is hopefully being addressed. You can monitor https://hub.docker.com/r/jenkins4eval/jenkins for builds for different architectures.

[peiying16]

@slide
I enter to the container by docker exec -it jenkins bash
Then I run apt update. It shows permission denied,
What is the password? I want to install npm in this container.

[batmat]

Hello,
Please use a dedicated subject and the mailing list for such questions:
https://jenkins.io/mailing-lists

Updating the container this way is unsupported and I am not sure there is
even password set (cannot check easily, bad connection here now).

Either create a custom image, or better run your workload in an agent. On
the mailing list there will be more people watching (hundreds) and able to
help. Here we discuss and fix bugs related to the Jenkins docker image.

Thanks!

Le dim. 4 août 2019 à 07:27, Peiying Yang(Brian) [e-mail]
a écrit :

@slide https://github.com/slide
I enter to the container by docker exec -it jenkins bash
Then I run apt update. It shows permission denied,
What is the password? I want to install npm in this container.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jenkinsci/docker/issues/686?email_source=notifications&email_token=AABWU3MHMJPSBOKUUUIPNPLQCZSDPA5CNFSM4FDRHR2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3P2YRA#issuecomment-517975108,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABWU3KOZZFKNGCG2CAJS63QCZSDPANCNFSM4FDRHR2A
.

[slide]

The Jenkins user that is created does not have permissions to install sw. You would need to create your own custom image based on this one. As batmat says, you should ask questions on the mailing lists.

[bmuschko]

This should be possible now with Buildx: https://engineering.docker.com/2019/04/multi-arch-images/. Buildx is still an experimental feature and needs to be enabled manually but I think it's the way to go.

[slide]

@bmuschko we are already building cross platform images under the jenkins4eval org on docker hub.

[lag-linaro]

The images on jenkins4eval looks good. Tested on my AArch64 platform.

How long will the image sit there before it is available in the main repo?

[jmreicha]

I made https://github.com/jenkinsci/docker/issues/872 to track the progress of combining the builds.

[brezerk]

the latest alpine images from jenkins4eval are not so good (at least for me).

2020-03-08 00:08:39.863+0000 [id=29]    INFO    o.s.c.s.AbstractApplicationContext#prepareRefresh: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@39057a8f: display name [Root WebApplicationContext]; startup date [Sun Mar 08 00:08:39 GMT 2020]; root of context hierarchy
2020-03-08 00:08:39.864+0000 [id=29]    INFO    o.s.c.s.AbstractApplicationContext#obtainFreshBeanFactory: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@39057a8f]: org.springframework.beans.factory.support.DefaultListableBeanFactory@8fe8502
2020-03-08 00:08:39.868+0000 [id=29]    INFO    o.s.b.f.s.DefaultListableBeanFactory#preInstantiateSingletons: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@8fe8502: defining beans [filter,legacy]; root of factory hierarchy
2020-03-08 00:08:40.477+0000 [id=29]    INFO    jenkins.InitReactorRunner$1#onAttained: Completed initialization
2020-03-08 00:08:41.423+0000 [id=19]    INFO    hudson.WebAppMain$3#run: Jenkins is fully up and running
--> setting agent port for jnlp
--> setting agent port for jnlp... done
2020-03-08 00:09:00.625+0000 [id=62]    INFO    h.TcpSlaveAgentListener$ConnectionHandler#run: Connection #1 failed: java.io.EOFException
2020-03-08 00:09:01.033+0000 [id=63]    INFO    h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #2 from /10.244.3.28:49292
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (assembler_aarch64.hpp:1156), pid=6, tid=0x0000007fa9abbaa0
#  guarantee(Rs != Rn && Rs != Rt) failed: unpredictable instruction
#
# JRE version: OpenJDK Runtime Environment (8.0_212-b04) (build 1.8.0_212-b04)
# Java VM: OpenJDK 64-Bit Server VM (25.212-b04 mixed mode linux-aarch64 compressed oops)
# Derivative: IcedTea 3.12.0
# Distribution: Custom build (Sat May  4 17:44:26 UTC 2019)
# Core dump written. Default location: //core or core.6
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid6.log
#
# Compiler replay data is saved as:
# /tmp/replay_pid6.log
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
#   https://icedtea.classpath.org/bugzilla
#

I build jenkins myself using openjdk:8-jdk-alpine as a base and seems like it has the same issue.

~2 month old images based on slim works fine tho. i guess there is something wrong with alpine and\or jdk runtime...

any one else observing the same behavior?

[slide]

I think you might be one of the first people actually using the image. They are alpha level images as we don't have much in the way of testing on the newer platforms. Can you give more details on the processor you are using?

[brezerk]

hi @slide

I am using Rockchip RK3328 Quad-Core SOC. Can reproduce the issue on all 5 boards i have.
I have been using jenkins4eval/jenkins images for some time before with no issues.
looks like only new images affected. I can try to find out last working image if it would be helpful.

jfyi: I just build jenkins-2.204.5 with arm64v8/openjdk:11-jre-slim base image and looks okay

Thanks!

[slide]

Yes, that would be helpful

[slide]

I have a 6 core Rockchip RK3399 SOC with Mali T860 MP4 GPU that I'll try on.

[brezerk]

it looks like the last working image is: jenkins4eval/jenkins:2.209-alpine-arm64 the others (2.210+) are failing with pretty the same error (jenkins4eval/jenkins-2.215-alpine-arm64 as an example):

Running from: /usr/share/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
2020-03-08 18:00:30.047+0000 [id=1]     INFO    org.eclipse.jetty.util.log.Log#initialized: Logging initialized @3180ms to org.eclipse.jetty.util.log.JavaUtilLog
2020-03-08 18:00:31.048+0000 [id=1]     INFO    winstone.Logger#logInternal: Beginning extraction from war file
2020-03-08 18:00:31.277+0000 [id=1]     WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
2020-03-08 18:00:31.716+0000 [id=1]     INFO    org.eclipse.jetty.server.Server#doStart: jetty-9.4.22.v20191022; built: 2019-10-22T13:37:13.455Z; git: b1e6b55512e008f7fbdf1cbea4ff8a6446d1073b; jvm 1.8.0_212-b04
2020-03-08 18:00:34.202+0000 [id=1]     INFO    org.eclipse.jetty.util.TypeUtil#<clinit>: JVM Runtime does not support Modules
2020-03-08 18:00:34.401+0000 [id=1]     INFO    o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
2020-03-08 18:00:34.761+0000 [id=1]     INFO    o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0
2020-03-08 18:00:34.762+0000 [id=1]     INFO    o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaults
2020-03-08 18:00:34.784+0000 [id=1]     INFO    o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 600000ms
2020-03-08 18:00:38.207+0000 [id=1]     INFO    hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
2020-03-08 18:00:39.272+0000 [id=1]     INFO    o.e.j.s.handler.ContextHandler#doStart: Started w.@19b89d4{Jenkins v2.215,/,file:///var/jenkins_home/war/,AVAILABLE}{/var/jenkins_home/war}
2020-03-08 18:00:39.416+0000 [id=1]     INFO    o.e.j.server.AbstractConnector#doStart: Started ServerConnector@1324409e{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2020-03-08 18:00:39.418+0000 [id=1]     INFO    org.eclipse.jetty.server.Server#doStart: Started @12555ms
2020-03-08 18:00:39.423+0000 [id=20]    INFO    winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
2020-03-08 18:00:51.239+0000 [id=29]    INFO    jenkins.InitReactorRunner$1#onAttained: Started initialization
2020-03-08 18:00:57.758+0000 [id=29]    INFO    jenkins.InitReactorRunner$1#onAttained: Listed all plugins
2020-03-08 18:01:44.561+0000 [id=28]    INFO    jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
2020-03-08 18:01:45.117+0000 [id=29]    INFO    jenkins.InitReactorRunner$1#onAttained: Started all plugins
2020-03-08 18:01:55.266+0000 [id=29]    INFO    h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate optional component hudson.plugins.build_timeout.operations.AbortAndRestartOperation$DescriptorImpl; skipping
2020-03-08 18:01:58.578+0000 [id=28]    INFO    jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
2020-03-08 18:01:59.170+0000 [id=29]    INFO    jenkins.model.Jenkins#setBuildsAndWorkspacesDir: Using non default workspaces directories: ${JENKINS_HOME}/workspace/${ITEM_FULLNAME}.
2020-03-08 18:02:00.680+0000 [id=28]    INFO    jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
2020-03-08 18:02:00.777+0000 [id=42]    INFO    hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started Download metadata
2020-03-08 18:02:00.854+0000 [id=29]    INFO    j.util.groovy.GroovyHookScript#execute: Executing /var/jenkins_home/init.groovy.d/tcp-slave-agent-port.groovy
2020-03-08 18:02:00.929+0000 [id=42]    INFO    hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Download metadata. 137 ms
2020-03-08 18:02:06.119+0000 [id=28]    INFO    o.s.c.s.AbstractApplicationContext#prepareRefresh: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@43f9fc66: display name [Root WebApplicationContext]; startup date [Sun Mar 08 18:02:06 GMT 2020]; root of context hierarchy
2020-03-08 18:02:06.121+0000 [id=28]    INFO    o.s.c.s.AbstractApplicationContext#obtainFreshBeanFactory: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@43f9fc66]: org.springframework.beans.factory.support.DefaultListableBeanFactory@1db98605
2020-03-08 18:02:06.211+0000 [id=28]    INFO    o.s.b.f.s.DefaultListableBeanFactory#preInstantiateSingletons: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1db98605: defining beans [ldapUserSearch,authenticationManager,bindAuthenticator,authoritiesPopulator,initialDirContextFactory]; root of factory hierarchy
2020-03-08 18:02:06.370+0000 [id=28]    INFO    o.a.l.DefaultInitialDirContextFactory#setProviderUrl:  URL 'ldaps://ldap.brezblock.org.ua/dc=brezblock,dc=org,dc=ua', root DN is 'dc=brezblock,dc=org,dc=ua'
2020-03-08 18:02:08.233+0000 [id=28]    INFO    o.s.c.s.AbstractApplicationContext#prepareRefresh: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@1bb9f9be: display name [Root WebApplicationContext]; startup date [Sun Mar 08 18:02:08 GMT 2020]; root of context hierarchy
2020-03-08 18:02:08.234+0000 [id=28]    INFO    o.s.c.s.AbstractApplicationContext#obtainFreshBeanFactory: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@1bb9f9be]: org.springframework.beans.factory.support.DefaultListableBeanFactory@250884ab
2020-03-08 18:02:08.244+0000 [id=28]    INFO    o.s.b.f.s.DefaultListableBeanFactory#preInstantiateSingletons: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@250884ab: defining beans [filter,legacy]; root of factory hierarchy
2020-03-08 18:02:09.134+0000 [id=28]    INFO    jenkins.InitReactorRunner$1#onAttained: Completed initialization
2020-03-08 18:02:10.009+0000 [id=19]    INFO    hudson.WebAppMain$3#run: Jenkins is fully up and running
--> setting agent port for jnlp
--> setting agent port for jnlp... done
--> setting agent port for jnlp
--> setting agent port for jnlp... done
2020-03-08 18:02:30.343+0000 [id=67]    INFO    h.TcpSlaveAgentListener$ConnectionHandler#run: Connection #1 failed: java.io.EOFException
2020-03-08 18:02:30.734+0000 [id=68]    INFO    h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #2 from /10.244.3.28:47302
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (assembler_aarch64.hpp:1156), pid=6, tid=0x0000007f9c329aa0
#  guarantee(Rs != Rn && Rs != Rt) failed: unpredictable instruction
#
# JRE version: OpenJDK Runtime Environment (8.0_212-b04) (build 1.8.0_212-b04)
# Java VM: OpenJDK 64-Bit Server VM (25.212-b04 mixed mode linux-aarch64 compressed oops)
# Derivative: IcedTea 3.12.0
# Distribution: Custom build (Sat May  4 17:44:26 UTC 2019)
# Core dump written. Default location: //core or core.6
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid6.log   
#
# Compiler replay data is saved as:
# /tmp/replay_pid6.log   
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
#   https://icedtea.classpath.org/bugzilla
#

note: it stating up fine, but fails randomly (i had ~200 restarts over the night). navigating: login -> manage -> configure reproducedes this issues for me constantly on a random step too;

[slide]

I tried a few images on my arm64 and they all worked fine for me. What OS are you running?

[slide]

Does it only happen when an agent connects?

[brezerk]

hi @slide
interesting. thanks for trying.

OS

[ marauder ] brezerk@pts/0:3  ~ $
 03/09/20 00:29:25 EET > lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.4 LTS
Release:        18.04
Codename:       bionic

One note, I am using k8s v1.17.3 and containerd://1.2.10 runtime (not a docker).

It does not seems to be happen on the agent connection, but during workload being running.
I do have couple plugins installed (nothing extra ordinary tho): jenkins-pipeline and ldap;

Let me see if I would be able to reproduce it on a clean setup.

[brezerk]

no luck.
clean jenkins4eval/jenkins-2.215-alpine-arm64 deploy into test namespace. went through base setup -> install suggested plugins -> An error occurred during installation: Bad Gateway :(

2020-03-08 22:48:46.358+0000 [id=77]    INFO    h.model.UpdateCenter$DownloadJob#run: Starting the installation of mailer on behalf of admin
2020-03-08 22:48:46.359+0000 [id=77]    INFO    h.m.UpdateCenter$InstallationJob#_run: Skipping duplicate install of: [email protected]
2020-03-08 22:48:46.359+0000 [id=77]    INFO    h.model.UpdateCenter$DownloadJob#run: Installation successful: mailer
2020-03-08 22:48:46.360+0000 [id=77]    INFO    h.m.UpdateCenter$CompleteBatchJob#run: Completing installing of plugin batch…
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (assembler_aarch64.hpp:1156), pid=8, tid=0x0000007fa9d3faa0
#  guarantee(Rs != Rn && Rs != Rt) failed: unpredictable instruction
#
# JRE version: OpenJDK Runtime Environment (8.0_212-b04) (build 1.8.0_212-b04)
# Java VM: OpenJDK 64-Bit Server VM (25.212-b04 mixed mode linux-aarch64 compressed oops)
# Derivative: IcedTea 3.12.0
# Distribution: Custom build (Sat May  4 17:44:26 UTC 2019)
# Core dump written. Default location: //core or core.8
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid8.log   
#
# Compiler replay data is saved as:
# /tmp/replay_pid8.log   
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
#   https://icedtea.classpath.org/bugzilla
#

full log: https://pastebin.com/Yape7aM2

there is a similar report for the make java error for arm64: https://github.com/openhab/openhab-docker/issues/233

Seemed to have resolved the issue by rolling back openjdk8 to the previous working version.

Looks like this issue is more related to openjdk/alpine rather than jenkins:

1. jenkins4eval/slim works fine (btw: why slim based images are quite outdated on jenkins4eval repo? :))
2. custom docker image based on arm64v8/openjdk:11-jre-slim works fine too

[karlhendrik]

Tried building official Dockerfile with docker buildx to arm64 architecture - didn't work. However this guy has Dockerfile (https://github.com/djdefi/rpi-jenkins) that builds Jenkins on top of official Rasbian Stretch image.

I built it and works fine on my RPi4.
https://hub.docker.com/repository/docker/karlhendrik/jenkins-arm

[jroose]

This image was starting fine for me a few months ago, but I noticed that it would crash every few minutes and the image would reboot. I'll try to get logs for you guys. Would you prefer me to start a new ticket to track the frequent crashes? I've got a setup using Kubernetes on Raspberry PI 4.

alaric@c4a-1:~/pods/jenkins$ docker --version
Docker version 19.03.2, build 6a30dfca03

IIRC: no builds were taking place at the time. It was sitting idle and still crashing. Previous images had the same issue.

I'm now waiting for it to reproduce with the latest image: jenkins4eval/jenkins:2.225-alpine-arm64

It crashed after about 20 minutes, with no agents connecting.

[jroose]

This fatal error occurred after more than 10 minutes of runtime.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (assembler_aarch64.hpp:1156), pid=6, tid=0x0000ffff900bdaa0
#  guarantee(Rs != Rn && Rs != Rt) failed: unpredictable instruction
#
# JRE version: OpenJDK Runtime Environment (8.0_212-b04) (build 1.8.0_212-b04)
# Java VM: OpenJDK 64-Bit Server VM (25.212-b04 mixed mode linux-aarch64 compressed oops)
# Derivative: IcedTea 3.12.0
# Distribution: Custom build (Sat May  4 17:44:26 UTC 2019)
# Core dump written. Default location: //core or core.6
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid6.log
#
# Compiler replay data is saved as:
# /tmp/replay_pid6.log
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
#   https://icedtea.classpath.org/bugzilla
#
Session ended, resume using 'kubectl attach jenkins-master-68df6766-zgtbq -c jenkins-master -i -t' command when the pod is running

[slide]

I noticed issues with JDK 8 on ARM64, I switched to JDK 11 and it worked better. Please do file a new issue, though it looks like an issue with the JDK on ARM64.