Configuration-as-code-plugin: Jenkins not starting after upgrading to 2.266

[apwrv]

Your checklist for this issue

🚨 Please review the guidelines for contributing to this repository.

• [x] Jenkins version
  2.266

• [x] Plugin version
  n/a

• [x] OS
  Oracle Linux 7 with Kernel 5.4.17-2036.100.6.1.el7uek.x86_64

Description

I upgraded Jenkins from v2.240 to v2.266 and since then Jenkins isn't starting.

There is 1 warning and 2 errors:

Warning

2020-11-10 21:26:01.632+0000 [id=30] WARNING jenkins.model.Jenkins$5#runTask: Loading global config failed perhaps due to plugin dependency issues java.lang.NoSuchMethodError: com.thoughtworks.xstream.io.HierarchicalStreamReader.peekNextChild()Ljava/lang/String; at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy$ConverterImpl.unmarshal(RoleBasedAuthorizationStrategy.java:579) at hudson.util.XStream2$AssociatedConverterImpl.unmarshal(XStream2.java:463) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:411) at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:333) at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:272) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50) at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1404) at hudson.util.XStream2.unmarshal(XStream2.java:161) at hudson.util.XStream2.unmarshal(XStream2.java:132) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1383) at hudson.XmlFile.unmarshal(XmlFile.java:180)
Caused: java.io.IOException: Unable to read /var/lib/jenkins/config.xml at hudson.XmlFile.unmarshal(XmlFile.java:183) at hudson.XmlFile.unmarshal(XmlFile.java:163) at jenkins.model.Jenkins.loadConfig(Jenkins.java:3143) at jenkins.model.Jenkins.access$1200(Jenkins.java:315) at jenkins.model.Jenkins$13.run(Jenkins.java:3244) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1128) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

Error 1

2020-11-10 21:26:02.046+0000 [id=28] SEVERE jenkins.model.Jenkins#save: An attempt to save Jenkins'' global configuration before it has been loaded has been made during milestone Configuration for all jobs updated. This is indicative of a bug in the caller and may lead to full or partial loss of configuration. java.lang.IllegalStateException: call trace at jenkins.model.Jenkins.save(Jenkins.java:3356) at jenkins.model.Jenkins.saveQuietly(Jenkins.java:3384) at jenkins.model.Jenkins.setSecurityRealm(Jenkins.java:2636) at jenkins.model.Jenkins$16.run(Jenkins.java:3328) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1128) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

Error 2

2020-11-10 21:26:02.052+0000 [id=28] SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed Finalizing set up java.lang.IllegalStateException: An attempt to save the global configuration was made before it was loaded at jenkins.model.Jenkins.save(Jenkins.java:3365) at jenkins.model.Jenkins.saveQuietly(Jenkins.java:3384) at jenkins.model.Jenkins.setSecurityRealm(Jenkins.java:2636) at jenkins.model.Jenkins$16.run(Jenkins.java:3328) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1128) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Caused: org.jvnet.hudson.reactor.ReactorException at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282) at jenkins.InitReactorRunner.run(InitReactorRunner.java:50) at jenkins.model.Jenkins.executeReactor(Jenkins.java:1161) at jenkins.model.Jenkins.<init>(Jenkins.java:961) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:295)
Caused: hudson.util.HudsonFailedToLoad at hudson.WebAppMain$3.run(WebAppMain.java:312)

Can you please help me on this?

Thank you

[jetersen]

Not sure how this is relevant for JCasC, these seems like Jenkins core issues.

[jetersen]

https://www.jenkins.io/changelog/

What's new in 2.266 (2020-11-10)

This release replaces several key components. The Acegi security library used for authentication has been replaced by Spring Security (JEP-227). A fork of the XStream library used to read and write XML files has been replaced by the upstream version of XStream (JEP-228). Refer to the Spring and XStream updates (breaking changes!) blog post for more details.

[apwrv]

Extremely sorry for this. Couldn't find the issues page for Jenkins core. More over all the Google searches were pointing to configuration as code. hence raised the bug here.
Can you please provide me the link to the jenkins core issue page? Will post my bug there.
Thank you.

[jetersen]

I suggest waiting with the upgrade to the latest core that was just released today.

[apwrv]

Agreed! Thank you very much.
This upgrade was part of the weekly server patching and it was done by oversight. I have excluded Jenkins auto patching from the weekly patching. Luckily it was the test server.

[jetersen]

I am concerned @jglick that https://github.com/jenkinsci/jenkins/pull/4944 broke a lot of plugins based on those exception posted above.

[jglick]

Update to https://github.com/jenkinsci/role-strategy-plugin/releases/tag/role-strategy-3.1 and you should be OK.

[jglick]

This can be closed, nothing to do with JCasC.

[jetersen]

nicely spotted @jglick

[jglick]

I have excluded Jenkins auto patching from the weekly patching.

Probably wise, unless you supplement that with a human checking release notes and upgrade guide. And/or increment _plugins_ to the latest, not just core.

[apwrv]

thanks a lot @jglick!
Before this is marked as closed, can you please point me to any guides that i can check on how to get this role-strategy-plugin on board a dead jenkins?

[jetersen]

@apwrv download the hpi and drop the hpi file into JENKINS_HOME/plugins folder.
https://updates.jenkins.io/download/plugins/role-strategy/

[apwrv]

Thank you very much!

[abdennour]

Thank you @apwrv for reporting the issue
Thank you @jetersen for the solution.
However, i am wondering why i need to add the plugin (role-strategy) manually ?
Also what's the compatible version of this plugin

[sambit15k]

@apwrv download the hpi and drop the hpi file into JENKINS_HOME/plugins folder.
https://updates.jenkins.io/download/plugins/role-strategy/

Thanks @jetersen , for the amazing help . I am now able to start jenkins .