Configuration-as-code-plugin: Google Compute slaves are disconnected during job on configuration reload

Created on 27 Jan 2019 · 14Comments · Source: jenkinsci/configuration-as-code-plugin

Jenkins version - checked on: 2.161 and [LTS] 2.150.1
Plugin version - 1.4
OS: checked on: arch linux and in docker from image jenkinsci/jenkins:lts

I have running jobs on dynamic slaves provisioned with google compute engine. When I use reload existing configuration then all these dynamic slaves are disconnected and the running jobs will not be finished properly.
My simplified configuration of casc yaml looks like this:

jenkins:
  slaveAgentPort: 50000
  agentProtocols:
    - "JNLP4-connect"
  nodes:
  - permanent:
      labelString: "some_slave"
      nodeDescription: 'some node'
      retentionStrategy: always
      mode: NORMAL
      name: "some_slave"
      numExecutors: 4
      remoteFS: "/home/jenkins-slave"
      launcher:
        jnlp:
          workDirSettings:
            disabled: true
            internalDir: remoting
  clouds:
  - computeEngine:
      cloudName: "some-resource"
      configurations:
      - bootDiskAutoDelete: true
        bootDiskSizeGbStr: "50"
        bootDiskSourceImageName: "https://www.googleapis.com/compute/v1/projects/some-resource/global/images/jenkins-image"
        bootDiskSourceImageProject: "some-resource"
        bootDiskType: "https://www.googleapis.com/compute/v1/projects/some-resource/zones/some-zone4-c/diskTypes/pd-ssd"
        description: ""
        externalAddress: true
        labelString: "some-name"
        launchTimeoutSecondsStr: "120"
        machineType: "https://www.googleapis.com/compute/v1/projects/some-resource/zones/some-zone4-c/machineTypes/n1-standard-2"
        mode: NORMAL
        namePrefix: "some-name"
        networkConfiguration:
          autofilled:
            network: "https://www.googleapis.com/compute/v1/projects/some-resource/global/networks/some-resource"
            subnetwork: "https://www.googleapis.com/compute/v1/projects/some-resource/regions/some-zone4/subnetworks/some-subnet"
        networkTags: "some-tag"
        numExecutorsStr: "1"
        preemptible: false
        region: "https://www.googleapis.com/compute/v1/projects/some-resource/regions/some-zone4"
        retentionTimeMinutesStr: "2"
        runAsUser: "jenkins"
        serviceAccountEmail: "[email protected]"
        useInternalAddress: false
        zone: "https://www.googleapis.com/compute/v1/projects/some-resource/zones/some-zone4-c"
      credentialsId: "jenkins@some-resource"
      instanceCapStr: "45"
      projectId: "some-resource"
credentials:
  system:
    domainCredentials:
    - credentials:
      - googleRobotPrivateKey:
          projectId: "jenkins@some-resource"
          serviceAccountConfig:
            json:
              prevJsonKeyFile: /var/lib/jenkins/jenkins-some-resource.key

To reproduce error I prepared simple pipeline job:

node('some-name') {
   sh ('sleep 120')
}

When pipeline is on sh sleep then I'm reloading configuration:

Started by user my name
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] node
Still waiting to schedule task
‘some-name-5ani2s’ is offline

Running on some-name-5ani2s in /home/jenkins/.jenkins-slave/workspace/pipe
[Pipeline] {
[Pipeline] sh

+ sleep 120
# here is moment of reloading configuration from page http://jenkins-host/configuration-as-code/
Cannot contact some-name-5ani2s: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on some-name-5ani2s failed. The channel is closing down or has closed down

Below are all jenkins logs:
```Jan 27 19:39:25 mypc jenkins[22120]: Jan 27, 2019 7:39:25 PM com.google.jenkins.plugins.computeengine.ComputeEngineCloud provision
Jan 27 19:39:25 mypc jenkins[22120]: INFO: Provisioning node from config com.google.jenkins.plugins.computeengine.InstanceConfiguration@1eab7e13 for excess workload of 1 units of label 'some-name'
Jan 27 19:39:25 mypc jenkins[22120]: Jan 27, 2019 7:39:25 PM com.google.jenkins.plugins.computeengine.ComputeEngineCloud availableNodeCapacity
Jan 27 19:39:25 mypc jenkins[22120]: INFO: Found capacity for 45 nodes in cloud some-resource
Jan 27 19:39:27 mypc jenkins[22120]: Sent insert request
Jan 27 19:39:27 mypc jenkins[22120]: Jan 27, 2019 7:39:27 PM com.google.jenkins.plugins.computeengine.ComputeEngineComputerLauncher launch
Jan 27 19:39:27 mypc jenkins[22120]: INFO: Launch will wait 120000 for operation operation-1548614365988-58074e12dc736-6e299853-65f14fa6 to complete...
Jan 27 19:39:27 mypc jenkins[22120]: Jan 27, 2019 7:39:27 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
Jan 27 19:39:27 mypc jenkins[22120]: INFO: Started provisioning some-name-5ani2s from gce-some-resource with 1 executors. Remaining excess workload: 0
Jan 27 19:39:27 mypc jenkins[22120]: Jan 27, 2019 7:39:27 PM com.google.jenkins.plugins.computeengine.ComputeEngineCloud$1 call
Jan 27 19:39:27 mypc jenkins[22120]: INFO: Waiting 120000ms for node some-name-5ani2s to connect
Jan 27 19:39:27 mypc jenkins[22120]: Jan 27, 2019 7:39:27 PM com.google.jenkins.plugins.computeengine.ComputeEngineCloud$1 call
Jan 27 19:39:27 mypc jenkins[22120]: INFO: 0ms elapsed waiting for node some-name-5ani2s to connect
Jan 27 19:39:32 mypc jenkins[22120]: waiting...
Jan 27 19:39:33 mypc jenkins[22120]: Jan 27, 2019 7:39:33 PM com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Jan 27 19:39:33 mypc jenkins[22120]: INFO: Launching instance: some-name-5ani2s
Jan 27 19:39:40 mypc jenkins[22120]: waiting...
Jan 27 19:39:40 mypc jenkins[22120]: Jan 27, 2019 7:39:40 PM com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Jan 27 19:39:40 mypc jenkins[22120]: INFO: bootstrap
Jan 27 19:39:40 mypc jenkins[22120]: Jan 27, 2019 7:39:40 PM com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Jan 27 19:39:40 mypc jenkins[22120]: INFO: Getting keypair...
Jan 27 19:39:40 mypc jenkins[22120]: Jan 27, 2019 7:39:40 PM com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Jan 27 19:39:40 mypc jenkins[22120]: INFO: Using autogenerated keypair
Jan 27 19:39:40 mypc jenkins[22120]: Jan 27, 2019 7:39:40 PM com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Jan 27 19:39:40 mypc jenkins[22120]: INFO: Authenticating as jenkins
Jan 27 19:39:41 mypc jenkins[22120]: Jan 27, 2019 7:39:41 PM com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Jan 27 19:39:41 mypc jenkins[22120]: INFO: Connecting to 35.204.68.97 on port 22, with timeout 10000.
Jan 27 19:39:48 mypc jenkins[22120]: Jan 27, 2019 7:39:48 PM com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Jan 27 19:39:48 mypc jenkins[22120]: INFO: Connected via SSH.
Jan 27 19:39:49 mypc jenkins[22120]: Jan 27, 2019 7:39:49 PM com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Jan 27 19:39:49 mypc jenkins[22120]: INFO: connect fresh as root
Jan 27 19:39:49 mypc jenkins[22120]: Jan 27, 2019 7:39:49 PM com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Jan 27 19:39:49 mypc jenkins[22120]: INFO: Connecting to 35.204.68.97 on port 22, with timeout 10000.
Jan 27 19:39:49 mypc jenkins[22120]: Jan 27, 2019 7:39:49 PM com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Jan 27 19:39:49 mypc jenkins[22120]: INFO: Connected via SSH.
Jan 27 19:39:49 mypc jenkins[22120]: Jan 27, 2019 7:39:49 PM com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Jan 27 19:39:49 mypc jenkins[22120]: INFO: Copying slave.jar to: /tmp
Jan 27 19:39:50 mypc jenkins[22120]: Jan 27, 2019 7:39:50 PM com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Jan 27 19:39:50 mypc jenkins[22120]: INFO: Verifying: java -fullversion
Jan 27 19:39:51 mypc jenkins[22120]: Jan 27, 2019 7:39:51 PM com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Jan 27 19:39:51 mypc jenkins[22120]: INFO: Launching Jenkins agent via plugin SSH: java -jar /tmp/slave.jar
Jan 27 19:40:15 mypc jenkins[22120]: Jan 27, 2019 7:40:15 PM hudson.slaves.NodeProvisioner$2 run
Jan 27 19:40:15 mypc jenkins[22120]: INFO: some-name-5ani2s provisioning successfully completed. We have now 3 computer(s)

------ Right now is clicked "Reload existing configuration" from UI http://jenkins-host/configuration-as-code/

Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.ComputeEngineCloud.cloudName = some-resource
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.ComputeEngineCloud.projectId = some-resource
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.ComputeEngineCloud.credentialsId = jenkins@some-resource
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.ComputeEngineCloud.instanceCapStr = 45
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.namePrefix = some-name
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.region = https://www.googleapis.com/compute/v1/projects/some-resource/regions/some-zone4
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.zone = https://www.googleapis.com/compute/v1/projects/some-resource/zones/some-zone4-c
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.machineType = https://www.googleapis.com/compute/v1/projects/some-resource/zones/some-zone4-c/machineTypes/n1-standard-2
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.numExecutorsStr = 1
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.preemptible = false
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.labelString = some-name
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.description =
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.bootDiskType = https://www.googleapis.com/compute/v1/projects/some-resource/zones/some-zone4-c/diskTypes/pd-ssd
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.bootDiskAutoDelete = true
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.bootDiskSourceImageName = https://www.googleapis.com/compute/v1/projects/some-resource/global/images/jenkins-image
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.bootDiskSourceImageProject = some-resource
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.bootDiskSizeGbStr = 50
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.AutofilledNetworkConfiguration.network = https://www.googleapis.com/compute/v1/projects/some-resource/global/networks/some-resource
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.AutofilledNetworkConfiguration.subnetwork = https://www.googleapis.com/compute/v1/projects/some-resource/regions/some-zone4/subnetworks/some-subnet
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.networkConfiguration = {autofilled={}}
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.externalAddress = true
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.useInternalAddress = false
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.networkTags = some-tag
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.serviceAccountEmail = [email protected]
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.retentionTimeMinutesStr = 2
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.launchTimeoutSecondsStr = 120
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.mode = NORMAL
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.runAsUser = jenkins
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.ComputeEngineCloud.configurations = [{}]
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class hudson.slaves.DumbSlave.name = some_slave
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class hudson.slaves.DumbSlave.remoteFS = /home/jenkins-slave
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class jenkins.slaves.RemotingWorkDirSettings.disabled = true
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class jenkins.slaves.RemotingWorkDirSettings.internalDir = remoting
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.Attribute setValue
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting [email protected] = jenkins.slaves.RemotingWorkDirSettings@6dd9cde9
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class hudson.slaves.DumbSlave.launcher = {jnlp={}}
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.Attribute setValue
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting hudson.slaves.DumbSlave[some_slave].mode = NORMAL
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.Attribute setValue
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting hudson.slaves.DumbSlave[some_slave].numExecutors = 4
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.Attribute setValue
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting hudson.slaves.DumbSlave[some_slave].retentionStrategy = hudson.slaves.RetentionStrategy$Always@4880b95e
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.Attribute setValue
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting hudson.slaves.DumbSlave[some_slave].labelString = some_slave
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.Attribute setValue
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting hudson.slaves.DumbSlave[some_slave].nodeDescription = GCE slave
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.credentials.oauth.GoogleRobotPrivateKeyCredentials.projectId = jenkins@some-resource
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.credentials.oauth.JsonServiceAccountConfig.prevJsonKeyFile = /var/lib/jenkins/jenkins-some-resource.key
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.BaseConfigurator createAttribute
Jan 27 19:40:28 mypc jenkins[22120]: WARNING: Can't handle class com.google.jenkins.plugins.credentials.oauth.JsonServiceAccountConfig#jsonKeyFile: type is abstract but not Describable.
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.credentials.oauth.GoogleRobotPrivateKeyCredentials.serviceAccountConfig = {json={}}
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.cloudbees.plugins.credentials.domains.DomainCredentials.credentials = [{googleRobotPrivateKey={}}]
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.Attribute setValue
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting [email protected] = [JNLP4-connect]
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.ComputeEngineCloud.cloudName = some-resource
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.ComputeEngineCloud.projectId = some-resource
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.ComputeEngineCloud.credentialsId = jenkins@some-resource
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.ComputeEngineCloud.instanceCapStr = 45
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.namePrefix = some-name
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.region = https://www.googleapis.com/compute/v1/projects/some-resource/regions/some-zone4
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.zone = https://www.googleapis.com/compute/v1/projects/some-resource/zones/some-zone4-c
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.machineType = https://www.googleapis.com/compute/v1/projects/some-resource/zones/some-zone4-c/machineTypes/n1-standard-2
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.numExecutorsStr = 1
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.preemptible = false
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.labelString = some-name
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.description =
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.bootDiskType = https://www.googleapis.com/compute/v1/projects/some-resource/zones/some-zone4-c/diskTypes/pd-ssd
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.bootDiskAutoDelete = true
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.bootDiskSourceImageName = https://www.googleapis.com/compute/v1/projects/some-resource/global/images/jenkins-image
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.bootDiskSourceImageProject = some-resource
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.bootDiskSizeGbStr = 50
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.AutofilledNetworkConfiguration.network = https://www.googleapis.com/compute/v1/projects/some-resource/global/networks/some-resource
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.AutofilledNetworkConfiguration.subnetwork = https://www.googleapis.com/compute/v1/projects/some-resource/regions/some-zone4/subnetworks/some-subnet
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.networkConfiguration = {autofilled={}}
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.externalAddress = true
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.useInternalAddress = false
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.networkTags = some-tag
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.serviceAccountEmail = [email protected]
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.retentionTimeMinutesStr = 2
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.launchTimeoutSecondsStr = 120
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.mode = NORMAL
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.InstanceConfiguration.runAsUser = jenkins
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.computeengine.ComputeEngineCloud.configurations = [{}]
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.Attribute setValue
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting [email protected] = 50000
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class hudson.slaves.DumbSlave.name = some_slave
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class hudson.slaves.DumbSlave.remoteFS = /home/jenkins-slave
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class jenkins.slaves.RemotingWorkDirSettings.disabled = true
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class jenkins.slaves.RemotingWorkDirSettings.internalDir = remoting
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.Attribute setValue
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting [email protected] = jenkins.slaves.RemotingWorkDirSettings@7b41d681
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class hudson.slaves.DumbSlave.launcher = {jnlp={}}
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.Attribute setValue
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting hudson.slaves.DumbSlave[some_slave].mode = NORMAL
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.Attribute setValue
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting hudson.slaves.DumbSlave[some_slave].numExecutors = 4
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.Attribute setValue
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting hudson.slaves.DumbSlave[some_slave].retentionStrategy = hudson.slaves.RetentionStrategy$Always@55c2556a
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.Attribute setValue
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting hudson.slaves.DumbSlave[some_slave].labelString = some_slave
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.Attribute setValue
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting hudson.slaves.DumbSlave[some_slave].nodeDescription = GCE slave
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.Attribute setValue
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting [email protected] = [hudson.slaves.DumbSlave[some_slave]]
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.credentials.oauth.GoogleRobotPrivateKeyCredentials.projectId = jenkins@some-resource
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.credentials.oauth.JsonServiceAccountConfig.prevJsonKeyFile = /var/lib/jenkins/jenkins-some-resource.key
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.BaseConfigurator createAttribute
Jan 27 19:40:28 mypc jenkins[22120]: WARNING: Can't handle class com.google.jenkins.plugins.credentials.oauth.JsonServiceAccountConfig#jsonKeyFile: type is abstract but not Describable.
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.google.jenkins.plugins.credentials.oauth.GoogleRobotPrivateKeyCredentials.serviceAccountConfig = {json={}}
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator tryConstructor
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting class com.cloudbees.plugins.credentials.domains.DomainCredentials.credentials = [{googleRobotPrivateKey={}}]
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.Attribute setValue
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting com.cloudbees.plugins.credentials.SystemCredentialsProvider@23a4a6d6.domainCredentials = [com.cloudbees.plugins.credentials.domains.DomainCredentials@471e551e]
Jan 27 19:40:28 mypc jenkins[22120]: Jan 27, 2019 7:40:28 PM io.jenkins.plugins.casc.Attribute setValue
Jan 27 19:40:28 mypc jenkins[22120]: INFO: Setting com.cloudbees.plugins.credentials.GlobalCredentialsConfiguration@3669979b.system = com.cloudbees.plugins.credentials.SystemCredentialsProvider@23a4a6d6
```
Looks like if I remove from casc configuration jenkins >> nodes, then cloud slaves are not disconnected on reloading configuration. But in production jenkins instance I need both types of slaves, and at this moment I'm only able to reload configuration when there is no jobs running in gcp slaves.

Permanent nodes always remains online after the configuration reload.

Source

devqore

Most helpful comment

Properly requires some more code for configuring nodes inside JCasC, not overwriting but appending and checking for conflicts and reporting those conflicts

jetersen on 26 Mar 2019

👍2

All 14 comments

This is properly an issue with the plugin and not an issue for JCasC to fix.

jetersen on 28 Jan 2019

@devqore I'm getting this issue too. Did you raise anything with the Google Compute Engine plugin? I'll notify them of this issue if not

robincsmith on 22 Mar 2019

@robincsmith thank you for reporting to gce :)

devqore on 22 Mar 2019

👍1

@casz I think there may be a wider issue here. Several plugins (docker, yet-another-docker, GCE) manage their ephemeral build slaves by temporarily adding nodes (visible under ${JENKINS_URL}/computer). During a configuration reload, if the jenkins.yaml contains a nodes section, then those ephemeral nodes get removed and the channels closed mid-build - regardless of the plugin. If there's no nodes section in the jenkins.yaml, it looks like JCasC does not attempt to remove the ephemeral nodes.

Since this an issue affecting the way several plugins manage ephemeral nodes, maybe it's something that can be considered by the JCasC?

Happy to report this as a separate issue if you prefer.

robincsmith on 26 Mar 2019

we experience the same issue. the bigger the config the more possible the connection loss is.
that why we only deploy new configurations once all jobs are done or we deploy at 2am.

HerrmannHinz on 26 Mar 2019

@HerrmannHinz out of curiosity.. why you need to change config that often?

ingwarsw on 26 Mar 2019

@ingwarsw I can give you few examples, why I need to reload configuration:

adding new credentials (credentials -> system -> domainCredentials)
changing permissions for ldap group/user (jenkins -> authorizationStrategy -> projectMatrix -> grantedPermissions)
adding new google cloud (jenkins -> clouds -> computeEngine -> configurations) to use new kind of images because changes of existing instances like machine setup or current image version are realized with template applied from the terraform (google_compute_instance_template)

devqore on 26 Mar 2019

@devqore I see I should start managing my server in that way, cause as I see everything can be managed that way :) Cool..

And I see you reload all config at once.. so this changes will be a loot..
I was thinking only about cloud configs..

ingwarsw on 26 Mar 2019

Properly requires some more code for configuring nodes inside JCasC, not overwriting but appending and checking for conflicts and reporting those conflicts

jetersen on 26 Mar 2019

👍2

The linked code below would require changes I assume without knowing precisely what the issue is :confused:
If someone would be willing to attempt a test and perhaps a fix even.

https://github.com/jenkinsci/configuration-as-code-plugin/blob/6d5d625d81829b6b31fa0d9b93eec14cf86b2c08/plugin/src/main/java/io/jenkins/plugins/casc/core/JenkinsConfigurator.java#L59-L64

jetersen on 26 Mar 2019

@HerrmannHinz out of curiosity.. why you need to change config that often?

@ingwarsw, like @devqore said:

almost daily we need to add several service users to our jenkins instances.
we have tons of credentials on the machines already:

$ wc deploy/manifests/jenkins/non-prod/casc/secrets.yaml
    2407    6710  113505 deploy/manifests/jenkins/non-prod/casc/secrets.yaml

some ideas what could cause the issue:

i have the feeling that reloading the massive configuration is causing some memory issue jenkins which leads to the disconnects? is that possible?
maybe also connections get reset when (re) adding (in our case) the connection to the kubernetes where the worker nodes do run on?

HerrmannHinz on 26 Mar 2019

@casz Im using a lot Google cloud plugin and code mentioned below would not filter them out..
If you want I can add 2 extra filters that should catch most of cases..

The linked code below would require changes I assume without knowing precisely what the issue is 😕
If someone would be willing to attempt a test and perhaps a fix even.

configuration-as-code-plugin/plugin/src/main/java/io/jenkins/plugins/casc/core/JenkinsConfigurator.java

Lines 59 to 64 in 6d5d625

ingwarsw on 27 Mar 2019

I would love that :blush:

jetersen on 27 Mar 2019

This should be fixed in v1.10

jetersen on 6 Apr 2019

Was this page helpful?

0 / 5 - 0 ratings