Azure-pipelines-agent: Hosted Azure DevOps Agents maintenance started failing with 2.161.1 with `Value cannot be null.` error on parameter name: extension
Have you tried troubleshooting?
I Ctrl+F'ed the troubleshooting stuff to see if I could find anything with respect to maintenance and did not find anything.
Agent Version and Platform
2.161.1
Azure DevOps Type and Version
https://dev.azure.com/DenverHealth-EpicCogito
What's not working?
Hosted Agent Maintenance in our default pool failed:
All of them threw this error
````
[section]Starting: Maintenance
2019-12-05T11:00:41.5519125Z ##[error]Value cannot be null.
Parameter name: extension
2019-12-05T11:00:41.5536476Z ##[section]Finishing: Maintenance`
````
Agent and Worker's Diagnostic Logs
Maintenance log downloaded from dev.azure.com:
2019-12-05T11:00:41.0219919Z ##[section]Starting: Maintenance
2019-12-05T11:00:41.2536005Z ##[section]Starting: Initialize job
2019-12-05T11:00:41.2536279Z Agent name: 'AzureDevops.Default.x64.MachineName.05'
2019-12-05T11:00:41.2536370Z Agent machine name: 'MachineName'
2019-12-05T11:00:41.2536415Z Current agent version: '2.161.1'
2019-12-05T11:00:41.2549261Z Agent running as: 'serviceAccountName'
2019-12-05T11:00:41.2598358Z Download all required tasks.
2019-12-05T11:00:41.2711995Z Start tracking orphan processes.
2019-12-05T11:00:41.2765377Z ##[section]Finishing: Initialize job
2019-12-05T11:00:41.3226556Z ##[section]Starting: Maintenance
2019-12-05T11:00:41.5519125Z ##[error]Value cannot be null.
Parameter name: extension
2019-12-05T11:00:41.5536476Z ##[section]Finishing: Maintenance
2019-12-05T11:00:41.5618782Z ##[section]Starting: Finalize Job
2019-12-05T11:00:41.5662029Z Start cleaning up orphan processes.
2019-12-05T11:00:41.5676931Z ##[section]Finishing: Finalize Job
2019-12-05T11:00:41.5736105Z ##[section]Finishing: Maintenance
Worker log obtained from VM (exception appears on line 267):
[2019-12-05 11:00:39Z INFO Program] Version: 2.161.1
[2019-12-05 11:00:39Z INFO Program] Commit: fb6be36621208f01256b18a42efce58342298802
[2019-12-05 11:00:39Z INFO Program] Culture: en-US
[2019-12-05 11:00:39Z INFO Program] UI Culture: en-US
[2019-12-05 11:00:39Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:39Z INFO HostContext] Well known directory 'Root': 'C:\Source\AzureDevops.Default.x64.MachineName.05'
[2019-12-05 11:00:39Z INFO HostContext] Well known config file 'Proxy': 'C:\Source\AzureDevops.Default.x64.MachineName.05\.proxy'
[2019-12-05 11:00:39Z INFO VstsAgentWebProxy] No proxy setting found.
[2019-12-05 11:00:39Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:39Z INFO HostContext] Well known directory 'Root': 'C:\Source\AzureDevops.Default.x64.MachineName.05'
[2019-12-05 11:00:39Z INFO HostContext] Well known config file 'Certificates': 'C:\Source\AzureDevops.Default.x64.MachineName.05\.certificates'
[2019-12-05 11:00:39Z INFO AgentCertificateManager] No certificate setting found.
[2019-12-05 11:00:39Z INFO Worker] Waiting to receive the job message from the channel.
[2019-12-05 11:00:39Z INFO Worker] Message received.
[2019-12-05 11:00:40Z INFO Worker] Job message:
{
"mask": [
{
"type": "regex",
"value": "***"
},
{
"type": "regex",
"value": "***"
}
],
"steps": [],
"variables": {
"maintenance.deleteworkingdirectory.daysthreshold": {
"value": "30"
},
"maintenance.jobtimeoutinminutes": {
"value": "60"
},
"system": {
"value": "poolmaintenance"
},
"system.hosttype": {
"value": "poolmaintenance"
},
"system.collectionId": {
"value": "351719c0-6acf-407c-9d9a-dbf3d5374438"
},
"system.servertype": {
"value": "Hosted"
},
"system.culture": {
"value": "en-US"
},
"system.teamFoundationCollectionUri": {
"value": "https://dev.azure.com/DenverHealth-EpicCogito/"
},
"system.taskDefinitionsUri": {
"value": "https://dev.azure.com/DenverHealth-EpicCogito/"
},
"System.TargetAgent": {
"value": "53"
},
"system.planId": {
"value": "6b0c5c09-b229-4c5e-b902-b1b35b827ddf"
},
"system.jobId": {
"value": "14295028-3d12-480d-a47c-417d533f6141"
},
"system.timelineId": {
"value": "6b0c5c09-b229-4c5e-b902-b1b35b827ddf"
},
"agent.retainDefaultEncoding": {
"value": "false"
},
"agent.disablelogplugin.TestResultLogPlugin": {
"value": "true"
},
"agent.disablelogplugin.TestFilePublisherPlugin": {
"value": "true"
}
},
"messageType": "PipelineAgentJobRequest",
"plan": {
"scopeIdentifier": "00000000-0000-0000-0000-000000000000",
"planType": "PoolMaintenance",
"version": 12,
"planId": "6b0c5c09-b229-4c5e-b902-b1b35b827ddf",
"planGroup": "6b0c5c09-b229-4c5e-b902-b1b35b827ddf",
"artifactUri": "vstfs:///DistributedTask/PoolMaintenance/1%2f91",
"artifactLocation": null,
"definition": {
"_links": {
"self": {
"href": "https://dev.azure.com/DenverHealth-EpicCogito/_apis/distributedtask/pools/1"
},
"web": {
"href": "https://dev.azure.com/DenverHealth-EpicCogito/_admin/_AgentPool?poolId=1&_a=poolsettings"
}
},
"id": 1,
"name": "PoolMaintenance"
},
"owner": {
"_links": {
"self": {
"href": "https://dev.azure.com/DenverHealth-EpicCogito/_apis/distributedtask/pools/1/maintenancejobs/91"
},
"web": {
"href": "https://dev.azure.com/DenverHealth-EpicCogito/_admin/_AgentPool?poolId=1&_a=maintenance"
}
},
"id": 91,
"name": "Requested by Microsoft.VisualStudio.Services.TFS"
}
},
"timeline": {
"id": "6b0c5c09-b229-4c5e-b902-b1b35b827ddf",
"changeId": 6,
"location": null
},
"jobId": "14295028-3d12-480d-a47c-417d533f6141",
"jobDisplayName": "Maintenance",
"jobName": "Maintenance",
"jobContainer": null,
"requestId": 0,
"lockedUntil": "0001-01-01T00:00:00",
"resources": {
"endpoints": [
{
"data": {
"ServerId": "351719c0-6acf-407c-9d9a-dbf3d5374438",
"ServerName": "DenverHealth-EpicCogito"
},
"name": "SystemVssConnection",
"url": "https://dev.azure.com/DenverHealth-EpicCogito/",
"authorization": {
"parameters": {
"AccessToken": ***
},
"scheme": "OAuth"
},
"isShared": false,
"isReady": false
}
]
},
"workspace": {}
}
[2019-12-05 11:00:40Z INFO JobRunner] Job ID 14295028-3d12-480d-a47c-417d533f6141
[2019-12-05 11:00:40Z INFO JobRunner] Creating job server with URL: https://dev.azure.com/DenverHealth-EpicCogito/
[2019-12-05 11:00:40Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2019-12-05 11:00:40Z INFO Worker] Listening for cancel message from the channel.
[2019-12-05 11:00:40Z INFO Worker] Waiting for the job to complete or for a cancel message from the channel.
[2019-12-05 11:00:40Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2019-12-05 11:00:40Z INFO JobServerQueue] Start process web console line queue.
[2019-12-05 11:00:40Z INFO JobServerQueue] Start process file upload queue.
[2019-12-05 11:00:40Z INFO JobServerQueue] Start process timeline update queue.
[2019-12-05 11:00:40Z INFO ConfigurationStore] currentAssemblyLocation: C:\Source\AzureDevops.Default.x64.MachineName.05\bin\Agent.Worker.dll
[2019-12-05 11:00:40Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:40Z INFO ConfigurationStore] binPath: C:\Source\AzureDevops.Default.x64.MachineName.05\bin
[2019-12-05 11:00:40Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:40Z INFO HostContext] Well known directory 'Root': 'C:\Source\AzureDevops.Default.x64.MachineName.05'
[2019-12-05 11:00:40Z INFO ConfigurationStore] RootFolder: C:\Source\AzureDevops.Default.x64.MachineName.05
[2019-12-05 11:00:40Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:40Z INFO HostContext] Well known directory 'Root': 'C:\Source\AzureDevops.Default.x64.MachineName.05'
[2019-12-05 11:00:40Z INFO HostContext] Well known config file 'Agent': 'C:\Source\AzureDevops.Default.x64.MachineName.05\.agent'
[2019-12-05 11:00:40Z INFO ConfigurationStore] ConfigFilePath: C:\Source\AzureDevops.Default.x64.MachineName.05\.agent
[2019-12-05 11:00:40Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:40Z INFO HostContext] Well known directory 'Root': 'C:\Source\AzureDevops.Default.x64.MachineName.05'
[2019-12-05 11:00:40Z INFO HostContext] Well known config file 'Credentials': 'C:\Source\AzureDevops.Default.x64.MachineName.05\.credentials'
[2019-12-05 11:00:40Z INFO ConfigurationStore] CredFilePath: C:\Source\AzureDevops.Default.x64.MachineName.05\.credentials
[2019-12-05 11:00:40Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:40Z INFO HostContext] Well known directory 'Root': 'C:\Source\AzureDevops.Default.x64.MachineName.05'
[2019-12-05 11:00:40Z INFO HostContext] Well known config file 'Service': 'C:\Source\AzureDevops.Default.x64.MachineName.05\.service'
[2019-12-05 11:00:40Z INFO ConfigurationStore] ServiceConfigFilePath: C:\Source\AzureDevops.Default.x64.MachineName.05\.service
[2019-12-05 11:00:40Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:40Z INFO HostContext] Well known directory 'Root': 'C:\Source\AzureDevops.Default.x64.MachineName.05'
[2019-12-05 11:00:40Z INFO HostContext] Well known config file 'Autologon': 'C:\Source\AzureDevops.Default.x64.MachineName.05\.autologon'
[2019-12-05 11:00:40Z INFO ConfigurationStore] AutoLogonSettingsFilePath: C:\Source\AzureDevops.Default.x64.MachineName.05\.autologon
[2019-12-05 11:00:40Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:40Z INFO HostContext] Well known directory 'Root': 'C:\Source\AzureDevops.Default.x64.MachineName.05'
[2019-12-05 11:00:40Z INFO HostContext] Well known config file 'Options': 'C:\Source\AzureDevops.Default.x64.MachineName.05\.options'
[2019-12-05 11:00:40Z INFO ConfigurationStore] RuntimeOptionsFilePath: C:\Source\AzureDevops.Default.x64.MachineName.05\.options
[2019-12-05 11:00:41Z INFO ConfigurationStore] Read setting file: 189 chars
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Root': 'C:\Source\AzureDevops.Default.x64.MachineName.05'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Diag': 'C:\Source\AzureDevops.Default.x64.MachineName.05\_diag'
[2019-12-05 11:00:41Z INFO JobRunner] Starting the job execution context.
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Root': 'C:\Source\AzureDevops.Default.x64.MachineName.05'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Work': 'C:\Source\AzureDevops.Default.x64.MachineName.05\_work'
[2019-12-05 11:00:41Z INFO JobRunner] Validating directory permissions for: 'C:\Source\AzureDevops.Default.x64.MachineName.05\_work'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Root': 'C:\Source\AzureDevops.Default.x64.MachineName.05'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Root': 'C:\Source\AzureDevops.Default.x64.MachineName.05'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Work': 'C:\Source\AzureDevops.Default.x64.MachineName.05\_work'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Root': 'C:\Source\AzureDevops.Default.x64.MachineName.05'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Externals': 'C:\Source\AzureDevops.Default.x64.MachineName.05\externals'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'ServerOM': 'C:\Source\AzureDevops.Default.x64.MachineName.05\externals\vstsom'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Root': 'C:\Source\AzureDevops.Default.x64.MachineName.05'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Work': 'C:\Source\AzureDevops.Default.x64.MachineName.05\_work'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Root': 'C:\Source\AzureDevops.Default.x64.MachineName.05'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Work': 'C:\Source\AzureDevops.Default.x64.MachineName.05\_work'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Root': 'C:\Source\AzureDevops.Default.x64.MachineName.05'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Work': 'C:\Source\AzureDevops.Default.x64.MachineName.05\_work'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Tools': 'C:\Source\AzureDevops.Default.x64.MachineName.05\_work\_tool'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Root': 'C:\Source\AzureDevops.Default.x64.MachineName.05'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Work': 'C:\Source\AzureDevops.Default.x64.MachineName.05\_work'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Temp': 'C:\Source\AzureDevops.Default.x64.MachineName.05\_work\_temp'
[2019-12-05 11:00:41Z INFO JobRunner] Creating task server with https://dev.azure.com/DenverHealth-EpicCogito/
[2019-12-05 11:00:41Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2019-12-05 11:00:41Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2019-12-05 11:00:41Z INFO JobRunner] Getting job extension.
[2019-12-05 11:00:41Z INFO ExtensionManager] Getting extensions for interface: 'Microsoft.VisualStudio.Services.Agent.Worker.IJobExtension'
[2019-12-05 11:00:41Z INFO ExtensionManager] Creating instance: Microsoft.VisualStudio.Services.Agent.Worker.Build.BuildJobExtension, Agent.Worker
[2019-12-05 11:00:41Z INFO ExtensionManager] Creating instance: Microsoft.VisualStudio.Services.Agent.Worker.Release.ReleaseJobExtension, Agent.Worker
[2019-12-05 11:00:41Z INFO ExtensionManager] Creating instance: Microsoft.VisualStudio.Services.Agent.Worker.Release.DeploymentJobExtension, Agent.Worker
[2019-12-05 11:00:41Z INFO ExtensionManager] Creating instance: Microsoft.VisualStudio.Services.Agent.Worker.Maintenance.MaintenanceJobExtension, Agent.Worker
[2019-12-05 11:00:41Z INFO JobRunner] Initialize job. Getting all job steps.
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Root': 'C:\Source\AzureDevops.Default.x64.MachineName.05'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Diag': 'C:\Source\AzureDevops.Default.x64.MachineName.05\_diag'
[2019-12-05 11:00:41Z INFO MaintenanceJobExtension] Run initial step from extension MaintenanceJobExtension.
[2019-12-05 11:00:41Z INFO MaintenanceJobExtension] Downloading task definitions.
[2019-12-05 11:00:41Z INFO MaintenanceJobExtension] Parsing all task's condition inputs.
[2019-12-05 11:00:41Z INFO MaintenanceJobExtension] Adding pre-job step from extension.
[2019-12-05 11:00:41Z INFO MaintenanceJobExtension] Adding post-job step from extension.
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Root': 'C:\Source\AzureDevops.Default.x64.MachineName.05'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Diag': 'C:\Source\AzureDevops.Default.x64.MachineName.05\_diag'
[2019-12-05 11:00:41Z INFO MaintenanceJobExtension] Adding post-job steps from tasks.
[2019-12-05 11:00:41Z INFO MaintenanceJobExtension] Total accessible running process: 88.
[2019-12-05 11:00:41Z INFO JobRunner] Total job steps: 1.
[2019-12-05 11:00:41Z INFO JobRunner] Run all job steps.
[2019-12-05 11:00:41Z INFO StepsRunner] Processing step: DisplayName='Maintenance', ContinueOnError=False, Enabled=True
[2019-12-05 11:00:41Z INFO ExpressionManager] Evaluating: SucceededNode()
[2019-12-05 11:00:41Z INFO ExpressionManager] Result: True
[2019-12-05 11:00:41Z INFO JobServerQueue] Try to append 1 batches web console lines for record '14295028-3d12-480d-a47c-417d533f6141', success rate: 1/1.
[2019-12-05 11:00:41Z INFO StepsRunner] Starting the step.
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Root': 'C:\Source\AzureDevops.Default.x64.MachineName.05'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Work': 'C:\Source\AzureDevops.Default.x64.MachineName.05\_work'
[2019-12-05 11:00:41Z INFO StepsRunner] Which: 'chcp'
[2019-12-05 11:00:41Z INFO StepsRunner] Location: 'C:\Windows\system32\chcp.COM'
[2019-12-05 11:00:41Z INFO ProcessInvokerWrapper] Starting process:
[2019-12-05 11:00:41Z INFO ProcessInvokerWrapper] File name: 'C:\Windows\system32\chcp.COM'
[2019-12-05 11:00:41Z INFO ProcessInvokerWrapper] Arguments: '65001'
[2019-12-05 11:00:41Z INFO ProcessInvokerWrapper] Working directory: 'C:\Source\AzureDevops.Default.x64.MachineName.05\_work'
[2019-12-05 11:00:41Z INFO ProcessInvokerWrapper] Require exit code zero: 'False'
[2019-12-05 11:00:41Z INFO ProcessInvokerWrapper] Encoding web name: ; code page: ''
[2019-12-05 11:00:41Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'False'
[2019-12-05 11:00:41Z INFO ProcessInvokerWrapper] Redirected STDIN: 'False'
[2019-12-05 11:00:41Z INFO ProcessInvokerWrapper] Persist current code page: 'True'
[2019-12-05 11:00:41Z INFO ProcessInvokerWrapper] Keep redirected STDIN open: 'False'
[2019-12-05 11:00:41Z INFO ProcessInvokerWrapper] High priority process: 'False'
[2019-12-05 11:00:41Z INFO ProcessInvokerWrapper] OOM score adjustment is Linux-only.
[2019-12-05 11:00:41Z INFO ProcessInvokerWrapper] Process started with process id 8020, waiting for process exit.
[2019-12-05 11:00:41Z INFO ProcessInvokerWrapper] Exited process 8020 with exit code 0
[2019-12-05 11:00:41Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
[2019-12-05 11:00:41Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
[2019-12-05 11:00:41Z INFO ProcessInvokerWrapper] Finished process 8020 with exit code 0, and elapsed time 00:00:00.0224282.
[2019-12-05 11:00:41Z INFO StepsRunner] Successfully returned to code page 65001 (UTF8)
[2019-12-05 11:00:41Z INFO ExtensionManager] Getting extensions for interface: 'Microsoft.VisualStudio.Services.Agent.Worker.Maintenance.IMaintenanceServiceProvider'
[2019-12-05 11:00:41Z INFO ExtensionManager] Creating instance: Microsoft.VisualStudio.Services.Agent.Worker.Build.BuildDirectoryManager, Agent.Worker
[2019-12-05 11:00:41Z ERR StepsRunner] Caught exception from step: System.ArgumentNullException: Value cannot be null.
Parameter name: extension
at Microsoft.VisualStudio.Services.Agent.ExtensionManager.Add[T](List`1 extensions, String assemblyQualifiedName)
at Microsoft.VisualStudio.Services.Agent.ExtensionManager.LoadExtensions[T]()
at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
at Microsoft.VisualStudio.Services.Agent.ExtensionManager.GetExtensions[T]()
at Microsoft.VisualStudio.Services.Agent.Worker.Maintenance.MaintenanceJobExtension.MaintainAsync(IExecutionContext executionContext, Object data)
at Microsoft.VisualStudio.Services.Agent.Worker.JobExtensionRunner.RunAsync()
at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
[2019-12-05 11:00:41Z INFO StepsRunner] Step result: Failed
[2019-12-05 11:00:41Z INFO StepsRunner] Update job result with current step result 'Failed'.
[2019-12-05 11:00:41Z INFO StepsRunner] Current state: job state = 'Failed'
[2019-12-05 11:00:41Z INFO JobRunner] Finalize job.
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Bin': 'C:\Source\AzureDevops.Default.x64.MachineName.05\bin'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Root': 'C:\Source\AzureDevops.Default.x64.MachineName.05'
[2019-12-05 11:00:41Z INFO HostContext] Well known directory 'Diag': 'C:\Source\AzureDevops.Default.x64.MachineName.05\_diag'
[2019-12-05 11:00:41Z INFO MaintenanceJobExtension] Total accessible running process: 88.
[2019-12-05 11:00:41Z INFO JobRunner] Job result after all job steps finish: Failed
[2019-12-05 11:00:41Z INFO JobRunner] Completing the job execution context.
[2019-12-05 11:00:41Z INFO JobRunner] Shutting down the job server queue.
[2019-12-05 11:00:41Z INFO JobServerQueue] Fire signal to shutdown all queues.
[2019-12-05 11:00:42Z INFO JobServerQueue] All queue process task stopped.
[2019-12-05 11:00:42Z INFO JobServerQueue] Try to append 1 batches web console lines for record '1c044240-d277-4a11-a219-ec79746c7b2e', success rate: 1/1.
[2019-12-05 11:00:42Z INFO JobServerQueue] Try to append 1 batches web console lines for record '66282d7f-b449-4f37-9bd0-966748dddc16', success rate: 1/1.
[2019-12-05 11:00:42Z INFO JobServerQueue] Try to append 1 batches web console lines for record '41b94241-687c-4b1d-bd13-ded0bdb37e25', success rate: 1/1.
[2019-12-05 11:00:42Z INFO JobServerQueue] Try to append 1 batches web console lines for record '14295028-3d12-480d-a47c-417d533f6141', success rate: 1/1.
[2019-12-05 11:00:42Z INFO JobServerQueue] Web console line queue drained.
[2019-12-05 11:00:42Z INFO JobServerQueue] Uploading 4 files in one shot.
[2019-12-05 11:00:42Z INFO JobServerQueue] Try to upload 4 log files or attachments, success rate: 4/4.
[2019-12-05 11:00:42Z INFO JobServerQueue] File upload queue drained.
[2019-12-05 11:00:42Z INFO JobServerQueue] Timeline update queue drained.
[2019-12-05 11:00:42Z INFO JobServerQueue] All queue process tasks have been stopped, and all queues are drained.
[2019-12-05 11:00:42Z INFO TempDirectoryManager] Cleaning agent temp folder: C:\Source\AzureDevops.Default.x64.MachineName.05\_work\_temp
[2019-12-05 11:00:42Z INFO JobRunner] Raising job completed event.
[2019-12-05 11:00:42Z INFO Worker] Job completed.
Microsoft folk, if you need to review our DevOps organization to figure this out, you have my permission (I am the admin of the organization) to do so.
We identified this issue because one of our VMs started running out of disk space and in the root cause analysis we identified this issue as contributing to the problem.
MarkIannucci
All 5 comments
Assigning to @jpricket since this is related some refactoring done for multi-repo
jtpetty
on 10 Dec 2019
Same error happens in combination of v2.162.0 with TFS 2017 (15.117.27024.0):
2019-12-11T03:00:34.6699533Z ##[section]Starting: Maintenance
2019-12-11T03:00:35.0462633Z ##[section]Starting: Initialize job
2019-12-11T03:00:35.0462915Z Agent name: 'XXXX'
2019-12-11T03:00:35.0463002Z Agent machine name: 'XXXX'
2019-12-11T03:00:35.0463052Z Current agent version: '2.162.0'
2019-12-11T03:00:35.0516387Z Agent running as: 'xxxx'
2019-12-11T03:00:35.0560870Z Download all required tasks.
2019-12-11T03:00:35.1588336Z Plugin: 'Test Result Parser plugin' is running in background.
2019-12-11T03:00:35.1588416Z Plugin: 'Test File Publisher plugin' is running in background.
2019-12-11T03:00:35.1588569Z Start tracking orphan processes.
2019-12-11T03:00:35.1643771Z ##[section]Finishing: Initialize job
2019-12-11T03:00:35.2200318Z ##[section]Starting: Maintenance
2019-12-11T03:00:35.6195332Z ##[error]Value cannot be null.
Parameter name: extension
2019-12-11T03:00:35.6213432Z ##[section]Finishing: Maintenance
2019-12-11T03:00:35.6363376Z ##[section]Starting: Finalize Job
2019-12-11T03:00:36.6925203Z Start cleaning up orphan processes.
2019-12-11T03:00:36.6936533Z ##[section]Finishing: Finalize Job
2019-12-11T03:00:36.7058985Z ##[section]Finishing: Maintenance
Cheers,
Frank
frkmlr
on 11 Dec 2019
@frkmlr - Thank you for confirming. We have triaged and know where the problem is and are working on getting a fix out ASAP.
jtpetty
on 11 Dec 2019
This fix was released in agent 2.164.6
jtpetty
on 28 Jan 2020
Thank you. We have 2.164.6 on prem and haven't seen this issue.
MarkIannucci
on 28 Jan 2020
Related issues
ActualReverend
路
3Comments
riezebosch
路
4Comments
Gordon-Beeming
路
3Comments
rusergeev
路
4Comments
averelon
路
3Comments