Azure-pipelines-agent: Cannot unregister Agent from server/pool, on prem, 2019, "TF400813: Resource not available for anonymous access. Client authentication required."

[ActualReverend]

Agent Version and Platform

Version of your agent?
2.155.1

OS of the machine running the agent?
RHEL 7.7

Azure DevOps Type and Version

on-premises Azure DevOps Server 2019

If on-premises, which release?
2019, "help" says: "Version Dev17.M153.5"

What's not working?

I am not able to "unregister" the agent from the ADS server pool. I get the error: "TF400813: Resource not available for anonymous access. Client authentication required."
Registration/configuration/setup worked just fine.
I am using the same PAT that registered the agent. (have tried others too, more info below)

I have tried other suggestions from this forum, including but not limited to:
-- different pats: full access pats, reduced permissions pats, even bogus PATs (oddly enough, I get the same error)
-- tried running interactively, also tried running unattended
-- double checked that Anonymous Authentication was enabled

I have not yet tried any newer versions of the agent yet, as I looked at the change log and didn't notice anything related to this problem.

Agent Diagnostic Logs

command ran:

#

[<serviceAccount>_jbrn@<hostname> agent]$ ./config.sh remove --auth PAT --token <token generated went here> 
Removing agent from the server
Failed: Removing agent from the server
TF400813: Resource not available for anonymous access. Client authentication required.

#

_diag:

#

[<part_of_service_account_name>_jbrn@<local_servername_was_here> _diag]$ cat Agent_20200605-143857-utc.log
[2020-06-05 14:38:57Z INFO AgentProcess] Agent is built for Linux (X64) - linux-x64.
[2020-06-05 14:38:57Z INFO AgentProcess] RuntimeInformation: Linux 3.10.0-1062.9.1.el7.x86_64 #1 SMP Mon Dec 2 08:31:54 EST 2019.
[2020-06-05 14:38:57Z INFO AgentProcess] Version: 2.155.1
[2020-06-05 14:38:57Z INFO AgentProcess] Commit: 2a1d4271df4d990b6c8ff0d65e15b14e957393d8
[2020-06-05 14:38:57Z INFO AgentProcess] Culture: en-US
[2020-06-05 14:38:57Z INFO AgentProcess] UI Culture: en-US
[2020-06-05 14:38:57Z INFO HostContext] Well known directory 'Bin': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/bin'
[2020-06-05 14:38:57Z INFO HostContext] Well known directory 'Root': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent'
[2020-06-05 14:38:57Z INFO AgentProcess] Validating directory permissions for: '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent'
[2020-06-05 14:38:57Z INFO HostContext] Well known directory 'Bin': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/bin'
[2020-06-05 14:38:57Z INFO HostContext] Well known directory 'Root': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent'
[2020-06-05 14:38:57Z INFO CommandLineParser] Parse
[2020-06-05 14:38:57Z INFO CommandLineParser] Parsing 5 args
[2020-06-05 14:38:57Z INFO CommandLineParser] parsing argument
[2020-06-05 14:38:57Z INFO CommandLineParser] HasArgs: False
[2020-06-05 14:38:57Z INFO CommandLineParser] Adding Command: remove
[2020-06-05 14:38:57Z INFO CommandLineParser] parsing argument
[2020-06-05 14:38:57Z INFO CommandLineParser] HasArgs: True
[2020-06-05 14:38:57Z INFO CommandLineParser] arg: auth
[2020-06-05 14:38:57Z INFO CommandLineParser] parsing argument
[2020-06-05 14:38:57Z INFO CommandLineParser] HasArgs: True
[2020-06-05 14:38:57Z INFO CommandLineParser] Adding option 'auth': 'PAT'
[2020-06-05 14:38:57Z INFO CommandLineParser] parsing argument
[2020-06-05 14:38:57Z INFO CommandLineParser] HasArgs: True
[2020-06-05 14:38:57Z INFO CommandLineParser] arg: token
[2020-06-05 14:38:57Z INFO CommandLineParser] parsing argument
[2020-06-05 14:38:57Z INFO CommandLineParser] HasArgs: True
[2020-06-05 14:38:57Z INFO CommandLineParser] Adding option 'token': '***'
[2020-06-05 14:38:57Z INFO AgentProcess] Arguments parsed
[2020-06-05 14:38:57Z INFO HostContext] Well known directory 'Bin': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/bin'
[2020-06-05 14:38:57Z INFO HostContext] Well known directory 'Root': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent'
[2020-06-05 14:38:57Z INFO HostContext] Well known config file 'Proxy': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/.proxy'
[2020-06-05 14:38:57Z INFO VstsAgentWebProxy] No proxy setting found.
[2020-06-05 14:38:57Z INFO HostContext] Well known directory 'Bin': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/bin'
[2020-06-05 14:38:57Z INFO HostContext] Well known directory 'Root': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent'
[2020-06-05 14:38:57Z INFO HostContext] Well known config file 'Certificates': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/.certificates'
[2020-06-05 14:38:57Z INFO AgentCertificateManager] No certificate setting found.
[2020-06-05 14:38:57Z INFO Agent] ExecuteCommand
[2020-06-05 14:38:57Z INFO ConfigurationStore] currentAssemblyLocation: /opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/bin/Agent.Listener.dll
[2020-06-05 14:38:57Z INFO HostContext] Well known directory 'Bin': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/bin'
[2020-06-05 14:38:57Z INFO ConfigurationStore] binPath: /opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/bin
[2020-06-05 14:38:57Z INFO HostContext] Well known directory 'Bin': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/bin'
[2020-06-05 14:38:57Z INFO HostContext] Well known directory 'Root': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent'
[2020-06-05 14:38:57Z INFO ConfigurationStore] RootFolder: /opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent
[2020-06-05 14:38:57Z INFO HostContext] Well known directory 'Bin': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/bin'
[2020-06-05 14:38:57Z INFO HostContext] Well known directory 'Root': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent'
[2020-06-05 14:38:57Z INFO HostContext] Well known config file 'Agent': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/.agent'
[2020-06-05 14:38:57Z INFO ConfigurationStore] ConfigFilePath: /opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/.agent
[2020-06-05 14:38:57Z INFO HostContext] Well known directory 'Bin': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/bin'
[2020-06-05 14:38:57Z INFO HostContext] Well known directory 'Root': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent'
[2020-06-05 14:38:57Z INFO HostContext] Well known config file 'Credentials': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/.credentials'
[2020-06-05 14:38:57Z INFO ConfigurationStore] CredFilePath: /opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/.credentials
[2020-06-05 14:38:57Z INFO HostContext] Well known directory 'Bin': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/bin'
[2020-06-05 14:38:57Z INFO HostContext] Well known directory 'Root': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent'
[2020-06-05 14:38:57Z INFO HostContext] Well known config file 'Service': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/.service'
[2020-06-05 14:38:57Z INFO ConfigurationStore] ServiceConfigFilePath: /opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/.service
[2020-06-05 14:38:57Z INFO HostContext] Well known directory 'Bin': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/bin'
[2020-06-05 14:38:57Z INFO HostContext] Well known directory 'Root': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent'
[2020-06-05 14:38:57Z INFO HostContext] Well known config file 'Autologon': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/.autologon'
[2020-06-05 14:38:57Z INFO ConfigurationStore] AutoLogonSettingsFilePath: /opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/.autologon
[2020-06-05 14:38:57Z INFO HostContext] Well known directory 'Bin': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/bin'
[2020-06-05 14:38:57Z INFO HostContext] Well known directory 'Root': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent'
[2020-06-05 14:38:57Z INFO HostContext] Well known config file 'Options': '/opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/.options'
[2020-06-05 14:38:57Z INFO ConfigurationStore] RuntimeOptionsFilePath: /opt/tfs_agents/ansibleCreated/<part_of_service_account_name>_jbrn/agent/.options
[2020-06-05 14:38:57Z INFO CommandSettings] Flag 'help': 'False'
[2020-06-05 14:38:57Z INFO CommandSettings] Flag 'version': 'False'
[2020-06-05 14:38:57Z INFO CommandSettings] Flag 'commit': 'False'
[2020-06-05 14:38:57Z INFO CommandSettings] Command 'configure': 'False'
[2020-06-05 14:38:57Z INFO CommandSettings] Command 'remove': 'True'
[2020-06-05 14:38:57Z INFO ConfigurationStore] IsServiceConfigured()
[2020-06-05 14:38:57Z INFO ConfigurationStore] IsServiceConfigured: False
[2020-06-05 14:38:57Z INFO Terminal] WRITE LINE: Removing agent from the server
[2020-06-05 14:38:57Z INFO ConfigurationStore] IsConfigured()
[2020-06-05 14:38:57Z INFO ConfigurationStore] IsConfigured: True
[2020-06-05 14:38:57Z INFO ConfigurationStore] HasCredentials()
[2020-06-05 14:38:57Z INFO ConfigurationStore] stored True
[2020-06-05 14:38:57Z INFO ConfigurationStore] Read setting file: 192 chars
[2020-06-05 14:38:57Z INFO ConfigurationManager] GetCredentialProvider
[2020-06-05 14:38:57Z INFO CommandSettings] Arg 'auth': 'PAT'
[2020-06-05 14:38:57Z INFO CommandSettings] Remove auth from Arg dictionary.
[2020-06-05 14:38:57Z INFO ConfigurationManager] Creating credential for auth: PAT
[2020-06-05 14:38:57Z INFO CredentialManager] GetCredentialProvider
[2020-06-05 14:38:57Z INFO CredentialManager] Creating type PAT
[2020-06-05 14:38:57Z INFO CredentialManager] Creating credential type: PAT
[2020-06-05 14:38:57Z INFO PersonalAccessToken] EnsureCredential
[2020-06-05 14:38:57Z INFO CommandSettings] Arg 'token': '***'
[2020-06-05 14:38:57Z INFO CommandSettings] Remove token from Arg dictionary.
[2020-06-05 14:38:57Z INFO PersonalAccessToken] GetVssCredentials
[2020-06-05 14:38:57Z INFO PersonalAccessToken] token retrieved: 52 chars
[2020-06-05 14:38:57Z INFO PersonalAccessToken] cred created
[2020-06-05 14:38:57Z INFO ConfigurationManager] cred retrieved
[2020-06-05 14:38:57Z INFO ConfigurationManager] Agent configured for deploymentGroup : False
[2020-06-05 14:38:57Z INFO ExtensionManager] Getting extensions for interface: 'Microsoft.VisualStudio.Services.Agent.Listener.Configuration.IConfigurationProvider'
[2020-06-05 14:38:57Z INFO ExtensionManager] Creating instance: Microsoft.VisualStudio.Services.Agent.Listener.Configuration.BuildReleasesAgentConfigProvider, Agent.Listener
[2020-06-05 14:38:57Z INFO ExtensionManager] Creating instance: Microsoft.VisualStudio.Services.Agent.Listener.Configuration.DeploymentGroupAgentConfigProvider, Agent.Listener
[2020-06-05 14:38:57Z INFO ExtensionManager] Creating instance: Microsoft.VisualStudio.Services.Agent.Listener.Configuration.SharedDeploymentAgentConfigProvider, Agent.Listener
[2020-06-05 14:38:57Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2020-06-05 14:38:58Z WARN VisualStudioServices] Authentication failed with status code 401.
X-TFS-ProcessId: 64a8285c-ebd1-4480-92ad-56359c286d1d
Strict-Transport-Security: max-age=31536000; includeSubDomains
ActivityId: aa09031f-7949-48bb-b573-d6c02a227dec
X-TFS-Session: 2f379482-e9cd-4007-80f6-17ee1643dcaa
X-VSS-E2EID: 07e81f36-d225-45a4-9688-088b7f39be58
WWW-Authenticate: Basic realm="https://<fully.qualified.servername.was.here>/", Negotiate, NTLM
X-Powered-By: ASP.NET
Access-Control-Allow-Origin: https://<fully.qualified.servername.was.here>
P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
Lfs-Authenticate: NTLM
X-Content-Type-Options: nosniff
Date: Fri, 05 Jun 2020 14:38:56 GMT

[2020-06-05 14:38:58Z WARN VisualStudioServices] Basic issued token provider instance 11898202 requires an interactive prompt which is not allowed by the current settings
[2020-06-05 14:38:58Z ERR  VisualStudioServices] GET request to https://<fully.qualified.servername.was.here>/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 is not authorized. Details: VS30063: You are not authorized to access https://<fully.qualified.servername.was.here>.
[2020-06-05 14:38:58Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2020-06-05 14:38:58Z INFO LocationServer] Catch exception during connect. 4 attempt left.
[2020-06-05 14:38:58Z ERR  LocationServer] Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: VS30063: You are not authorized to access https://<fully.qualified.servername.was.here>.
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Location.Client.LocationHttpClient.GetConnectionDataAsync(ConnectOptions connectOptions, Int64 lastChangeId, CancellationToken cancellationToken, Object userState)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.GetConnectionDataAsync(ConnectOptions connectOptions, Int32 lastChangeId, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.ConnectAsync(ConnectOptions connectOptions, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Agent.LocationServer.ConnectAsync(VssConnection jobConnection)
[2020-06-05 14:38:58Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2020-06-05 14:38:58Z WARN VisualStudioServices] Authentication failed with status code 401.
X-TFS-ProcessId: 64a8285c-ebd1-4480-92ad-56359c286d1d
Strict-Transport-Security: max-age=31536000; includeSubDomains
ActivityId: aa090300-7949-48bb-b573-d6c02a227dec
X-TFS-Session: 2f379482-e9cd-4007-80f6-17ee1643dcaa
X-VSS-E2EID: 756c9b95-7b01-4701-b94d-9635787b526c
X-TFS-SoapException: %3C%3Fxml%20version%3D%221.0%22%20encoding%3D%22utf-8%22%3F%3E%3Csoap%3AEnvelope%20xmlns%3Asoap%3D%22http%3A%2F%2Fwww.w3.org%2F2003%2F05%2Fsoap-envelope%22%3E%3Csoap%3ABody%3E%3Csoap%3AFault%3E%3Csoap%3ACode%3E%3Csoap%3AValue%3Esoap%3AReceiver%3C%2Fsoap%3AValue%3E%3Csoap%3ASubcode%3E%3Csoap%3AValue%3EUnauthorizedRequestException%3C%2Fsoap%3AValue%3E%3C%2Fsoap%3ASubcode%3E%3C%2Fsoap%3ACode%3E%3Csoap%3AReason%3E%3Csoap%3AText%20xml%3Alang%3D%22en%22%3ETF400813%3A%20Resource%20not%20available%20for%20anonymous%20access.%20Client%20authentication%20required.%3C%2Fsoap%3AText%3E%3C%2Fsoap%3AReason%3E%3C%2Fsoap%3AFault%3E%3C%2Fsoap%3ABody%3E%3C%2Fsoap%3AEnvelope%3E
X-TFS-ServiceError: TF400813%3A%20Resource%20not%20available%20for%20anonymous%20access.%20Client%20authentication%20required.
WWW-Authenticate: Bearer, Basic realm="https://<fully.qualified.servername.was.here>/", Negotiate, NTLM
X-Powered-By: ASP.NET
Access-Control-Allow-Origin: https://<fully.qualified.servername.was.here>
P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
Lfs-Authenticate: NTLM
X-Content-Type-Options: nosniff
Date: Fri, 05 Jun 2020 14:38:56 GMT

[2020-06-05 14:38:58Z WARN VisualStudioServices] Basic issued token provider instance 11898202 requires an interactive prompt which is not allowed by the current settings
[2020-06-05 14:38:58Z ERR  VisualStudioServices] GET request to https://<fully.qualified.servername.was.here>/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 is not authorized. Details: TF400813: Resource not available for anonymous access. Client authentication required.
[2020-06-05 14:38:58Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2020-06-05 14:38:58Z INFO LocationServer] Catch exception during connect. 3 attempt left.
[2020-06-05 14:38:58Z ERR  LocationServer] Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: TF400813: Resource not available for anonymous access. Client authentication required.
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Location.Client.LocationHttpClient.GetConnectionDataAsync(ConnectOptions connectOptions, Int64 lastChangeId, CancellationToken cancellationToken, Object userState)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.GetConnectionDataAsync(ConnectOptions connectOptions, Int32 lastChangeId, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.ConnectAsync(ConnectOptions connectOptions, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Agent.LocationServer.ConnectAsync(VssConnection jobConnection)
[2020-06-05 14:38:58Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2020-06-05 14:38:58Z WARN VisualStudioServices] Authentication failed with status code 401.
X-TFS-ProcessId: 64a8285c-ebd1-4480-92ad-56359c286d1d
Strict-Transport-Security: max-age=31536000; includeSubDomains
ActivityId: aa090301-7949-48bb-b573-d6c02a227dec
X-TFS-Session: 2f379482-e9cd-4007-80f6-17ee1643dcaa
X-VSS-E2EID: af7e65b6-cef4-47ed-a8b3-4ccd9de8908f
X-TFS-SoapException: %3C%3Fxml%20version%3D%221.0%22%20encoding%3D%22utf-8%22%3F%3E%3Csoap%3AEnvelope%20xmlns%3Asoap%3D%22http%3A%2F%2Fwww.w3.org%2F2003%2F05%2Fsoap-envelope%22%3E%3Csoap%3ABody%3E%3Csoap%3AFault%3E%3Csoap%3ACode%3E%3Csoap%3AValue%3Esoap%3AReceiver%3C%2Fsoap%3AValue%3E%3Csoap%3ASubcode%3E%3Csoap%3AValue%3EUnauthorizedRequestException%3C%2Fsoap%3AValue%3E%3C%2Fsoap%3ASubcode%3E%3C%2Fsoap%3ACode%3E%3Csoap%3AReason%3E%3Csoap%3AText%20xml%3Alang%3D%22en%22%3ETF400813%3A%20Resource%20not%20available%20for%20anonymous%20access.%20Client%20authentication%20required.%3C%2Fsoap%3AText%3E%3C%2Fsoap%3AReason%3E%3C%2Fsoap%3AFault%3E%3C%2Fsoap%3ABody%3E%3C%2Fsoap%3AEnvelope%3E
X-TFS-ServiceError: TF400813%3A%20Resource%20not%20available%20for%20anonymous%20access.%20Client%20authentication%20required.
WWW-Authenticate: Bearer, Basic realm="https://<fully.qualified.servername.was.here>/", Negotiate, NTLM
X-Powered-By: ASP.NET
Access-Control-Allow-Origin: https://<fully.qualified.servername.was.here>
P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
Lfs-Authenticate: NTLM
X-Content-Type-Options: nosniff
Date: Fri, 05 Jun 2020 14:38:56 GMT

[2020-06-05 14:38:58Z WARN VisualStudioServices] Basic issued token provider instance 11898202 requires an interactive prompt which is not allowed by the current settings
[2020-06-05 14:38:58Z ERR  VisualStudioServices] GET request to https://<fully.qualified.servername.was.here>/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 is not authorized. Details: TF400813: Resource not available for anonymous access. Client authentication required.
[2020-06-05 14:38:58Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2020-06-05 14:38:58Z INFO LocationServer] Catch exception during connect. 2 attempt left.
[2020-06-05 14:38:58Z ERR  LocationServer] Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: TF400813: Resource not available for anonymous access. Client authentication required.
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Location.Client.LocationHttpClient.GetConnectionDataAsync(ConnectOptions connectOptions, Int64 lastChangeId, CancellationToken cancellationToken, Object userState)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.GetConnectionDataAsync(ConnectOptions connectOptions, Int32 lastChangeId, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.ConnectAsync(ConnectOptions connectOptions, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Agent.LocationServer.ConnectAsync(VssConnection jobConnection)
[2020-06-05 14:38:58Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2020-06-05 14:38:58Z WARN VisualStudioServices] Authentication failed with status code 401.
X-TFS-ProcessId: 64a8285c-ebd1-4480-92ad-56359c286d1d
Strict-Transport-Security: max-age=31536000; includeSubDomains
ActivityId: aa090302-7949-48bb-b573-d6c02a227dec
X-TFS-Session: 2f379482-e9cd-4007-80f6-17ee1643dcaa
X-VSS-E2EID: 58f980b6-8fda-4f99-9d4a-6c86be8fe622
X-TFS-SoapException: %3C%3Fxml%20version%3D%221.0%22%20encoding%3D%22utf-8%22%3F%3E%3Csoap%3AEnvelope%20xmlns%3Asoap%3D%22http%3A%2F%2Fwww.w3.org%2F2003%2F05%2Fsoap-envelope%22%3E%3Csoap%3ABody%3E%3Csoap%3AFault%3E%3Csoap%3ACode%3E%3Csoap%3AValue%3Esoap%3AReceiver%3C%2Fsoap%3AValue%3E%3Csoap%3ASubcode%3E%3Csoap%3AValue%3EUnauthorizedRequestException%3C%2Fsoap%3AValue%3E%3C%2Fsoap%3ASubcode%3E%3C%2Fsoap%3ACode%3E%3Csoap%3AReason%3E%3Csoap%3AText%20xml%3Alang%3D%22en%22%3ETF400813%3A%20Resource%20not%20available%20for%20anonymous%20access.%20Client%20authentication%20required.%3C%2Fsoap%3AText%3E%3C%2Fsoap%3AReason%3E%3C%2Fsoap%3AFault%3E%3C%2Fsoap%3ABody%3E%3C%2Fsoap%3AEnvelope%3E
X-TFS-ServiceError: TF400813%3A%20Resource%20not%20available%20for%20anonymous%20access.%20Client%20authentication%20required.
WWW-Authenticate: Bearer, Basic realm="https://<fully.qualified.servername.was.here>/", Negotiate, NTLM
X-Powered-By: ASP.NET
Access-Control-Allow-Origin: https://<fully.qualified.servername.was.here>
P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
Lfs-Authenticate: NTLM
X-Content-Type-Options: nosniff
Date: Fri, 05 Jun 2020 14:38:56 GMT

[2020-06-05 14:38:58Z WARN VisualStudioServices] Basic issued token provider instance 11898202 requires an interactive prompt which is not allowed by the current settings
[2020-06-05 14:38:58Z ERR  VisualStudioServices] GET request to https://<fully.qualified.servername.was.here>/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 is not authorized. Details: TF400813: Resource not available for anonymous access. Client authentication required.
[2020-06-05 14:38:58Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2020-06-05 14:38:58Z INFO LocationServer] Catch exception during connect. 1 attempt left.
[2020-06-05 14:38:58Z ERR  LocationServer] Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: TF400813: Resource not available for anonymous access. Client authentication required.
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Location.Client.LocationHttpClient.GetConnectionDataAsync(ConnectOptions connectOptions, Int64 lastChangeId, CancellationToken cancellationToken, Object userState)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.GetConnectionDataAsync(ConnectOptions connectOptions, Int32 lastChangeId, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.ConnectAsync(ConnectOptions connectOptions, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Agent.LocationServer.ConnectAsync(VssConnection jobConnection)
[2020-06-05 14:38:58Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2020-06-05 14:38:58Z WARN VisualStudioServices] Authentication failed with status code 401.
X-TFS-ProcessId: 64a8285c-ebd1-4480-92ad-56359c286d1d
Strict-Transport-Security: max-age=31536000; includeSubDomains
ActivityId: aa090303-7949-48bb-b573-d6c02a227dec
X-TFS-Session: 2f379482-e9cd-4007-80f6-17ee1643dcaa
X-VSS-E2EID: 2ed17d9f-fcda-470f-807c-cf8c39d0afce
X-TFS-SoapException: %3C%3Fxml%20version%3D%221.0%22%20encoding%3D%22utf-8%22%3F%3E%3Csoap%3AEnvelope%20xmlns%3Asoap%3D%22http%3A%2F%2Fwww.w3.org%2F2003%2F05%2Fsoap-envelope%22%3E%3Csoap%3ABody%3E%3Csoap%3AFault%3E%3Csoap%3ACode%3E%3Csoap%3AValue%3Esoap%3AReceiver%3C%2Fsoap%3AValue%3E%3Csoap%3ASubcode%3E%3Csoap%3AValue%3EUnauthorizedRequestException%3C%2Fsoap%3AValue%3E%3C%2Fsoap%3ASubcode%3E%3C%2Fsoap%3ACode%3E%3Csoap%3AReason%3E%3Csoap%3AText%20xml%3Alang%3D%22en%22%3ETF400813%3A%20Resource%20not%20available%20for%20anonymous%20access.%20Client%20authentication%20required.%3C%2Fsoap%3AText%3E%3C%2Fsoap%3AReason%3E%3C%2Fsoap%3AFault%3E%3C%2Fsoap%3ABody%3E%3C%2Fsoap%3AEnvelope%3E
X-TFS-ServiceError: TF400813%3A%20Resource%20not%20available%20for%20anonymous%20access.%20Client%20authentication%20required.
WWW-Authenticate: Bearer, Basic realm="https://<fully.qualified.servername.was.here>/", Negotiate, NTLM
X-Powered-By: ASP.NET
Access-Control-Allow-Origin: https://<fully.qualified.servername.was.here>
P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
Lfs-Authenticate: NTLM
X-Content-Type-Options: nosniff
Date: Fri, 05 Jun 2020 14:38:56 GMT

[2020-06-05 14:38:58Z WARN VisualStudioServices] Basic issued token provider instance 11898202 requires an interactive prompt which is not allowed by the current settings
[2020-06-05 14:38:58Z ERR  VisualStudioServices] GET request to https://<fully.qualified.servername.was.here>/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 is not authorized. Details: TF400813: Resource not available for anonymous access. Client authentication required.
[2020-06-05 14:38:58Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2020-06-05 14:38:58Z INFO Terminal] WRITE LINE: Failed: Removing agent from the server
[2020-06-05 14:38:58Z ERR  Agent] Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: TF400813: Resource not available for anonymous access. Client authentication required.
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Location.Client.LocationHttpClient.GetConnectionDataAsync(ConnectOptions connectOptions, Int64 lastChangeId, CancellationToken cancellationToken, Object userState)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.GetConnectionDataAsync(ConnectOptions connectOptions, Int32 lastChangeId, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.ConnectAsync(ConnectOptions connectOptions, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Agent.LocationServer.ConnectAsync(VssConnection jobConnection)
   at Microsoft.VisualStudio.Services.Agent.Listener.Configuration.ConfigurationManager.IsHostedServer(String serverUrl, VssCredentials credentials)
   at Microsoft.VisualStudio.Services.Agent.Listener.Configuration.ConfigurationManager.UnconfigureAsync(CommandSettings command)
   at Microsoft.VisualStudio.Services.Agent.Listener.Agent.ExecuteCommand(CommandSettings command)
[2020-06-05 14:38:58Z ERR  Terminal] WRITE ERROR: TF400813: Resource not available for anonymous access. Client authentication required.
[<part_of_service_account_name>_jbrn@<local_servername_was_here> _diag]$ 

#

Thank you for the help.

[ActualReverend]

I found an answer:
You MUST select the PAT to be for "All accessible organizations" in order to be used for removing the agent. If your PAT is only for the collection when built, it will only be able to add it to that pool.

(I figured this would happen a few hours after I posted this, after hours this week trying to find the answer)

I am not going to close this issue at this time, as I don't know if this is a bug, a feature, or just plain bad luck....
I would figure that if a PAT could add something, it should be able to remove it, and not require an "administrative PAT" to remove an agent from a pool.

[zleight1]

I found an answer:
You MUST select the PAT to be for "All accessible organizations" in order to be used for removing the agent. If your PAT is only for the collection when built, it will only be able to add it to that pool.

(I figured this would happen a few hours after I posted this, after hours this week trying to find the answer)

I am not going to close this issue at this time, as I don't know if this is a bug, a feature, or just plain bad luck....
I would figure that if a PAT could add something, it should be able to remove it, and not require an "administrative PAT" to remove an agent from a pool.

This is a really silly issue, it should be able to unregister with the same access it used to register... or at least provide a better message to track this down.

[anatolybolshakov]

Hi @ActualReverend @zleight1 this sounds like a permissions issue on Azure DevOps Server side - could you please create the ticket on https://developercommunity.visualstudio.com/spaces/21/index.html to get right eyes on it?
Let me close this one since it doens't seem to be related to agent itself.