Azure-pipelines-agent: TFS Build Agent failing against TFS 2017 on premise TF400813: Resource not available for anonymous access. Client authentication required.

Created on 18 Jan 2017 · 25Comments · Source: microsoft/azure-pipelines-agent

Agent Version of your agent: 2.105.7
OS of the machine running the agent: Windows
TFS: On Prem, Version 2017

Issue:
The build server has started to fail all builds with the error:
Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: TF400813: Resource not available for anonymous access. Client authentication required.

I upgraded from TFS 2015.3 to TFS 2017, upgraded the build agent, and everything was working including builds. A day later the builds start to fail with the above error. I have been investigating and trying may different things to resolve the issue, but have had no luck. We have tried setting up another build server, and every server we configure it on, we always get the same error. We have gone as far as making our build user an administrator of TFS. The build user has administer permissions for the agent pool. Below is an excerpt of the work diag log, and the http fiddler trace.

Worker's diag log

[2017-01-18 18:00:21Z INFO Worker] Listening for cancel message from the channel.
[2017-01-18 18:00:21Z INFO Worker] Waiting for the job to complete or for a cancel message from the channel.
[2017-01-18 18:00:22Z INFO Worker] Job completed.
[2017-01-18 18:00:22Z ERR  Program] Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: TF400813: Resource not available for anonymous access. Client authentication required.
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.<SendAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.<SendAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.ThrottlingReportHandler.<SendAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__45.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__42`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Microsoft.VisualStudio.Services.Location.Client.LocationHttpClient.<GetConnectionDataAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.<ConnectAsync>d__41.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.JobServer.<ConnectAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Worker.JobRunner.<RunAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Worker.Worker.<RunAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Worker.Program.<MainAsync>d__1.MoveNext()

Fiddler HTTP Trace
I have traced the HTTP Messages
All communication looks good until it attempts a request to a Project Collection api

Build Agent Request

GET https://tfs.*****.com/tfs/Sandbox/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 HTTP/1.1
Connection: Keep-Alive
Accept-Encoding: gzip
Accept-Language: en-US
Authorization: Bearer <Authroization Bearer Code>
Expect: 100-continue
User-Agent: VSServices/15.255.65000.0 (NetStandard) VstsAgentCore-win7-x64/2.105.7 (Microsoft Windows 10.0.14393)
X-TFS-FedAuthRedirect: Suppress
X-TFS-Session: d3e5a5ea-dec4-4c21-b186-b78496d19414
Host: tfs.*****.com

TFS Server Response:

HTTP/1.1 401 Unauthorized
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/10.0
X-TFS-ProcessId: 96021cb3-66d9-48f2-898f-d5412ff210a6
Access-Control-Allow-Origin: *
Access-Control-Max-Age: 3600
Access-Control-Allow-Methods: OPTIONS,GET,POST,PATCH,PUT,DELETE
Access-Control-Expose-Headers: ActivityId,X-TFS-Session,X-MS-ContinuationToken
Access-Control-Allow-Headers: authorization
X-FRAME-OPTIONS: SAMEORIGIN
X-TFS-SoapException: %3c%3fxml+version%3d%221.0%22+encoding%3d%22utf-8%22%3f%3e%3csoap%3aEnvelope+xmlns%3asoap%3d%22http%3a%2f%2fwww.w3.org%2f2003%2f05%2fsoap-envelope%22%3e%3csoap%3aBody%3e%3csoap%3aFault%3e%3csoap%3aCode%3e%3csoap%3aValue%3esoap%3aReceiver%3c%2fsoap%3aValue%3e%3csoap%3aSubcode%3e%3csoap%3aValue%3eUnauthorizedRequestException%3c%2fsoap%3aValue%3e%3c%2fsoap%3aSubcode%3e%3c%2fsoap%3aCode%3e%3csoap%3aReason%3e%3csoap%3aText+xml%3alang%3d%22en%22%3eTF400813%3a+Resource+not+available+for+anonymous+access.+Client+authentication+required.%3c%2fsoap%3aText%3e%3c%2fsoap%3aReason%3e%3c%2fsoap%3aFault%3e%3c%2fsoap%3aBody%3e%3c%2fsoap%3aEnvelope%3e
X-TFS-ServiceError: TF400813%3a+Resource+not+available+for+anonymous+access.+Client+authentication+required.
WWW-Authenticate: Bearer
WWW-Authenticate: Basic realm="https://tfs.*****.com/tfs"
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
X-Powered-By: ASP.NET
P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
Lfs-Authenticate: NTLM
X-Content-Type-Options: nosniff
Date: Wed, 18 Jan 2017 15:03:12 GMT
Content-Length: 19552

Source

slautebach

Most helpful comment

Microsoft got back to us and gave a recommendation to try with no guarantee. It successfuly worked.

It appears that during our upgrade TFS did not register a OAuth token for the server.

The SQL recommend to execute was:
using the FQDN (ie tfs-server.doman.com)

EXEC prc_SetRegistryValue 1,'#\Configuration\OAuth\TrustedIssuers\tfs-sever.domain.com\','Microsoft.TeamFoundation.Framework.Server.OAuth.ClientAuthTokenValidator'

This prevented all users from logging in after a TFS server reboot.
We then tried with just the server name (ie tfs-server)

EXEC prc_SetRegistryValue 1,'#\Configuration\OAuth\TrustedIssuers\tfs-server\','Microsoft.TeamFoundation.Framework.Server.OAuth.ClientAuthTokenValidator'

This than allowed normal user login and the builds to finally work after a server reboot and re-registering all the build agent.

slautebach on 1 Feb 2017

👍2 🎉1

All 25 comments

What do your IIS auth settings look like? Mine are like this:

Anonymous enabled on the website
thumbnail_image002

Anonymous and Windows auth on the webapp
thumbnail_image001

Negotiate and then NTLM providers
And Enable Kernel-mode authentication checked on advanced settings

iisreset may be required after changes.

ericsciple on 18 Jan 2017

oops i got the pictures backwards

ericsciple on 18 Jan 2017

can you send ersciple the worker diag log (at microsoft com). it is under the _diag folder under the agent directory.

do you have a proxy server?

ericsciple on 18 Jan 2017

if you're not using a proxy, then can you send a fiddler trace too. instructions for redirecting the agent traffic to fidder is here

ericsciple on 18 Jan 2017

@slautebach
If you have access to your TFS AT, can you check the event log? there should be an error correspond to the client side error.

TingluoHuang on 23 Jan 2017

We ended up opening a Microsoft Support incident ticket, and they have been looking at the issue. They haven't been able to come up with a solution yet (we sent them the logs, event logs, fiddler traces, etc). I checked the settings as @ericsciple mentioned and they are already already configured as suggested. @TingluoHuang I checked the event log viewer, and found the following error message below.

Log Name:      Microsoft-Team Foundation Server/Debug
Source:        Microsoft-Team Foundation Server
Date:          2017-01-31 9:16:49 AM
Event ID:      0
Task Category: This task has debugging events
Level:         Error
Keywords:      Info Messages
User:          NETWORK SERVICE
Computer:      win2016-tfs.****.com
Description:
TFS ProductTrace Entry
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Team Foundation Server" Guid="{80761876-6844-47D5-8106-F8ED2AA8687B}" />
    <EventID>0</EventID>
    <Version>3</Version>
    <Level>2</Level>
    <Task>1</Task>
    <Opcode>10</Opcode>
    <Keywords>0x8000000000000001</Keywords>
    <TimeCreated SystemTime="2017-01-31T14:16:49.288277300Z" />
    <EventRecordID>196212</EventRecordID>
    <Correlation ActivityID="{4D46F8E4-C464-4FF9-AB3B-3FCBD3D5596C}" />
    <Execution ProcessID="17132" ThreadID="12948" />
    <Channel>Microsoft-Team Foundation Server/Debug</Channel>
    <Computer>win2016-tfs.****.com</Computer>
    <Security UserID="S-1-5-20" />
  </System>
  <UserData>
    <Info TraceId="{00000001-0001-0001-0000-000000000000}  " xmlns="http://schemas.microsoft.com/TeamFoundation/2010/Framework">
      <Tracepoint>5510500</Tracepoint>
      <ServiceHost>{5AEBEE0F-ADC8-4475-AB68-1E6909B5FB45}</ServiceHost>
      <ContextId>3768253</ContextId>
      <ProcessName>w3wp</ProcessName>
      <Username>
      </Username>
      <VSID>{00000000-0000-0000-0000-000000000000}</VSID>
      <Service>
      </Service>
      <Method>
      </Method>
      <Area>Authentication</Area>
      <Layer>Service</Layer>
      <UserAgent>VSServices/15.255.65000.0 (NetStandard) VstsAgentCore-win7-x64/2.105.7 (Microsoft Windows 10.0.14393)</UserAgent>
      <Uri>/tfs/Sandbox/_apis/connectionData?connectOptions=1&amp;lastChangeId=-1&amp;lastChangeId64=-1</Uri>
      <Path>
      </Path>
      <UniqueIdentifier>{C3E50C9D-22B4-4E60-BE0E-CC5675623F41}</UniqueIdentifier>
      <UserDefined>
      </UserDefined>
      <ExceptionType>
      </ExceptionType>
      <Message>Could not find validator for token: {"typ":"JWT","alg":"RS256","x5t":"qRATbduhB09zX3CyRha4WCWNGkQ"}.{"nameid":"c92dd17d-a981-4959-98b6-8d21f06df5a2","scp":"app_token","aui":"7b710508-7fe3-42d9-8b29-c28cd48b292e","sid":"94a5fd78-7406-44c0-8dfb-90831b2dc395","iss":"win2016-tfs.****.com","aud":"win2016-tfs.****.com","nbf":1485872206,"exp":1485876106}</Message>
    </Info>
  </UserData>
</Event>

slautebach on 31 Jan 2017

Microsoft got back to us and gave a recommendation to try with no guarantee. It successfuly worked.

It appears that during our upgrade TFS did not register a OAuth token for the server.

The SQL recommend to execute was:
using the FQDN (ie tfs-server.doman.com)

EXEC prc_SetRegistryValue 1,'#\Configuration\OAuth\TrustedIssuers\tfs-sever.domain.com\','Microsoft.TeamFoundation.Framework.Server.OAuth.ClientAuthTokenValidator'

This prevented all users from logging in after a TFS server reboot.
We then tried with just the server name (ie tfs-server)

EXEC prc_SetRegistryValue 1,'#\Configuration\OAuth\TrustedIssuers\tfs-server\','Microsoft.TeamFoundation.Framework.Server.OAuth.ClientAuthTokenValidator'

This than allowed normal user login and the builds to finally work after a server reboot and re-registering all the build agent.

slautebach on 1 Feb 2017

👍2 🎉1

@slautebach cool, customer support reach us, and we provide the sql script, glad you unblock and thanks for reporting this issue. :)

TingluoHuang on 1 Feb 2017

I had the same issue and I took a leap of faith and ran the stored procedure described above with the name of my load balanced end point. It worked.

I had to reboot the TFS application servers, but there was no need to re-register the build agents in my case.

Thanks

Jared

jaredfholgate on 18 Feb 2017

Does the just released update1 solve this issue?

alaiaedu on 8 Mar 2017

@alaiaedu i don't think Tfs2017 update1 has any fix for this particularly issue. since the root cause is a configuration issue.

TingluoHuang on 9 Mar 2017

Thank you slautebach and jaredfholgate! I had just updated from TFS2010 to TFS2017, and was experiencing this problem. I ran the stored proc, then tfsservicecontrol quiesce/unquiesce to restart the application tier. Now my agents are talking to the system. I did not need to reinstall the agents.

jamesmbristow on 25 Oct 2017

@slautebach was there a check to confirm TFS did not register an OAuth token?

slandsaw on 11 Dec 2017

TFS 2018 contains a fix to address this problem.

TingluoHuang on 11 Dec 2017

I am seeing this issue on my TFS 2017 server. I have tried running agent on a Windows 10 workstation and from a Windows Server 2012 R2 box both with a designated service account and with the TFS admin account. My XAML builds are working fine and I can't risk moving to 2018 and having no builds. I tried the stored procedure, but it didn't help. If I run the stored procedure with the server's FQDN, I get “Warning! The maximum key length for a clustered index is 900 bytes. The index 'PK_blah_blah' has maximum length of 1044 bytes. For some combination of large values, the insert/update operation will fail.”.
We had everything working fine until I had to rebuild the server on new hardware. Same computer name, but slightly different hardware. I'm using the same certificate.

ttolley on 12 Jan 2018

The warning from "EXEC prc_SetRegistryValue" is expected. That's just SQL being SQL.

davekenn on 12 Jan 2018

Setting it to the FQDN and rebooting didn't seem to help either. Everything works fine in a browser for the account, but I can't get this Agent to work since I rebuilt the server.

ttolley on 12 Jan 2018

@ttolley can you verify it matches the values in the server url configured in the admin console?

ericsciple on 9 Feb 2018

Yes, the URL that I use in the stored procedure matches the admin console public URL. It's also the same URL I use when trying to run config.cmd. If I try a different account that shouldn't have permissions, it connects enough to know that that account does not have permissions to configure the agent. The users can connect via visual studio and in the browser. The user that I'm trying to configure the build server with can connect via browser and visual studio. XAML builds still work. The only thing that appears to be broken is the build agent.

ttolley on 12 Feb 2018

@ttolley can you confirm you ran the SQL command against the config db and not the collection db?

ericsciple on 14 Feb 2018

I'm was using TFVS 2018 with the same problem.
this script solved my problem.

EXEC prc_SetRegistryValue 1,'#\Configuration\OAuth\TrustedIssuers\tfs-server\','Microsoft.TeamFoundation.Framework.Server.OAuth.ClientAuthTokenValidator'

janmohammadi on 27 Mar 2018

@janmohammadi - it would be interesting to know the sequence of events for a repro. We know the fix, thought we understood the repo (scenarios where external url changes) but then couldn't repro. Was this an AT move, collection upgrade via attach, server name change, AT(s) behind a new VIP/NLB, etc....?? We had a fix ready to automated running that sql command but then couldn't repro ...

bryanmacfarlane on 28 Mar 2018

Hi @bryanmacfarlane ,
we are experiencing the same issue.
What we've done:

Installation of TFS 2018
Use of TFS with internal IP (fqdn of vm hosting tfs)
Changing external url to a public url
using tfs with public url only (Firewall/NAT)
installation/configuring tfs build agent using internal name and ip without NAT (inside our closed environment)
configuring test build using public url
run test build on worker -> crashes with error description above

any recommondations?

in our dev -environment (without confuriguring and using public url), builds work fine

LukasK07 on 25 Apr 2018

Yes, the URL that I use in the stored procedure matches the admin console public URL. It's also the same URL I use when trying to run config.cmd. If I try a different account that shouldn't have permissions, it connects enough to know that that account does not have permissions to configure the agent. The users can connect via visual studio and in the browser. The user that I'm trying to configure the build server with can connect via browser and visual studio. XAML builds still work. The only thing that appears to be broken is the build agent.

Hi @ttolley

Did you mange to resolve this issue. We are experieneing the exact same problem and the suggested fix does not help. Our agents cannot connect to our TFS instance. Users can connect without issues.

scottunik on 2 Jan 2019

Yes, I did. It was very weird and what I found probably will not help you. We had enabled "Client Certificate Mapping With Active Directory" in IIS. Uninstalling that role from the server fixed my issue. But, this was the second time I ran into this. The first time, I migrated to a new domain and TFS healed itself. It's all very strange and frustrating.

The build agent is extremely finicky and it's making me (as the administrator) strongly recommend that we consider using an alternate source control project. I'm tired of troubleshooting this every time a security change is made on my network. I never had this kind of trouble with the build server pre the agents.

ttolley on 8 Jan 2019

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Exported YAML not valid - "Mapping values are not allowed in this context"

simonvane · 3Comments

Installing Visual Studio Test Agent 2017 removes MSBuild capability

CodeCasterNL · 4Comments

Git checkout submodule in YAML

riezebosch · 4Comments

fatal: could not read Username for 'https://github.com': terminal prompts disabled

muthurathinam · 3Comments

Exported YAML not valid - "Unexpected value 'condition'"

simonvane · 4Comments