Azure-pipelines-agent: Builds of shelvesets that include files at paths not in the repository mappings causes build to fail

[ppejovic]

Have you try trouble shooting doc?

Link to trouble shooting doc: https://www.visualstudio.com/en-us/docs/build/troubleshooting
Yes

Agent version and platform

Version of your agent? 2.112.0


OS of the machine running the agent? Windows Server 2012 R2 Standard

VSTS type and version

VisualStudio.com or On-Prem TFS? On-Prem


If On-Prem TFS, which release, 2015 RTM/QU1/QU2? TFS 2017 Update 1

What's not working?

For a build definition that has a repository mapping of $/x/y/z, the build will fail if you attempt to build a shelveset that includes changes of files not under $/x/y/z. The following error causes the build to fail:

No appropriate mapping exists for $/path/to/file/not/in/mappings/file.cs.

E.g. the following log is from a build that has a repository mapping of $/TFS Test/XXXX/ShelvesetRepro/Production. I created a shelveset that includes changes of files under this path _and_ changes to files under a different location at $/TFS Test/XXXX/ShelvesetRepro/Uat (note different leaf folders of Uat vs Production). Attempting to build the shelveset generates this log and fails:

tf vc get /version:416237 /recursive /overwrite D:\B\1\2\s /loginType:OAuth /login:.,******** /noprompt
All files are up to date.
tf vc unshelve "Hello Shelveset;XXXXX\x.xxxxxxx" /loginType:OAuth /login:.,******** /noprompt
No appropriate mapping exists for $/TFS Test/XXXX/ShelvesetRepro/Uat/ConsoleApp1/Program.cs. ConsoleApp1:
Unshelving edit: Program.cs
Exit code 1 returned from process: file name 'tf', arguments 'vc unshelve "Hello Shelveset;XXXXX\x.xxxxxxx" /loginType:OAuth /login:.,******** /noprompt'.

This is a change of behavior from XAML build as the unmapped path would be ignored and the build succeed. With the current behavior of the TF Build agent you are required to map every possible path that could be included in a shelveset (even those not relevant to the build itself), which ultimately leads to creating a single mapping close to the root of the repository. This is not desirable in large repositories.

Agent and Worker's diag log

Agent log for get sources step:

2017-07-19T21:30:18.2337518Z ##[debug]Attempting to find a workspace: 'ws_2_86'
2017-07-19T21:30:18.2337518Z ##[debug]Skipping workspace: 'ws_1_79'
2017-07-19T21:30:18.2337518Z ##[debug]Skipping workspace: 'ws_1_82'
2017-07-19T21:30:18.2337518Z ##[debug]Skipping workspace: 'ws_1_86'
2017-07-19T21:30:18.2337518Z ##[debug]Skipping workspace: 'ws_10_74'
2017-07-19T21:30:18.2337518Z ##[debug]Skipping workspace: 'ws_11_74'
2017-07-19T21:30:18.2337518Z ##[debug]Skipping workspace: 'ws_2_82'
2017-07-19T21:30:18.2337518Z ##[debug]Candidate workspace: 'ws_2_86'
2017-07-19T21:30:18.2337518Z ##[debug]Definition mapping[0]: cloak 'False', recursive 'True', server path '$/TFS Test/XXXX/ShelvesetRepro/Production', local path 'D:\B\1\2\s'
2017-07-19T21:30:18.2337518Z ##[debug]Found mapping[0]: cloak 'False', recursive 'True', server path '$/TFS Test/XXXX/ShelvesetRepro/Production', local path 'D:\B\1\2\s'
2017-07-19T21:30:18.2337518Z ##[debug]Matching workspace found.
2017-07-19T21:30:18.2337518Z ##[debug]tf vc status /workspace:ws_2_86 /recursive /nodetect /format:xml /collection:http://XXXXXX:8080/tfs/Main/ /loginType:OAuth /login:.,******** /noprompt
2017-07-19T21:30:18.7337824Z ##[debug]<Status />
2017-07-19T21:30:18.7650348Z ##[command]tf vc get /version:416237 /recursive /overwrite D:\B\1\2\s /loginType:OAuth /login:.,******** /noprompt
2017-07-19T21:30:19.8744513Z All files are up to date.
2017-07-19T21:30:19.8900791Z ##[command]tf vc unshelve "Hello Shelveset;XXXXXX\X.XXXXX" /loginType:OAuth /login:.,******** /noprompt
2017-07-19T21:30:21.5620307Z No appropriate mapping exists for $/TFS Test/XXXX/ShelvesetRepro/Uat/ConsoleApp1/Program.cs.
2017-07-19T21:30:21.5620307Z ConsoleApp1:
2017-07-19T21:30:21.5620307Z Unshelving edit: Program.cs
2017-07-19T21:30:21.8121044Z ##[error]Exit code 1 returned from process: file name 'tf', arguments 'vc unshelve "Hello Shelveset;XXXXXX\X.XXXXX" /loginType:OAuth /login:.,******** /noprompt'.
2017-07-19T21:30:21.8121044Z ##[debug]Microsoft.VisualStudio.Services.Agent.ProcessExitCodeException: Exit code 1 returned from process: file name 'tf', arguments 'vc unshelve "Hello Shelveset;XXXXXX\X.XXXXX" /loginType:OAuth /login:.,******** /noprompt'.
at Microsoft.VisualStudio.Services.Agent.ProcessInvoker.<ExecuteAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.Agent.Worker.Build.TfsVCCommandManager.<RunCommandAsync>d__26.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.Agent.Worker.Build.TFCommandManager.<UnshelveAsync>d__25.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.Agent.Worker.Build.TfsVCSourceProvider.<GetSourceAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.Agent.Worker.Build.BuildJobExtension.<PrepareAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.Agent.Worker.JobExtensionRunner.<RunAsync>d__30.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.<RunAsync>d__0.MoveNext()
2017-07-19T21:30:21.8276982Z ##[section]Finishing: Get Sources

[ppejovic]

I think this could be solved by passing the sources directory as the itemspec to the unshelve command with the /recursive switch, e.g.:

tf vc unshelve "Hello Shelveset;XXXXX\x.xxxxxxx" D:\B\1\2\s\* /recursive

[ericsciple]

@ppejovic thanks. i wonder if it's the right thing to always do, or whether it should be a setting on the definition

[ericsciple]

I'll discuss with @vijayma to figure out how he wants the user experience

[ppejovic]

Thanks @ericsciple. I'd just like to point you to a uservoice for the same issue. Cheers.

[vijayma]

@ppejovic, I understand that it is not ideal to go to the root of the repo in workspace mappings, especially for large repos. But, with the fine grain control you now have in specifying workspace mappings, you can include $/A/B, $/C, and cloak $/C/D. We rely quite a bit on the workspace mappings being as accurate as possible for computing source versions, commits to be included, etc, and I am trying to see whether to need to deviate from that for these scenarios.

[ppejovic]

Thanks @vijayma. In a variety of the current open issues with workspace mappings, e.g. this one and those for builds with no root repository mapping, I think providing a wildcarded sources directory itemspec (e.g. D:\B\1\2\s\*) and /recursive to the various tf vc commands ensures the command is only operating with the paths defined in the workspace mappings. I'm not aware of any scenarios where this would have an adverse affect? This change doesn't really affect how those mappings are defined by the users.

Specific to the unshelve case, I don't believe there would be any scenarios where we would want the unshelve to fail when there are files outside of the workspace mappings that have been changed?

[ppejovic]

On further testing, whilst the suggested change works for the unshelve scenario it does not result in the desired behavior for all tf vc commands, e.g scorch. See comment here.

[vijayma]

@ppejovic Focusing on the unshelve scenario... if there are files outside of the workspace mapping that have changed, we would not show those commits in the build. The "SourceVersion" information of the build would be incorrect as well. Someone reviewing the build for a list of commits would not get accurate information. Thoughts?

[ppejovic]

@vijayma If I understand your points correctly, my opinion would be that the scope of the included changesets should be defined by the workspace mappings not the contents of the shelveset. For a regular non-shelveset build, files modified outside of the workspace mappings are irrelevant. Can the same be argued for a shelveset? In our environment I can't think of any cases where these outlier files would be relevant as they have no impact on the build output.

[japj]

Just checking since I am looking into migrating to vNext build agent in the very near future.

We currently have a gated XAML build definition for a project that contains both a "Source" and "Documentation" folders. Only the "Source" folder is actually mapped since the doc files are not used in our build.

However, people actually checkin changes to both sources and documentation with a single commit, which means there are files outside the repository mapping in a commit.

From looking at this bug, it seems a checkin will fail in vNext due this "include files at paths not in the repository mappings causes build to fail" issue. Is my understanding of this correct?

[ericsciple]

That's correct. A current workaround is map both folders on the CI machine.

[janniksam]

Any updates on this issue @ericsciple ?
We just migrated from 2013 to 2018, XAML ignores the files while vNext does fail whenever we try to include files that are not mapped, which was already stated above.

[livioos]

What for is workspace mapping if it's NOT used in GetSources step during builds? Are you planning to fix it?

In may opinion It's bad for your product. Please fix it. During Get Sources agent must solve that problem. Agent must get only codes from changeset which belongs to workspace mapping. Otherwise please REMOVE WORKSPACE MAPPING from TFS because it's useless. I fell like it's a big manipulation from MS.

Please tell us what are you planning to do with that. If nothing we will say goodbye.

[dchwartz-optimal]

Any update on repairing this loss of functionality in vNext?

[itsho]

on this matter - is there a way to have one of the following:

• a step that will run before the "get sources" step?
• a better error message ("your shelve contains files outside the scope of this build")

thanks,

[github-actions[bot]]

This issue has had no activity in 180 days. Please comment if it is not actually stale

[itsho]

Yes, I am still waiting. hopefully, I'm not the only one...

[itsho]

Bad bot.