Msbuild: Wildcard expansion is silently disabled when a wildcard includes a file over MAX_PATH

[rainersigwald]

This was reported as a Connect issue.

The repro is simple:

<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <ItemGroup>
        <MyItem Include="MyDir\**\*.*" Exclude="MyDir\node_modules\**\*.*;MyDir\tmp\**\*.*" />
    </ItemGroup>

    <Target Name="TestInputs">
        <Warning Text="Inputs = @(MyItem)" />
    </Target>
</Project>

Where there are files in MyDir\node_modules that exceed MAX_PATH. That returns:

C:\Users\raines\Downloads>msbuild test.proj
Microsoft (R) Build Engine version 14.0.24720.0
Copyright (C) Microsoft Corporation. All rights reserved.

Build started 12/16/2015 12:46:15 PM.
Project "C:\Users\raines\Downloads\test.proj" on node 1 (default targets).
C:\Users\raines\Downloads\test.proj(8,3): warning : Inputs = MyDir\**\*.*
Done Building Project "C:\Users\raines\Downloads\test.proj" (default targets).

Build succeeded.

Note that the wildcards weren't actually expanded.

The original issue mentions an expectation that the Exclude element would prevent this from happening because that directory and all its contents are excluded. The reason that doesn't work is because we build the Include list fully first, then build the Exclude list, then do a subtraction. The failure occurs when building the Include list, so Exclude has no effect.

[rainersigwald]

There are a couple of questions here.

1. We have (until we figure out #53 for full-framework MSBuild) nice errors that we are supposed to emit when something goes over MAX_PATH. Why didn't they fire here?
2. Should we precompute the impact of the exclude list as this user seems to expect?

For 2, I think the answer is "no". Being able to do pattern-based set exclusions seems like an awful lot of work for fairly minimal gain.

[shift-evgeny]

Thanks for troubleshooting this! You're right about my expectations. At minimum MSBuild should report a meaningful error here, but yes, I do expect that excluding the directory will avoid any problems with it.

"Precomputing the impact of the exclude list" is an implementation detail. From the user point of view, the ItemList is "everything in MyDir, except node_modules" - same as manually listing every other sub-directory in the include list (which works).

Yes, it is a problem with NPM that it builds a directory structure that exceeds MAX_PATH, but I have no control over that. All I can do is exclude the directory.

[rainersigwald]

Thinking about it a bit more, it may not be that difficult to consider the Exclude list inline with Include, but I haven't looked at that corner of the code in a while. If it's easy we should do it.

If it's not easy, I think fixing the long-path behavior would be sufficient to avoid this problem--it's inefficient to add and then remove things from a list but would produce the desired behavior.

[rainersigwald]

@shift-evgeny Exclude should now behave as you expect, thanks to improvements made for lazy item evaluation (#770). MSBuild now considers the exclude list while recursing through the include list and will stop when it hits an excluded directory.

This should already be available in VS "15" Preview 5.

I'm not closing this issue because the bad doesn't-expand-wildcards behavior persists if you _don't_ have an exclude. But at least the obvious workaround for node_modules works now.

[Daniel15]

The built-in EmbeddedResource stuff totally breaks because of this because it uses a wildcard like **/*.resx in %ProgramFiles%\Microsoft Visual Studio\2017\community\msbuild\Sdks\Microsoft.NET.Sdk\build\Microsoft.NET.Sdk.DefaultItems.props:

<EmbeddedResource Include="**/*.resx" Exclude="$(DefaultItemExcludes);$(DefaultExcludesInProjectFolder)" Condition=" '$(EnableDefaultEmbeddedResourceItems)' == 'true' " />

Resulting in the GenerateResource task throwing this error:

Microsoft.Common.CurrentVersion.targets(2867,5): error MSB3552: Resource file "**/*.resx" cannot be found.

https://github.com/dotnet/cli/issues/6561

If it's difficult to properly fix this, could you please at least update the resx stuff in Microsoft.Common.CurrentVersion.targets so it doesn't fail due to this problem?

[RehanSaeed]

Just started getting this error from Travis CI here. My path doesn't even seem that long.

/usr/share/dotnet/sdk/1.0.1/Microsoft.Common.CurrentVersion.targets(2865,5): error MSB3552: Resource file "*/.resx" cannot be found. [/home/travis/build/RehanSaeed/Schema.NET/Source/Schema.NET/Schema.NET.csproj]

I guess my solution is to somehow tell Travis CI to put my code in folder at a lower level?

[Daniel15]

One cause for that error in web apps is node_modules. Older versions of npm
had very deep directory structures. If you're using npm, you should ensure
that it's the latest version rather than an older version like npm2. You
can run "npm install --global npm" to upgrade it.

Sent from my phone.

On Jul 2, 2017 9:43 AM, "Muhammad Rehan Saeed" [e-mail]
wrote:

Just started getting this error from Travis CI here
https://travis-ci.org/RehanSaeed/Schema.NET/jobs/249300823. My path
doesn't even seem that long.

/usr/share/dotnet/sdk/1.0.1/Microsoft.Common.CurrentVersion.targets(2865,5):
error MSB3552: Resource file "*/.resx" cannot be found.
[/home/travis/build/RehanSaeed/Schema.NET/Source/
Schema.NET/Schema.NET.csproj]

I guess my solution is to somehow tell Travis CI to put my code in folder
at a lower level.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/msbuild/issues/406#issuecomment-312502798,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAFnHRWNLx_ydPYoDzxayCOS0jDk8oyIks5sJ8i2gaJpZM4G221q
.

[RehanSaeed]

Not using npm, just a simple class library.

[Daniel15]

Try running find . in your build script (which will recursively list all the files in the directory) and see if there's any very long paths. A long path anywhere in the directory will break the wildcard expansion.

[RehanSaeed]

@Daniel15 Thanks for the suggestion. Just tried that and this is what I found:

The longest path from the project causing the build error is 110 characters:

/home/travis/build/RehanSaeed/Schema.NET/Source/Schema.NET/health-lifesci/MedicalGuidelineContraindication.cs

The longest path from another project in the solution is 118 characters:

/home/travis/build/RehanSaeed/Schema.NET/Source/Schema.NET.Tool/Overrides/AddNumberTypeToMediaObjectHeightAndWidth.cs

These paths are well short of the 260 character max path, unless I'm missing something about Linux or MacOS where I'm running the dotnet build command.

[cellvia]

i am totally blocked by this. I am also on osx and having the same results as @RehanSaeed
looks like msbuild / dotnet core are not actually ready for unix :-/

[cellvia]

@RehanSaeed for what its worth, it seems like some issue with the hidden keys folder thats generated when launching debug session.

to solve this: i created a keys folder (not hidden), and then within that a file with the same filename as the one that had been generated in the hidden .keys folder. i left the file empty.

now every time i build / launch it works fine...

[rashidovich]

VM Ubuntu 17.04
.net core 2.0

dotnet build throws same issue.

basically i cannot deploy to production.
on local machine (OSX) builds fine.

i don't see any '.keys' folder.

Thanks in advance.

[dasjestyr]

Having same issue on both osx and ubuntu. Getting the latest version of npm did not help. Moving the project to root in order to shorten the path also did not work. No keys file either. Won't build locally or as a docker build on osx or ubuntu, but builds fine in a windows environment both locally and as docker build.

[radical]

Can you try running find . -name '*\\*' in your project directory?

[dasjestyr]

@radical not sure if you were talking to me, but if I run it, it just returns a single file which is an NServiceBus log file.

[radical]

@dasjestyr I hit this issue too and have a workaround. The problem is that the wildcard expansion is skipped if there is any error while trying to expand the wildcard. And the \ in the path here gets converted to / on !windows and that causes a failure later. My workaround really just skips such a directory, and so doesn't really solve the issue. https://gist.github.com/radical/27ab744e235badd5f7b24dbfc85ec9af
I will try to look at a proper solution later. But anyone else should feel free to take a stab at it meanwhile :)

[radical]

@dasjestyr um btw, was it a file or a directory for you? My patch might need to be fixed to handle files too. But this is a hack anyway.

[dasjestyr]

@radical It returned: "./DAS.Gateways.Services.Api/C:\Logs\DAS\DAS.Infrastructure\log-{Date}.log"

[radical]

@dasjestyr It should be breaking only for a directory. Could you run$ file ./DAS.Gateways.Services.Api/C:\Logs\DAS\DAS.Infrastructure\log-{Date}.log?

[dasjestyr]

./DAS.Gateways.Services.Api/C:LogsDASDAS.Infrastructurelog-{Date}.log: cannot open `./DAS.Gateways.Services.Api/C:LogsDASDAS.Infrastructurelog-{Date}.log' (No such file or directory)

[radical]

You missed the \es in the path. Just check if C:\Logs\DAS\DAS.Infrastructure\log-{Date}.log is a directory or a file. I just want to confirm that I have the correct issue and we don't need to dig more.

[dasjestyr]

Oh, it's a directory

[olehdutko]

@cellvia , thanks for solution of fixing this issue!
Where I need to create keys folder?

[andersforsgren]

The globbing seems to fail if any part of the traversal fails. In my case the problem was that the project output directory contained an NTFS junction that wasn't traversable, giving the same error as if it was traversable but longer than MAX_PATH. So being over MAX_PATH is not the only problem that gives these symptoms.

This is probably hard to fix for the general case with MAX_PATH, but it would be great if the globbing could report sensible errors, or even revover from some problems (e.g diagnostics that describe it did not glob one subdirectory because access was denied or a path was too long)