Sdk: Implicit globbing breaks ability to do any code-gen before compile.

[fearthecowboy]

Steps to reproduce

Have a tool that generates .cs files, and have it run in the .csproj file before the Compile Step.

Expected behavior

This used to work back around build 4275. (before the implicit globbing)
While I can see why automatically including **/*.cs seems like a really good idea, it breaks any ability to support code generation in msbuild, since the globbing happens before targets run.

Actual behavior

If the Target that does the .cs files generation generates .cs files that were not present before the implicit globbing happens, they aren't picked up by the Compile step.

If I have my own globbing in there too, it complains about finding duplicate items.

Environment data

dotnet --info output:

.NET Command Line Tools (1.0.0-rc4-004527)

Product Information:
 Version:            1.0.0-rc4-004527
 Commit SHA-1 hash:  49c2a4b6b6

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.15007
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\1.0.0-rc4-004527

I had a workaround where I removed the explicit **/*.cs in my original .csproj file, and used an InitialTargets to

<Project ToolsVersion="4.0" InitialTargets="EnumerateInputs" >
  <!-- the codegen tool -->
  <ItemGroup><DotNetCliToolReference Include="dotnet-codegentool" Version="1.0.0" /></ItemGroup>

  <!-- this collects the 'Compile' items *after* the code gen step -->
 <Target Name="EnumerateInputs" >
   <ItemGroup Condition="$(DesignTimeBuild) != true">
    <Compile Include="**\*.cs" Exclude="obj\**" />
    <EmbeddedResource Include="**\*.resx" />
    <EmbeddedResource Include="compiler\resources\**\*" />
   </ItemGroup>
  </Target>

  <Target Name="RunCodeGenTool" BeforeTargets="EnumerateInputs" >
    <Message Text="Running CodeGenTool to generate .cs files [$(MSBuildProjectName)]" Importance="high" />
    <Exec Condition="$(SolutionDir) != ''" Command="dotnet codegentool $(MSBuildProjectDirectory) $(RootNamespace)" EchoOff="true" WorkingDirectory="$(MSBuildProjectDirectory)" IgnoreExitCode="true" /> 
  </Target>

</Project>  

At first, I thought that this was just some weird regression, and rolled back to an earlier dotnet sdk. After trying again this morning, I realize it's a sinister plot to destroy all that is good and holy.

If you want the SDK to do select the files, then please stick the ItemGroup into a Target, and set the BeforeTargets to Build or something...

Either that, or merge duplicate item sets.

[dasMulli]

There is a property <EnableDefaultCompileItems>False</EnableDefaultCompileItems> that disables the implicit compile items. see here

[fearthecowboy]

The other thing I noticed is that opening the project up in Visual Studio 2017, you don't see any .cs files in the solution explorer.

[TheRealPiotrP]

@fearthecowboy it looks like the original issue is unblocked. Can we close the issue?

The VS issue is fixed in recent builds.

[TheRealPiotrP]

@srivatsn @dsplaisted to share the implicit reference feedback.

[dsplaisted]

@fearthecowboy Before implicit globbing, did the project file have something like <Compile Include="**\*.cs" />, or did it list the files explicitly? If it used a glob, then I would expect the same behavior whether the glob was explicitly in the project file or implicit in the SDK. In either case the compile items would be evaluated before any targets are run.

Generally I would expect code-gen targets to put the output of codegen'd files in the intermediate output folder, and use a Compile element in an ItemGroup to only add the files that were code-gen'd in the target. Since they're in the obj folder they won't be included in the default globs (whether explicit or implicit), won't show up as items to add to source control, etc.

It's also expected that items added during a target don't show up in solution explorer. If you do want them to show up, then you could include them via evaluation, add them again in the target that codegens them, and then use the RemoveDuplicates task to remove the duplicates. That should ensure that if the files exist, they show up in solution explorer, while if they don't exist when the build starts, they get added as Compile items before the compile target runs.

FYI, we've added a better error message for the case where you are getting duplicate items because of the implicit includes, which will tell you about the property you can use to turn them off, as well as have a link for more information. So I think the experience will be improved for other people who run into duplicate item errors.

[fearthecowboy]

Yes, @dsplaisted it did have the <Compile Include"**/*.cs" /> previously. Initially removing that removed the conflict. (but when I removed the explicit one in my .csproj , visual studio stopped showing me source files)

Generally I would expect code-gen targets to put the output of codegen'd files in the intermediate output folder, and use a Compile element in an ItemGroup to only add the files that were code-gen'd in the target. Since they're in the obj folder they won't be included in the default globs (whether explicit or implicit), won't show up as items to add to source control, etc.

Generally, I'd agree with you. Unfortunately, I inherited the aforementioned tool of the devil incarnate, and it's not built with msbuild in mind-given a folder, it recursively generates .cs files from it's source files and sticks the generated files alongside the source files.

I'd have to do more work to fix the tool to work per-file (instead of recursively on a folder) and then build a .targets file to process them (and then, what, put the generated files into the /obj/ folder somewhere and add them to <Compile> That's probably not the end of the world; I will see if that's doable without me sacrificing a human in the name of the dark lord.

@piotrpMSFT -- don't close this quite yet; gimme a day or two to ensure that I'm not blocked elsewhere. (I ran into a case with this where my xuint tests were getting duplicate files, regardless of the property being set, but I didn't finish digging into that)

[blackdwarf]

@fearthecowboy did you manage to do more investigations?

[fearthecowboy]

I've gone the other way with this and fixed the generator problem.