Sdk: <EnableDefaultItems>true</EnableDefaultItems> not configurable

[jzabroski]

Example usage:

<PropertyGroup>
    <EnableDefaultItems>true</EnableDefaultItems>
    <DefaultItems>
      <DefaultItem Type="Compile" Include="**/*.cs" Exclude="**/*.user; **/*.*proj; **/*.sln; **/*.vssscc" />
      <DefaultItem Type="EmbeddedResource" Include="**/*.resx; **/*.sql" Exclude="**/*.user; **/*.*proj; **/*.sln; **/*.vssscc" />
      <DefaultItem Type="None" Include="**/*" Exclude="**/*.user; **/*.*proj; **/*.sln; **/*.vssscc" />
    </DefaultItems>
</PropertyGroup>

https://aka.ms/sdkimplicititems lays out the current behavior of EnableDefaultItems - I find it strange that the only way to auto-include resources if they have a resx extension.

I love the ability to do the following, but it does not allow me to recursively include an entire folder as a certain Build Action (like EmbeddedResource):

<PropertyGroup>
    <EnableDefaultItems>false</EnableDefaultItems>
</PropertyGroup>

I don't know about most developers, but I hate project file merge conflicts, and we have to deal with a lot of them in our SQL projects using FluentMigrator. Our directory structure looks like:

Database.csproj
 \_ Schema
    \_ Functions
    \_ Security
    \_ Stored Procedures
    \_ Triggers
    \_ Types
    \_ Views

Ideally, I want the ability to say "any item in this folder tree is an EmbeddedResource". I realize you can use wildcards like <EmbeddedResource Include="Schema/**/*.sql" /> but the user story doesn't work well - adding new SQL files includes them as Content, etc. Here is a link to a StackOverflow question where someone else opines on the same pain in a different context of RazorTemplate engine: https://stackoverflow.com/questions/27839645/automatically-add-files-as-embedded

[etbyrd]

This issue was transferred to https://github.com/dotnet/sdk/issues/3026

[jzabroski]

@etbyrd What's the point in transferring the issue from project-system to sdk if you are going to close it in both? Was this a mistake?

[etbyrd]

Sorry, that was a mistake! I didn't realize that Github updated the transfer behavior and I commented in the project-system issue before it resolved the transfer I think.

[livarcocc]

@dsplaisted can you comment?

[nguerrera]

The proposal doesn't have valid msbuild syntax.

The closest thing in spirit that I can imagine is:

<ItemGroup>
    <DefaultItem Include="Compile" IncludePattern="**/*.cs" ExcludePattern="**/*.user; **/*.*proj; **/*.sln; **/*.vssscc" />
    <DefaultItem Include="EmbeddedResource" IncludePattern="**/*.resx; **/*.sql" ExcludePattern="**/*.user; **/*.*proj; **/*.sln; **/*.vssscc" />
    <DefaultItem Include="None" IncludePattern="**/*" ExcludePattern="**/*.user; **/*.*proj; **/*.sln; **/*.vssscc" />
</ItemGroup>

But I can't see a way to consume that in evaluation. This was my attempt:

<ItemGroup>
   <EmbeddedResource
      Include="@(DefaultItem->WithMetadataValue('Identity', 'EmbeddedResource')->'%(IncludePattern)')"
      Exclude="@(DefaultItem->WithMetadataValue('Identity', EmbeddedResource')->'%(ExcludePattern)')" 
      />
</ItemGroup>

But this creates an item with literal identity **/*.resx; **/*.sql rather than globbing resx and sql files.

@rainersigwald is there any way to go get metadata from one item into the globbing of other items? I suspect there isn't.

A lower tech approach would be to have properties for each of the default items:

<PropertyGroup>
   <DefaultEmbeddedResourceInclude>**/*.resx;**/*.sql</DefaultEmbeddedResourceInclude>
   <DefaultEmbeddedResourceExclude>**/*.noembed.sql</DefaultEmbeddedResourceExclude>
</PropertyGroup>

etc. for Compile/None

We could put the defaults in props and project could do

<PropertyGroup>
   <DefaultEmbeddedResourceInclude>$(DefaultEmbeddedResourceInclude);**/*.sql</DefaultEmbeddedResourceInclude>
</PropertyGroup>

But this isn't as easy to read as just adding EmbeddedResource includes, which brings me to the problem you identified with that approach:

I realize you can use wildcards like <EmbeddedResource Include="Schema/**/*.sql" /> but the user story doesn't work well - adding new SQL files includes them as Content, etc.

Perhaps the best remedy here would be to fixup globbed None/Content items in targets to exclude anything in EmbeddedResource. IIRC, we did something similar for Page vs ApplicationDefinition. Maybe we can generalize?

[jzabroski]

Sorry for the invalid msbuild syntax - I was just prototyping my thought process, by literally translating/reifying the table in https://aka.ms/sdkimplicititems as a configuration that is relatively not noisy.

Honestly, at the end of the day, I just want simpler branch merges. Another way to achieve the same goal would be to lexicographically sort items in ItemGroup collections. I can't say it's as good as this proposal to make default items configurable, but it would probably improve my code merges by at least 50%. Every single time someone adds a new stored procedure, there is a merge conflict. Upgrading to the .NET SDK project type has eliminated .cs merges for database migrations, which has been another 50% productivity improvement.

Also, it's important to note that the whole user story is not just around MSBuild projects, but around Visual Studio and it's ability to efficiently auto-include files and mark-up their File Properties dialog with the right data. You can't really have one without the other, as things will get messy as developers make edits in Visual Studio.

[dsplaisted]

@jzabroski It sounds like the root of the problem is that you can express the globs you want in the project file, but then there are cases where Visual Studio doesn't respect them. IE this part:

I realize you can use wildcards like but the user story doesn't work well - adding new SQL files includes them as Content, etc

When you say "adding new SQL files includes them as Content", how are you adding the files? Via Add New Item in VS? Or are you adding existing files from other folders, or what?

[nguerrera]

@dsplaisted, @jzabroski included a sample of what wasn't working:

I realize you can use wildcards like <EmbeddedResource Include="Schema/**/*.sql" />

They xml got stripped from my first attempt to quote it. I believe the issue is that this doesn't remove anything from sdk None globs or web sdk Content globs, leading to the unfortunate behaviors in VS (or whatever else consumes None/Content in an arbitrary build).

[jzabroski]

Thanks for being my champion, @nguerrera . Can I ask, as I'm not an MSBuild regular, when you use the word glob, is that short for "global property" or as in "glob of data" (as in a fancy way of saying bulk data). The closest I can find to a definition of the phrase "globbing" is here: https://github.com/Microsoft/VSProjectSystem/blob/master/doc/overview/globbing_behavior.md - but it's a bit circular in that it more explains something about globbing than what is globbing. (In the same article, it uses the acronym CPS which to the best of my ability to infer, seems to mean Common Project System).

I just ask because although I read Vince's Buildmaster book a decade ago, and a book about configuring/extending MSBuild 11 years ago, I really don't know your vocabulary as well as I should. - Hence why I also made the silly mistake about properties while quickly prototyping. I mainly use the same MSBuild tasks for the last 10 years, and copy-paste them from project to project, so I never had to really care until .NET Core came along trying to make my life better (hooray).

[dsplaisted]

I see. I think we see once again that the None glob was probably a bad choice.

However, in that case it sounds like the workaround could be to set the EnableDefaultNoneItems property to false.

The Content glob in the Web SDK isn't quite as bad, it only globs for all files under the wwwroot folder, and then for .json and .config files from the project root.

[dsplaisted]

By globbing we mean using wildcards to specify files, ie: https://en.wikipedia.org/wiki/Glob_%28programming%29

And for SDK projects we have default globs for source files, resource files, etc.

[jzabroski]

Thanks - I wish I could contribute more but I've never benchmarked file watching in an IDE before, so I don't know what solution would be best for Visual Studio to implement on its side of things.

Also,

I see. I think we see once again that the None glob was probably a bad choice.

Without knowing all your thoughts on this, I will just say as a customer I'm very happy with the .cs glob as a default item for .NET SDK project type. It solves about 90% of my scenarios. But the final annoying one is really sql files for database migrations. (I don't use .sqlproj since DACPAC is a black box and has many deployment shortcomings, and I know many engineers who feel the same way/come to the same conclusion.)

I guess lexicographic ordering would be helpful, if it could be done, for PackageReference as well. This would solve problems where two different developers add the same package and then have to merge it later. This would make the 3-way merge a seamless question of "Do I have the latest version of everything?" At the end of the day, that's really what a merge is all about in a Git Flow branching model. Some developers I work with just blindly accept one side of a merge, revert it, and then just go in Visual Studio and add all existing files manually, and then click update PackageReferences. I share this anecdote because I think it helps crystallize for toolsmiths like yourself how end users actually think about this stuff. I have a team of developers in India who are not the next Steve Wozniak who think this way, and I think it's a pretty intelligent way to go about safely merging project files absent anything automated.

[dsplaisted]

I think we see once again that the None glob was probably a bad choice.

This comment was really directed at members of my team, so let me explain it better:

For SDK projects, by default we include all .cs files (assuming it's a .csproj) as Compile items, and all .resx files as EmbeddedResource items. We include everything else as None items. These three are what we mean by "default globs". You can see the code for this here. (Note that Web projects have additional default globs).

We included "everything else" as None items as a way of making all of the files in your project folder show up in the Solution Explorer in Visual Studio. However, this causes issues (which you may have hit). If you do something like <EmbeddedResource Include="Schema/**/*.sql" />, then your .sql files will end up as both EmbeddedResource and None items. Visual Studio expects that each file maps to a single MSBuild item, so it can end up doing weird things in this case. To avoid this, you can set the EnableDefaultNoneItems property to false, or you could also add a Remove for the None items: <None Remove="Schema/**/*.sql" />

So my recommendation would be:

• Set the EnableDefaultNoneItems property to false
• Use globs such as <EmbeddedResource Include="Schema/**/*.sql" /> for items you need to include that aren't covered by the default globs (for .cs and .resx files)
• Let us know if there are still any issues you have

As far as ordering PackageReference items, I would open a separate issue for that discussion (probably in https://github.com/dotnet/project-system). Link to this issue from the new issue, and tag me on it.

[rainersigwald]

@rainersigwald is there any way to go get metadata from one item into the globbing of other items? I suspect there isn't.

Yeah, I haven't been able to come up with anything either.

I like @dsplaisted's recommendations (this will not be surprising to folks that have worked with me on this: I don't like the default globs in general).

[nguerrera]

@jzabroski Does this work for you:

<PropertyGroup>
   <EnableDefaultNoneItems>false</EnableDefaultNoneItems>
</PropertyGroup>
<ItemGroup>
  <EmbeddedResource Include="Schema/**/*.sql" />
</ItemGroup>

[jzabroski]

@nguerrera Yes, that does work and appears to solve the issue with if I right-click on an existing .sql file in the solution, select copy, and hit Control+V for paste, it being added to the csproj as a None item. One question about this - how does it "know" whether something should be globbed or not at the time I copy and paste a file in Visual Studio Solution Explorer? I tried testing to see if it matters at all the order in which I specify things in the csproj, such as putting

  <ItemGroup>
    <EmbeddedResource Include="Schema/**/*.sql" />
  </ItemGroup>

at the top. It doesn't seem to matter, which is great and what I would expect. Sorry if such test cases seem naive and simple, but I'm just trying to make sure whatever solution we come up with truly works.

A different way to ask this question: Is there any documentation on glob conflicts?

[jzabroski]

OK, so, this seems to work perfectly, but as I re-read https://aka.ms/sdkimplicititems , I didn't grok the first couple of times I read this page that the answer was pretty much right in front of my eyes:

In a similar way, you can use <EnableDefaultNoneItems> to disable the implicit None glob.

Now that I've seen how the magic trick works, I'm not sure I can improve the documentation to make it clearer to others, but it definitely flew over my head that this was the trick to make everything work. Another very senior developer I respect also missed this trick. Maybe the docs should say:

In a similar way, you can use <EnableDefaultNoneItems>false</EnableDefaultNoneItems> to disable the implicit None glob.

Again, not sure I would have realized that was the solution even if it was written that way, but it might be one step in the right direction.

[jzabroski]

@nguerrera @dsplaisted I'm good with closing this if you are.

[StingyJack]

Honestly, at the end of the day, I just want simpler branch merges.

This is most easily solved by simply enforcing an ordering for the proj files, as should be done for any code generated and most non-user-only-managed files.

@jzabroski did mention this, and I have echoed the scenarios in other issues (and would like to point out that this is generally still the best way to handle sln file conflicts). That simple solution to so many merge problems (not just Git, either) seems to have been missed by the MSFT participants in this thread. @livarcocc - are you already tracking this? If not I'll open an issue for it.