Project-system: Code Generation

[MarkPflug]

I currently work on a project where we generate tens of thousands of lines of code using a SingleFileGenerator (CustomTool) in a VSIX. This approach has some drawbacks. First, I would like to not add the generated code to version control, because it just adds a lot of unnecessary noise, and potential merge effort. Managing and versioning the code generator becomes a bit of a hassle, because the code has to be manually regenerated inside VS.

The only other approach to code-gen that I know about is to use an MSBuild task to generate the code. But the drawback is that code generated at build time is not visible at design time via Intellisense. In hopes of improving this situation I opened a suggestion issue against the Roslyn repository, hoping to expand upon the proposed generators feature.

As it turns out, I recently came to discover that I can already get Intellisense for code generated by an MSBuild task. I found this by looking at the CodeGeneration.Roslyn project. AArnott uses a feature I wasn't aware of <Generator>MSBuild:_____</Generator>. I'm aware of the Generator metadata, as SFGs use that as well, but I had never seen this MSBuild prefix, nor can I find it documented anywhere.

After some experimentation, I was able to get the MSBuild generator to work, but it required a bit of a hack.

I had defined:

<ItemGroupDefinition>
<MyItemType>
<Generator>MSBuild:MyGenerator</Generator>
</MyItemType>
</ItemGroupDefinition>

However, my generator wasn't running when I saved my files. After further experimentation I found that if I register my generator against the Compile item group, that it would run when I saved .cs files, which isn't what I want. I want it to run when I modify a .foo file, which is the input to my generator. So, my hack was to add my custom file type to the Compile items to force the generator to run:

<ItemGroupDefinition>
<Compile>
<Generator>MSBuild:MyGenerator</Generator>
</Compile>
<MyItemType>
<Generator>MSBuild:MyGenerator</Generator>
</MyItemType>
</ItemGroupDefinition>

<MyItemType Include="**/*.foo"/>
<Compile Include="@(MyItemType)"/>

Then, after the code is generated, I remove my stuff from the compile items so that it doesn't get sent to CSC, which would fail since it isn't C#.

<Compile Remove="@(MyItemType)"/>

This works, but it feels like a hack. Is there some reason this <Generator>MSBuild: feature doesn't work against custom item types? The fact that this isn't documented also makes me doubt whether it is safe to expect this to work in the future. Is there documentation for this somewhere? Is it safe to expect it to continue working?

Additionally, it would be nice if VS would prevent editing of generated code files. We've had developers on our team modify generated code accidentally after F12-ing into it and not realizing they were in a generated file. When you navigate into symbols in external assemblies you are presented with a readonly "from-metadata" view. It would be nice if generated code was also presented in an obviously readonly way to prevent accidental modification.

My apologies if this isn't the right project for this issue. Hopefully you can direct me to the right one if it isn't.

[Pilchie]

I think this is a fine place to discuss this. The generators should also work for things like .xaml files, so I'm not sure why that didn't work in your case.

I agree that it would be good to investigate and document this process though.

[Pilchie]

Also explicitly tagging @aarnott in case he has any ideas here.

[MarkPflug]

For reference, this is a real world example where I'm using this:
https://github.com/MarkPflug/Elemental.JsonResource

It functions as is, but my concern is that with this hack only one such code generator can operate in a single project build process, since it requires "ownership" of the <Compile><Generator> element to enable intellisense. As such, I wouldn't be able to use my JsonResource and the CodeGeneration.Roslyn package in the same project.

[MarkPflug]

I had a thought. What if MSBuild defined the target, but the default "MSBuildCodeGenerators" target did nothing but provide a target for others to attach to.

<ItemDefinitionGroup>
    <Compile>
        <Generator>MSBuild:MSBuildCodeGenerators</Generator>
    </Compile>
<ItemDefinitionGroup>

Then, anyone who wanted to generate code could simply attach their target before the MSBuildCodeGenerator target:

<Target Name="MyCodeGenerator" BeforeTargets="MSBuildCodeGenerator">
...
</Target>

This would also allow multiple AOP style generators (like CodeGenerator.Roslyn) to define an ordering if necessary.

<Target Name="GenerateCodeFromAttributes" DependsOnTargets="$(GenerateCodeFromAttributesDependsOn)">
</Target>

I could then control the ordering using the "GenerateCodeFromAttributesDependsOn" property, so the generated output of one code generator could be used as input into another code generator. If AOP style code generators become more common it seems like this could become a real issue.

I haven't tested that this actually works in practice, but it seems like it should.

[Pilchie]

Also tagging @andygerlicher here.

[AArnott]

After further experimentation I found that if I register my generator against the Compile item group, that it would run when I saved .cs files, which isn't what I want. I want it to run when I modify a .foo file, which is the input to my generator.

@MarkPflug Which project type are you using here? This git repo tracks the project type that generally has a TargetFramework property defined and an Sdk attribute in the Project tag. Is that the type you're working with?
If so, then what you're probably missing is a .xaml file to describe your custom item type that includes the Generator item metadata so that CPS knows to look for it.

Additionally, it would be nice if VS would prevent editing of generated code files.

That sounds like a good idea.

[MarkPflug]

Thanks for the reply @AArnott. I tried adding the xaml ProjectSchemaDefinitions to my project:

<ProjectSchemaDefinitions 
    xmlns="http://schemas.microsoft.com/build/2009/properties">

    <ContentType
        Name="JsonResource" 
        DisplayName="Json resource file" 
        ItemType="JsonResource" />

    <ItemType 
        Name="JsonResource" 
        DisplayName="Json Resource File">

    </ItemType>

    <FileExtension 
        Name=".resj" 
        ContentType="JsonResource" />

</ProjectSchemaDefinitions>

And referencing it from my .targets file:

    <ItemGroup>
        <PropertyPageSchema Include="$(MSBuildThisFileDirectory)JsonResourceSchema.xaml" />
    </ItemGroup>

However, it still doesn't work. I can see the "Json Resource File" ItemType that I've defined in the "Build Actions" drop down of the properties pane. However, my ".resj" files are not being associated with it, but are using "None" instead. When I choose "Json Resource File" from the dropdown, that file is removed from my project. When I include it again, it is back to using "None".

It still functions when building with MSBuild, but the IDE is confused and doesn't know about the generated code so I don't get intellisense for anything.

[AArnott]

@jviau do you think you can help @MarkPflug with this CPS extensibility, or loop someone in who can?

[jviau]

@MarkPflug , for items being included as <None /> that is because the SDK-style .csproj as a glob that includes all leftover (non .cs, .resx, etc) as None. You will want to include these globs in your targets:

<ItemGroup>
  <None Remove="**/*.resj" />
  <JsonResource Include="**/*.resj" />
</ItemGroup>

For single-file generator help in CPS, @adrianvmsft can help there.

[adrianvmsft]

You may want to take a look at this page that explains how to add a Single File Generator, including the item type rule (however that tutorial uses a classic generator instead of msbuild target).

Project system extensibility uses the a similar approach (via msbuild target) to generate .cs files based on the .xaml rules that added to the project type.

That logic resides inside the Microsoft.VisualStudio.ProjectSystem.SDK.Tools nuget package, so you could take a look to see how it is hooked up in the .targets file so that intellisense picks it up.

https://www.nuget.org/packages/Microsoft.VisualStudio.ProjectSystem.SDK.Tools/

[MarkPflug]

@AArnott, @jviau, @adrianvmsft, thanks for the help! With your combined assistance I was able to get it working. It is pretty compelling when it works: a nuget package that enables a DSL with IDE integration.

Is it possible to have custom item types show up in the Add New Item dialog? What about providing custom icons for the solution explorer?

The only remaining thing was my suggestion to have the IDE prevent modification of generated code. Possibly this could be done by looking for the conventional g.cs file extension, or require specific msbuild metadata be defined on the items.

Thanks again!

[adrianvmsft]

Nice! I am really glad that you got it working!

In order to show the new item type in the Add New Item dialog you need to add a new item template. This page explains how to do that.

I like the idea of having a way to mark generated files as readonly. If you create an item on https://visualstudio.uservoice.com/ that would give the option to other users to vote on it. Alternatively, you could use the Send Feedback option in Visual Studio to log it (main menu -> Help -> Send Feedback).
Meantime, perhaps you could prefix the generated file with a comment to explain it is generated and changes would be lost? That approach is already used by various generators in VS (e.g. resx, .xaml).

Thanks!
Adrian

[jviau]

To clarify how to add custom icons and add new items, this will require a vsix. The general design we approach with extending CPS is to put all build functionality into your nuget package (which you have done), and then any additional design-time features into a vsix, which then turn on when it detects your nuget package installed (done via the nuget package adding capabilities to the project). This makes the vsix entirely optional to users, and not needed for build machines.

For changing icons, you will want to implement a IProjectTreePropertiesProvider

For making generated files readonly, I don't really agree with that. You can't actually make them readonly, because then your code gen will fail, and there isn't too much value in making only VS block the editing of it. If a user wants to edit that code, then they will just use another editor. The best method would be to add comments on the generated file saying it is generated, do not edit, and changes will be overwritten, etc. Also, make sure to put the generated files into $(IntermediateOutputPath) and include them in your targets. This way it does not pollute the users solution explorer, but they can still get intellisense from it and F12 into it if needed.

[MarkPflug]

I saw that page that you linked, but I feel like I'm missing some context. The tutorial starts by saying:

_In the Solution Explorer, Right click on the "Rules" subfolder of your ProjectType project (typically located under [ProjectType]\BuildSystem\Rules)_

I am meant to have an existing project open, but it doesn't say anything about that project type. As a guess, I tried creating a C#/Extensibility/C# Project Template project, and a C# Item Template project, but neither of those appear to have what these instructions require.

Even if I was able to follow the instructions, this would result in producing a vsix? I was hoping to include everything in a nuget pacakge, so I think this would achieve what I'm after anyway.

As far as the readonly view for generated code goes, I created a user voice, if you would like to lend your vote: https://visualstudio.uservoice.com/forums/121579-visual-studio-ide/suggestions/32271405-display-generated-souce-code-files-in-a-readonly-v

Again, thanks for all the help!

[MarkPflug]

@jviau Indeed, the readonly view isn't as much of an issue when you are writing to the IntermediateOutputPath using an MSBuild generator. Since the code will be regenerated every time you do a build and doesn't get added to source.

However, consider a file generated from an ISingleFileGenerator: if you F12 into a symbol in the middle of the file it can be easy to not see the warning comment at the top of the file. Additionally, SFG output files would be included in source control typically, and don't get regenerated at build time, so accidental edits can go unnoticed for some time.

I don't mean to make the file system file readonly, I mean to have VS present it in a view that prevents editing, or at least requiring confirmation before you can edit it, since your edits will likely be lost.

[jviau]

We do have an SDK to assist in creating a new project type, but you only need some additional types added to MEF composition in VS. You do this by creating a new VSIX project, add our reference nuget packages https://www.nuget.org/packages/Microsoft.VisualStudio.ProjectSystem/ and adding your assembly as a MefComponent in the source.extension.manifest. Unfortunately this functionality cannot be delivered via a nuget package, as there is no way to add a dll from a nuget package to the VS MEF composition.

Ah, I was under the impression you were using the msbuild generator here. If you really want to block the editing of those files, CPS does have an extension point for handling commands. You could handle the open command on those files only and open in a read-only editor (which you may have to implement as well).

[davkean]

Going to close this out, it appears to be have resolved.