Source-build: Consider adding Humanizer to source-build

[crummel]

Roslyn uses Humanizer to pluralize and convert to/from CamelCase for human-friendly automatic variable names and debug output. This was removed in https://github.com/dotnet/source-build/pull/1766 to remove the prebuilt. We should either work with Roslyn to eliminate this dependency or add the Humanizer repo to source-build.

[dagood]

For 5.0 GA, https://github.com/dotnet/source-build/pull/1845 removes the patch that had removed the Humanizer prebuilt. Humanizer got a lot more usage in GA and it wasn't reasonable to try to reapply it all while trying to get GA through. We'll either need to recreate the patch or add the repo. /cc @dseefeld

The repo looks like a normal-enough csproj... I think we should try that. I'm wondering if some of the new usages might be more impactful than the old ones, and would make more of a feature gap if we tried to patch them out manually.

[dagood]

@omajid @tmds I think adding this repo might be a reasonable thing for one of you to pick up if you can. (Shouldn't involve as much delving into binlogs and such.)

[omajid]

Humanizer is also used by roslyn-analyzers

[omajid]

The Microsoft SDK doesn't include Humanzier, does it?

$ tar tf dotnet-sdk-5.0.100-rc.2.20479.15-linux-x64.tar.gz | grep -i human
<empty>

Is this shipped with the SDK?

[dagood]

Huh... maybe none of the projects (as in csproj) that build using Humanizer actually ship in the .NET SDK? Considering we had to actually patch .cs files that contain using Humanizer; that's the only thing that makes sense to me right now.

If we can disable unused projects, that'd be ideal. If for some reason that doesn't work yet we do know the projects aren't used, we could put Humanizer in SBRP.

@crummel, do you know more about this, since you implemented the first patch?
@dseefeld, any insight on how this has potentially changed in GA?

The last possibility I can think of is Humanizer being used at build-time but not in the SDK, somehow. I don't have any evidence pointing to it. If we disable the projects or put Humanizer as a ref package, it seems more likely that we'll see a build break rather than something subtle (a string breaking somewhere in the SDK), so I don't think this is a big concern.

[crummel]

I'm pretty sure Microsoft.CodeAnalysis.CSharp.Workspaces should at least be shipping, the other code style projects maybe not. I'm taking a look now.

[omajid]

I tried building roslyn after removing the using Humanizer references and get:

/home/omajid/devel/dotnet/roslyn/src/Workspaces/SharedUtilitiesAndExtensions/Compiler/CSharp/Extensions/SemanticModelExtensions.cs(236,80): error CS1061: 'string' does not contain a definition for 'Pluralize' and no accessible extension method 'Pluralize' accepting a first argument of type 'string' could be found (are you missing a using directive or an assembly reference?) [/home/omajid/devel/dotnet/roslyn/src/Workspaces/CSharp/Portable/Microsoft.CodeAnalysis.CSharp.Workspaces.csproj]
/home/omajid/devel/dotnet/roslyn/src/Workspaces/SharedUtilitiesAndExtensions/Compiler/CSharp/Extensions/SemanticModelExtensions.cs(262,46): error CS1061: 'string' does not contain a definition for 'Pluralize' and no accessible extension method 'Pluralize' accepting a first argument of type 'string' could be found (are you missing a using directive or an assembly reference?) [/home/omajid/devel/dotnet/roslyn/src/Workspaces/CSharp/Portable/Microsoft.CodeAnalysis.CSharp.Workspaces.csproj]
/home/omajid/devel/dotnet/roslyn/src/Workspaces/SharedUtilitiesAndExtensions/Compiler/CSharp/Extensions/SemanticModelExtensions.cs(236,80): error CS1061: 'string' does not contain a definition for 'Pluralize' and no accessible extension method 'Pluralize' accepting a first argument of type 'string' could be found (are you missing a using directive or an assembly reference?) [/home/omajid/devel/dotnet/roslyn/src/CodeStyle/CSharp/Analyzers/Microsoft.CodeAnalysis.CSharp.CodeStyle.csproj]
/home/omajid/devel/dotnet/roslyn/src/Workspaces/SharedUtilitiesAndExtensions/Compiler/CSharp/Extensions/SemanticModelExtensions.cs(262,46): error CS1061: 'string' does not contain a definition for 'Pluralize' and no accessible extension method 'Pluralize' accepting a first argument of type 'string' could be found (are you missing a using directive or an assembly reference?) [/home/omajid/devel/dotnet/roslyn/src/CodeStyle/CSharp/Analyzers/Microsoft.CodeAnalysis.CSharp.CodeStyle.csproj]
/home/omajid/devel/dotnet/roslyn/src/Workspaces/SharedUtilitiesAndExtensions/Compiler/CSharp/Extensions/SemanticModelExtensions.cs(236,80): error CS1061: 'string' does not contain a definition for 'Pluralize' and no accessible extension method 'Pluralize' accepting a first argument of type 'string' could be found (are you missing a using directive or an assembly reference?) [/home/omajid/devel/dotnet/roslyn/src/Workspaces/CSharp/Portable/Microsoft.CodeAnalysis.CSharp.Workspaces.csproj]
/home/omajid/devel/dotnet/roslyn/src/Workspaces/SharedUtilitiesAndExtensions/Compiler/CSharp/Extensions/SemanticModelExtensions.cs(262,46): error CS1061: 'string' does not contain a definition for 'Pluralize' and no accessible extension method 'Pluralize' accepting a first argument of type 'string' could be found (are you missing a using directive or an assembly reference?) [/home/omajid/devel/dotnet/roslyn/src/Workspaces/CSharp/Portable/Microsoft.CodeAnalysis.CSharp.Workspaces.csproj]
0 Warning(s)

The RC2 SDK includes Microsoft.CodeAnalysis.CSharp.CodeStyle.dll, one of the projects that are erroring out:

$ tar tf dotnet-sdk-5.0.100-rc.2.20479.15-linux-x64.tar.gz | grep -i Microsoft.CodeAnalysis.CSharp.CodeStyle.dll
./sdk/5.0.100-rc.2.20479.15/Sdks/Microsoft.NET.Sdk/codestyle/cs/Microsoft.CodeAnalysis.CSharp.CodeStyle.dll

[dagood]

It looks like Roslyn removed the references from a lot of projects except Microsoft.CodeAnalysis.CSharp.Workspaces: https://github.com/dotnet/roslyn/pull/47297, and there are some (unfulfilled?) plans to add a Humanizer reference to dotnet/sdk. Our dotnet/roslyn commit seems to be in a weird spot: after Humanizer refs were added, but in the wrong places.

Microsoft.CodeAnalysis.CSharp.Workspaces should at least be shipping, the other code style projects maybe not.

I'm seeing kind of the opposite in terms of the DLLs being in the SDK. Nothing for find . -iname '*workspaces*' in the SDK, but Microsoft.CodeAnalysis.CSharp.CodeStyle.dll is where @omajid found it.

Which would be a good thing for us. Maybe Microsoft.CodeAnalysis.CSharp.Workspaces is VS-specific and we don't care about all this?

[dagood]

@333fred, can you help with this question for 5.0 roslyn source-buildability? Is Microsoft.CodeAnalysis.CSharp.Workspaces shipped in the SDK, and if not and we stop building it in source-build, will the Humanizer prebuilt dependency disappear?

[333fred]

@333fred, can you help with this question for 5.0 roslyn source-buildability? Is Microsoft.CodeAnalysis.CSharp.Workspaces shipped in the SDK, and if not and we stop building it in source-build, will the Humanizer prebuilt dependency disappear?

Yes, it is. In fact, https://github.com/dotnet/roslyn/pull/48999 just added the CodeStyle layer because that is shipped in source-build, and that has an explicit dependency on the workspaces layer.

[dagood]

@333fred Interesting, when/where do you expect Humanizer.dll to show up in the .NET SDK? 5.0 GA, 5.0.1? It seems odd that we haven't been able to find it anywhere. What can we compare source-build against to make sure we have this right?

[tmds]

@333fred functionally roslyn doesn't need Humanizer, right? Would we want to consider removing it from roslyn all together for .NET 6?

[333fred]

Interesting, when/where do you expect Humanizer.dll to show up in the .NET SDK?

I'm not sure; the question was whether the Workspaces layer is in source-build, and that should be true.

functionally roslyn doesn't need Humanizer, right?

The compiler does not, but many of our analyzers do. For example, @dagood you mentioned the other day that you were trying to get the roslyn-analyzers repo into source-build, and that has a dependency on humanizer as well: https://github.com/dotnet/roslyn-analyzers/blob/79c2638f54b9e7ef0a3954cc838ec4c13ea63c1f/src/NetAnalyzers/Core/Microsoft.CodeQuality.Analyzers/Microsoft.CodeQuality.Analyzers.csproj#L15, which it uses for CA1714. It's possible that the IDE could move the humanizer dependency out of the Workspaces layer (I couldn't find a reference to the extension methods actually in the workspaces layer, mostly in the features layer), but I'm not sure whether that would actually get rid of the dependency given that roslyn-analyzers ship in-box in .NET 5 and we'd like to be able to source-build them as well.

[dagood]

Interesting, when/where do you expect Humanizer.dll to show up in the .NET SDK?

I'm not sure; the question was whether the Workspaces layer is in source-build, and that should be true.

That's not what I meant by my question--when I say the SDK here I mean the .NET SDK:

Is Microsoft.CodeAnalysis.CSharp.Workspaces shipped in the SDK

(Even if something's built in source-build, that doesn't necessarily mean it's shipped in a .NET SDK, source-built or not. For example, we build dotnet/arcade.)

@jaredpar you're also listed as a Roslyn champ, maybe you can help figure this out?

[jaredpar]

@dagood what question do we need resolved here?

[dagood]

Two questions:

Is Microsoft.CodeAnalysis.CSharp.Workspaces shipped in the SDK?

When/where do you expect Humanizer.dll to show up in the .NET SDK?

[333fred]

@dagood before that though, I do think it's important to address this question:

It's possible that the IDE could move the humanizer dependency out of the Workspaces layer (I couldn't find a reference to the extension methods actually in the workspaces layer, mostly in the features layer), but I'm not sure whether that would actually get rid of the dependency given that roslyn-analyzers ship in-box in .NET 5 and we'd like to be able to source-build them as well.

I can't exclude workspaces from source-build, particularly as we just added codestyle to source-build. And I don't know whether moving the Humanizer dependency from the workspaces layer to a higher layer would help, because other analyzers depend on humanizer and since they ship in-box with .NET 5, I expect they're in the SDK. Looking at the SDK, I see the analyzers shipped in-box are Microsoft.CodeAnalysis.(CSharp|VisualBasic).NetAnalyzers. Cracking them open in ILSpy, I see that they also reference Workspaces.

Is Microsoft.CodeAnalysis.CSharp.Workspaces shipped in the SDK?

So, Microsoft.CodeAnalysis.CodeStyle.dll is shipped in the SDK: 5.0.100-rc.2.20479.15/Sdks/Microsoft.NET.Sdk/codestyle/cs/Microsoft.CodeAnalysis.CodeStyle.dll. Somehow, the Workspaces dll is _not_ currently shipped in the SDK, which is very odd to me as both CodeStyle and the in-box NetAnalyzers have an explicit dependency on Workspaces.

When/where do you expect Humanizer.dll to show up in the .NET SDK?

Well, I'm surprised Workspaces isn't showing up in the SDK. And I'm therefore surprised that the dependencies of Workspaces aren't showing up in the SDK. I _think_ moving the Humanizer dlls to the features layer would be possible: I think all the functionality that depends on it is actually in the features layer, not the workspaces layer.

[omajid]

I have a question that's mostly unrelated to the issue at hand. I see terms like "workspaces" and "codestyle", and I dont know what they mean. Is there some document I can read up on that clarifies what these components are and which of these are shipped (or should be shipped) as part of the .NET SDK?

[333fred]

I see terms like "workspaces" and "codestyle", and I dont know what they mean.

These are roslyn feature layers. There's no real document that I know of, but here's the basics:

For every layer, there is a non-language-specific core, and 2 language specific implementations.

Layer 1: The actual compiler dlls. These are responsible for actually compiling code, and exposing APIs for analyzers to use.
Layer 1.5: csc and vbc. These are command-line executables that run the compiler. They have no upwards dependencies.
Layer 2: Workspaces. This layer provides abstractions that make it easier for analyzers to manage project state.
Layer 3: Features. This layer provides IDE features that are not tied to a specific editor. For example, fetching a list of completion items.
Layer 4: EditorFeatures. This layer provides IDE features that are tied to a specific editor. For example, the VS navbar services.
Layer 5: I know there's more IDE layers here, but I'm not actually sure what comes next 😅

The CodeStyle project is a set of analyzers that have a dependency on Layer 2 here, and they ship in the SDK. The .NET 5 in-box analyzers also have a dependency on layer 2.

[dagood]

Thanks, a lot of answers. 😄 I don't have any objections to adding Humanizer to source-build and it seemed like a reasonable dependency--just takes some work since it's not on the usual infra. (Maybe since it's now in .NET Foundation we will be able to work with the maintainers in the future to add arcade + arcade-powered source-build? 🤞) Removing it is interesting, but it sounds like we're beyond being able to do that for 5.0 without losing features vs. the Microsoft-built SDK. (Which is my focus.)

My main concern here is how weird and confusing it is, how some DLLs aren't showing up where we expect. I wonder if this rises to the level of something to investigate with the Microsoft-built .NET SDK build as well? For now I guess we can assume that as long as the source-built .NET SDK has similar files, we'll at least have parity.

On that note... here's a link to the current 5.0 SDK we're targeting for source-build 5.0 GA https://github.com/dotnet/source-build/pull/1845:
https://dotnetcli.blob.core.windows.net/dotnet/Sdk/5.0.100-rtm.20526.5/dotnet-sdk-5.0.100-linux-x64.tar.gz

[omajid]

Circling back since roslyn-analyzers was recently added: it looks like only one part of roslyn-analzyers needs Humanizer. And that part of roslyn-analzyers is not used by source-build.

[dagood]

We still need it for Roslyn though, right? @omajid are you still planning to work on this?

[omajid]

We still need it for Roslyn though, right?

I am still a little confused as to why it's not shipped as part of the Microsoft SDK ...

@omajid are you still planning to work on this?

If we (still) need it in source-build, sure, I can look into this.

[dagood]

When/where do you expect Humanizer.dll to show up in the .NET SDK?

Well, I'm surprised Workspaces isn't showing up in the SDK. And I'm therefore surprised that the dependencies of Workspaces aren't showing up in the SDK.

@333fred have you looked into this?

I am still a little confused as to why it's not shipped as part of the Microsoft SDK ...

We could throw it in SBRP if it's only compiled against and never executed, I guess. Since it's not shipping in the SDK that seems possible.

We weren't able to patch out usages in the 5.0.0 GA PR (https://github.com/dotnet/source-build/issues/1767#issuecomment-719730564) so it doesn't seem worthwhile to try that approach again.

[omajid]

Fixed with https://github.com/dotnet/source-build/pull/1895