Home: Enable EmbedInteropTypes for PackageReference (matching Packages.Config capability)

[AArnott]

In the packages.config world, when references are actually added to the user's project file, the project system will add <EmbedInteropTypes>true</EmbedInteropTypes> metadata to the reference when its sniff-test tells it that the assembly should be linked rather than referenced. This is an important distinction for several reasons.

But in the project.json world, this is lost -- the project system has no say in the matter, and the build system seems to reference everything even when it should be linked. This will break scenarios when Visual Studio SDK extensions use NuGet packages for such references as Microsoft.VisualStudio.Imaging.Interop.14.0.DesignTime as the types won't be embedded, and although the extension might work on that developer's machine (who has the VS SDK installed), the extension will fail at runtime on the end user's machine because that assembly isn't present. It was supposed to be embedded into the extension author's assembly!

I discussed this in a design meeting with @davidfowl and @jasonmalinowski in mid-September, 2015. But I haven't heard that anything came of it afterward. The design IIRC was that the NuPkg should support a new link folder next to lib and ref. The assemblies in the link folder get passed to the compiler with /link: instead of /ref:.

So a package with this file layout:

MyPackage.nuspec
lib\
   net45\
      MyRegularAssembly.dll
link\
   net45\
      MyContracts.EmbedME.dll

When consumed by a C# project would produce a command line like this:

csc.exe /ref:MyPackage\1.0.0\lib\net45\MyRegularAssembly.dll /link:MyPackage\1.0.0\link\net45\MyContracts.EmbedME.dll

Design Spec - https://github.com/NuGet/Home/wiki/Embed-Interop-Types-with-PackageReference

[yishaigalatzer]

I think as a workaround, the vs packages can use msbuild targets for now. I'm moving this out.

[AArnott]

Yes, as a workaround I've published http://nuget.org/packages/microsoft.visualstudio.sdk.embedinteroptypes

[jaredpar]

@AArnott how would a consumer of a interop package differentiate if they wanted to link or not? Today I can /reference:EnvDTE or /link:EnvDTE. I would assume EnvDTE comes in a single package (maybe that's wrong). As a consumer though how would I control the behavior?

[AArnott]

@jaredpar You shouldn't. You should _only_ reference EnvDTE. It isn't an embeddable assembly because (IIRC) it defines constants. Yes, the compiler lets you embed it, till you happen to reference some of those non-embeddable types and then the compiler screams at you and you're forced to reference it instead of link it. And as the CLR and compiler work best when _everyone_ either links or references in unison, everyone should either link an assembly or everyone should reference it. There should not be a mix.

I think EnvDTE10 was the first truly embeddable assembly in that family.

[jaredpar]

@AArnott I think you're missing my point. Ignore the policy and compat issues around EnvDTE.

The core issue is that there are interop assemblies in existence today that some developers link and others reference. How can I distinguish between those two cases today in project.json?

And as the CLR and compiler work best when everyone either links or references in unison, everyone should either link an assembly or everyone should reference it. There should not be a mix.

That's a nice policy. Reality though is developers do bad things and we have to support that scenario. Not supporting mixing embed and not embed won't fix developers patterns, it will just be an adoption blocker.

[AArnott]

I understood your point that today users choose whether to embed or not. But I don't see how them losing the choice is an adoption blocker. If folks referenced an assembly and when they update their nuget package it starts linking instead, I can't imagine how that would break them or cause them to avoid the upgrade.
If folks were linking an assembly that should be referenced instead, sure, that would cause at least a hiccup for them as they may need to deploy an assembly that they didn't before. But are there scenarios that you know of where that actually would happen? The compiler _usually_ doesn't let you link an assembly that shouldn't be. The only exception I can think of is EnvDTE, and that doesn't matter if we start referencing instead of linking because VS ships it anyway.

So what is the blocking scenario that you're thinking of if consumers can't pick?

[jaredpar]

@AArnott

So what is the blocking scenario that you're thinking of if consumers can't pick?

Exactly what I said. Today I have a library that sometimes I /link and sometimes I /reference. How in project.json can I specify the setting I want in the project I want?

I feel like this is a question that should have a very straight forward answer. Either:

• There is no way to do this. That to me is an adoption blocker. Customers don't care about best practices. They care about converting their code, as it exists today. Asking them to change deployment semantics is going to get push back.
• There is a way to do this and the syntax is X.

In the case there is no way to do this then project.json is not functionally equivalent to what I have today. It cannot fully express the flexibility of the compiler.

[AArnott]

It cannot fully express the flexibility of the compiler.

It already cannot. We're talking about giving project.json the ability to express more of what the compiler can do.

In the case there is no way to do this then project.json is not functionally equivalent to what I have today.

Sure it is: if you upgrade to the latest NuGet that adds support, _nothing changes_. You'd also need to update one of your NuGet package dependencies to a newer version that leverages the feature.

Exactly what I said

But you haven't answered my question: can you name a _specific_ scenario where it's _important_ for a customer that the behavior not change? This won't cause linked assemblies to start being referenced (since they're all referenced today, unless special msbuild authoring has been written and this feature wouldn't break that). This feature _may_ cause assemblies you were referencing to start linking (when you update the package), and that won't screw up your deployment semantics, which is what you mentioned being concerned about. You can always deploy an assembly you've linked without side effects. But now you _can_ stop deploying it.

So I repeat: I'd be interested in seeing a concrete scenario that breaks with this feature.

[jaredpar]

We're talking about giving project.json the ability to express more of what the compiler can do.

If we're fixing it then lets fix it. Instead of starting from first princples about how builds _should_ work we should start with how they actually work. Any gap in the flexibility is a potential adoption blocker.

So I repeat: I'd be interested in seeing a concrete scenario that breaks with this feature.

I've provided a concrete scenario. There is a lib that the customer sometimes /link: and sometimes /reference:. How can they replicate their build, and hence their deployment dependencies, with this proposal.

This feature may cause assemblies you were referencing to start linking (when you update the package), and that won't screw up your deployment semantics,

So you're guaranteeing that every customer out there will be 100% happy with us suddenly starting to link assemblies they were previously referencing? When they push back on this what will way say? Sorry this is how you should have been building?

[AArnott]

So you're guaranteeing that every customer out there will be 100% happy with us suddenly starting to link assemblies they were previously referencing?

No. You keep leaving out the point that we're not changing behavior for _any_ existing solutions. _You have to upgrade to see the new behavior_.

I've provided a concrete scenario. There is a lib that the customer sometimes /link: and sometimes /reference:. How can they replicate their build, and hence their deployment dependencies, with this proposal.

That sounds more abstract than concrete. It's a straw man. Can you give me a single _actual_ assembly that customers reference today that would break the customer's scenario after upgrading the package when it starts to link instead?

Otherwise, I don't see why you're so passionate about it.

[int19h]

A similar workaround NuGet package is needed for VS 2017, since it has some new assemblies that require embedding, e.g. Microsoft.VisualStudio.Imaging.Interop.15.0.DesignTime.

[AArnott]

@int19h: I've pushed this package that includes the assembly you mention: Microsoft.VisualStudio.SDK.EmbedInteropTypes 14.1.7

[int19h]

Another one is Microsoft.VisualStudio.Shell.Interop.15.0.DesignTime.

I'm not sure if there's some reliable place to get a complete list of these...

[AArnott]

There isn't an exhaustive list, sadly. So as they are reported, we add them.
I've pushed a new version that adds that assembly as well.

Microsoft.VisualStudio.SDK.EmbedInteropTypes 15.0.4

[jaredpar]

Still feel like this is the wrong approach. The question of

How can I embed the interop types from NuGet package X?

Shouldn't be

Andrew creates a new NuGet package

That's not scalable. I should be able to declare that I want an existing package embedded as is. The existing solution is introducing an artificial barrier into NuGet package consumption that shouldn't exist.

[AArnott]

I totally agree that I don't scale.

But I still think the package author should be able to indicate that an assembly is meant to be linked rather than referenced so consumers don't have to be super smart and take extra steps to do it right.

[tmat]

Any updates on this? This is PITA to deal with.

[IlyaBiryukov]

It would be great to have it in metadata. E.g.

<PackageReference Include="MyEmbeddableNuget" Version="1.0.0">
    <EmbedInteropTypes>true</EmbedInteropTypes>
</PackageReference>

[AArnott]

@IlyaBiryukov since packages can contain several assemblies, and those assemblies may be a mix of those that should be linked vs. those that should be referenced, how would you resolve that in your proposal?

Jared and I discuss this option above of letting the consumer influence the linking of assemblies, which you can review. My argument is an assembly can and should be linked if and only if its author went to special steps to enable and maintain it. There is no circumstance (that I'm aware of) that the consumer could make a better decision about link vs. ref than the library author. So I'm arguing that the decision be made and expressed in the package itself.

[BobbyCannon]

I just realized that the new PackageReference won't let me "EmbedInteropTypes" == false for certain assembly references... so I have to roll back a really long conversion from packages.config to PackageReference, 14 projects, wish I would have known this earlier today.

Anyway, what does this mean for projects like my open source project that requires "EmbeddedInteroptTypes" reference option? I guess I should just post a notice to not migrate to PackageReference until there is a way to handle this. Kinda stinks because people are going to pull my nuget package and think TestR doesn't work. Super sadface... I think this is important for the minority (aka me) :)

[AArnott]

@BobbyCannon I suspect you meant that it won't let you EmbedInteropTypes=true, since false is the default.
And you can workaround this limitation on an individual basis by adding an msbuild .targets file to your package that you publish, like I did.

[BobbyCannon]

@AArnott Actually my issue is I need to set it to false for Interop.UIAutomationClient. I'm still not sure I understand. I'll look more into msbuild .targets file, you have any good sources of information on them. Specifically on how to setup a nuget package with one?

[BobbyCannon]

@AArnott, thanks for your help. I think I figured out this part... now to figure out why my other assembly is not being copied to the bin/debug folder... I don't understand because it should be. Works with packages.config? so...

[AArnott]

The CopyLocal behavior of your assembly is a different question from whether the assembly is linked or referenced. You should look for or file a separate issue to track that.

[nkolev92]

@BobbyCannon
Are you still having issues with the assembly copying?
If you do, please create another issue for that.

We're driving an effort to help migrate everyone from PC -> PR right now, and we'd be happy to work with you to get your package PackRef compatible.

[BobbyCannon]

Sorry been distracted the last few days. I'm good now. I created a .targets file and got it working that way? Everything seems to be working well... so I guess I'm good.

[BobbyCannon]

Is there a way to setup a nuget to set
"<ExcludeAssets>runtime</ExcludeAssets>"
this via the .nuget file?

[nkolev92]

@BobbyCannon I'm not sure what do you mean by the .nuget file.

If you have a different concern, would you mind creating a different issue so we don't pollute this one with not relevant info. https://github.com/NuGet/Home/issues/new

Thanks.

[nkolev92]

Closed the issue in the PR, so adding a bit more context here.

Due to performance considerations, we have decided to move the responsibility for correct authoring to the package author.

We have analyzed the number of packages, impact and we think this is a suitable course of actions.

More details and an example here:

https://github.com/NuGet/Samples/tree/master/NuGet.Samples.Interop

[AArnott]

I wonder what the perf considerations were. Can you share?

Thanks for the sample.

[AArnott]

From the sample:

Additionally, by default the build assets do not flow transitively. Packages authored as described here work differently when they are pulled as a transitive dependency from a project to project reference. The package consumer can allow them to flow by modifying the PrivateAssets default value to not include build.

This is the part that hurts the worst. It's not entirely in the package author's hands then -- middle-tier consumers have to be aware and do something special. That's unfortunate.

[nkolev92]

@AArnott

We would need load the assembly to detect whether it's interop, so adding it in either restore/build it adds ms range overhead even with simple packages.
In addition, there's added complexities when dealing with multi-targeting packages, since we will need to load the same assembly name.

The build assets flowing case is something we are looking into and hope to share more info/have a discussion soon.

[AArnott]

We would need load the assembly to detect whether it's interop

Why would you need to auto-detect whether it's interop? As I mentioned here, the auto-detection logic that project systems already have isn't quite 100% accurate. So it's both reasonable and performant to leave it to the package author to use a link/ directory instead of ref/ or lib/ to specify assemblies that should be embedded (passed to the /link: switch of the compiler). That means you don't need to load the assembly at all as you're installing/restoring packages since it's obvious based on the file layout.

[jainaashish]

@nkolev92 Did we ever discussed /link folder to specify embedded libraries? This seems like a reasonable solution as well.

[nkolev92]

We did. We opted for the above solution because it was a minimal fix.

In addition, adding a "link" folder makes the packages incompatible with older clients which is something we wanted to avoid.

[jainaashish]

This is the part that hurts the worst. It's not entirely in the package author's hands then -- middle-tier consumers have to be aware and do something special. That's unfortunate.

AFAIK build artifacts not flowing transitively shouldn't impact this case, since an assembly has already been embedded in one package then it's always available for all the parent packages. It just that this assembly will be referenced instead of linked in destination project, which should not cause any trouble.

Example, ProjA -> ProjB -> PackageA (A.interop.dll)
ProjB -> linked A.interop.dll (so if we create a package out of ProjB and install it some where then A.interop.dll will be available there)
ProjA -> reference A.interop.dll and ProjB.dll (so creating a package out of ProjA, and installing will also make sure A.interop.dll is available as part of ProjB.dll)

[AArnott]

AFAIK build artifacts not flowing transitively shouldn't impact this case, since an assembly has already been embedded in one package then it's always available for all the parent packages. It just that this assembly will be referenced instead of linked in destination project, which should not cause any trouble.

I'm afraid that's not true. The middle-ware package will link it, and the app will reference it, and that will cause trouble. Referencing instead of linking can cause runtime failures for the app. And the C# compiler (if you're lucky) will emit a warning indicating that your transitive closure of references uses an inconsistent style of linking vs. referencing and warn you that you're referencing instead of linking.

We opted for the above solution because it was a minimal fix.

It doesn't sound like a fix, really. It's just documenting the workaround we're already using, and filed this issue in order to ask for a proper fix.
Also, the workaround is incomplete, I believe, because a linked assembly should not ever be copied to the app's output directory. Copying it defeats part of the point of linking it, and will cover up bugs where the assembly was referenced instead of linked.

[nkolev92]

Reopening based on @AArnott's feedback.
We will revisit this topic internally and discuss again.

[AArnott]

Thanks.

In addition, adding a "link" folder makes the packages incompatible with older clients which is something we wanted to avoid.

FWIW, we are already using the workaround. So if you add a feature that only works with future nuget clients, I think that's OK actually. We'll just keep using the workaround until the new nuget client version is the dominant version and then switch to using the feature. :)

[TobiasBreuer]

Hi all,
is there any news to this one? I'm having really big issues with this topic at the moment. Although I have to admit I'm not sure if this issue here is the real culprit of my problem or if it is an MSBuild issue.

We have a package that contains a library that needs to be embedded. I used the given sample to create that package (it is being hosted internally on a private ProGet server however).

To break it down:

• I have SolutionA containing ProjectA having a nuget reference to the Package.
• SolutionA can be build with VisualStudio and MSBuild without any issue. See UPDATE below
• In another Solution, SolutionB, ProjectA is also included (as existing project) and referenced by other projects.

Now SolutionB cannot be build by MSBuild if ProjectA has not been build before. There will be errors regarding the return types of certain interface members of the package library (Somehow the return type changes if the library is embedded - I'm not an expert regarding interop types, but I think it is not the point).

However if MSBuild is triggered a second time (note that the 'obj' folder of ProjectA is filled with the respective nuget targets and spec files etc.) the build will succeed...

So it seems, that during the first build approach, there is a (maybe timing related?) issue that leads the build to fail, but the "nuget specific" tasks are done. On the second time, MSBuild detects that the "nuget specific" tasks have been done before, and directly builds the project and succeeds.

If anyone has some hints or insights I would be happy to listen.

Update This second bullet point regarding SolutionA building always is not true. It most of the time can be build but I just realized that it sometimes also fails (at least with MSBuild - VS seems to work ok). This makes me more certain that there is some kind of timing issue.

[adam-stewart-reddeer]

How can i get a version of Microsoft.Office.Interop.Excel that is ready for PackageReference ? Thanks

[jainaashish]

Hey Guys, to resolve this issue, current proposal is to introduce a new folder structure /embed (open for suggestion on name) inside .nupkg similar to /lib where authors can put embedInteropType assemblies. These assets will be in new section Embed in project.assets.json file so that project system knows these would be linked unlike referencing compile assets.

Now, there is one open question, should there be a new IncludeAssets value called Embed similar to Compile, Runtime, ContentFiles, Build, Analyzers? or should these embed assets always be included in the project and flow transitively. I personally believe that we should add another value to let consumer decides the behavior but by default they will be included and flow transitively. But I don't have full context and doesn't want to add an option which might never be used. So please let me know what you guys think?

[AArnott]

I fully support the /embed folder structure solution.

Regarding the IncludeAssets question, I think we should let the compile value control both lib and embed folder propagation. My reasons are:

1. Most people aren't familiar with embedding assemblies. So adding an option would be making prominent something that most people won't understand.
2. For all practical purposes, an embedded assembly offers contracts just like referencing an assembly does. The compile option that already exists is how people express interest in the public types from a package today. If they know the package offers types but "compile" doesn't get it, they'll be thoroughly confused. They shouldn't have to know the implementation detail of the package that they need to opt into some embed value instead.
3. Transitive dependencies: if pkg A is embedded and pkg B consumes it, when C consumes B, it must also consume A in order to guarantee that it can call all of B's public API (since it might expose types from A). This is a very strong reason why lib folder references propagate transitively today, and linking/embedding is no different.

I can't think of any time where someone should need to distinguish between linking and referencing assemblies within a package. And if they just wanted to avoid embedding the types because they didn't need them, the compiler already only embeds types that are actually used by that assembly, so nothing will be embedded unless they actually need it.

[jainaashish]

if they just wanted to avoid embedding the types because they didn't need them, the compiler already only embeds types that are actually used by that assembly, so nothing will be embedded unless they actually need it

Good to know this, if that's the case then I agree compile can work for both lib as well as embed and consumer doesn't need to know underline implementation of reference vs linking.

Let us know if anyone else has any other concerns or proposal.

[jainaashish]

After some more discussion with the current proposal, the other concern is in order to author a new interop package which works for packages.config as well as PackageReference, package author will need to duplicate interop assemblies in /lib (for PC) and /embed (for PR) which doesn't seem like an ideal solution.

So another proposal is to add a new value to existing metadata PackageType in nuspec file to let package author say that it’s interop type package. And if an author set PackageType to Interop then we’ll treat this package as interop type and tell project system to link all it’s /lib assemblies instead of referenced. This will not change anything to packages.config which will continue to work as of today.

Only concern with this new design is that it won’t support creating a package with both interop as well as non-interop type assemblies. Now, I assume nobody would already be doing that or will need to do something, there will always be a separate package for interop types. I also ran a little experiment to look through the top x% packages at NuGet.org with multiple assemblies to see if there are such packages and found nothing. But I'm not so certain about the same for private feeds.

So let me know if anyone has such packages or such scenario which we should consider or how do you guys feel about second proposal... I'm more inclined towards PackageType solution which seems more cleaner and clearer.

[nkolev92]

I disagree about PackageType being clearer.
PackageType is this scenario is a misuse of the feature.

PackageType is used to call out more fundamental different between packages, examples being "libraries" vs "tools" vs "chocolatey packages" etc.

In the end, interops are part of compilation, which is exactly what libraries or the "dependency" package type is for.

[jainaashish]

interop is a special type of dependency type packages, which is why I said there could be multiple package types for a single package as in this case it would be dependency and interop or embed or something. Since dependency is default so authors will only need to specify one. Also interop are not applicable for tools packages which is the other type.

Anyways, we'll discuss it more, but the new proposal is to support interop types through nuspec metadata instead of adding a new folder structure. If people like that, then we'll decide upon using existing metadata vs introducing a new one.

[nkolev92]

Interops are still a compile time decision though, defining it on the package is too broad.

dependency is the default if nothing is specified.
With that proposal, you'd need to explicitly specify dependency and embed for it work.

Furthermore right now, the UI will "fail" when installing packages with multiple package types.
We have even had some reservation about the design of package type, see https://github.com/NuGet/Home/issues/6298. The reason for that being that the currently known package types have significant differences in how they behave, from dependency, to dotnettool, to dotnetclireference to msbuild sdks.

[AArnott]

Interops are still a compile time decision though, defining it on the package is too broad.

I disagree. The package author knows whether they designed the assembly for linking or referencing, since linking requires very special considerations. The user should not reference an assembly that is meant to be linked. That causes problems like compiler warnings, failures at runtime when the assembly isn't even present, etc.

BTW, on that subject, nuget should not deploy a embed/link assembly to the application's bin folder (i.e. it should be Private=false, CopyLocal=false).

[jainaashish]

With that proposal, you'd need to explicitly specify dependency and embed for it work.

No you don't. With new proposal, PackageType:

• Default continue to be dependency (no change)
• embed which means its dependency package but interop type.
• dotnettool continue to be same
• dotnetclitool continue to be same
  ......................

BTW, on that subject, nuget should not deploy a embed/link assembly to the application's bin folder (i.e. it should be Private=false, CopyLocal=false).

I'd imagine Project System will take care of this since NuGet doesn't directly add any reference.

[AArnott]

BTW, on that subject, nuget should not deploy a embed/link assembly to the application's bin folder (i.e. it should be Private=false, CopyLocal=false).

I'd imagine Project System will take care of this since NuGet doesn't directly add any reference.

Can you verify this as part of this feature work, please? I don't know how the lines divide nuget vs. csproj build authoring goes, but it's the responsibility of the thing that creates the Reference/ReferencePath items to set metadata on those items to prevent copying to the bin folder. I expected NuGet owned the MSBuild targets that converted PackageReference to ReferencePath, which if that's the case, suggests that my request is a nuget responsibility, IMO.

[nkolev92]

I disagree. The package author knows whether they designed the assembly for linking or referencing, since linking requires very special considerations. The user should not reference an assembly that is meant to be linked. That causes problems like compiler warnings, failures at runtime when the assembly isn't even present, etc.

@AArnott
I worded that badly. I am not suggesting it's a user decision.
It's an author decision.
I meant solely as "link" vs "ref" during compilation.

Even if it's not a scenario where people have packages that don't do both, I don't see the need to change a well established pattern that doesn't have any obvious issues.

@jainaashish
If you want to do both is what I mean.

[jainaashish]

Even if it's not a scenario where people have packages that don't do both, I don't see the need to change a well established pattern that doesn't have any obvious issues.

I'm sorry but loosing the point here, when did it become a well established pattern? which nobody uses...? interop types are already a advance concept which only handful of people use so they know what they are doing.

[nkolev92]

The well established pattern is the folders for different asset groups.
ContentFiles has a similar approach to this where the build action is specified by the nuspec, but even that's for a specific asset group rather than on a whole package.

[jainaashish]

in that sense, these are still compile assets which are being linked instead of reference which is a implementation details. So essentially these are still not entirely a different asset group rather a different package type, IMO.

[jainaashish]

Updated Spec - https://github.com/NuGet/Home/wiki/Embed-Interop-Types-with-PackageReference

[brinko99]

@nkolev92

More details and an example here: https://github.com/NuGet/Samples/tree/master/NuGet.Samples.Interop

You're sample seems to fail for me when the package's target framework doesn't match that of the importing project. For example, your sample targets net46. When I add the package to Core 3.1 project, the file is neither linked or referenced. VS doesn't even show the containing assemblies.

If I modify the package to include the same assembly in netcoreapp3.1 folder, it does work. Is this expected behavior? Is there any further workaround at this time while we wait for 'embed'?

Edit: This is under VS 16.6.3, dotnet 3.1.301

Edit: I've got it working...

My assemblies were located under lib\net35; I had the .targets under build\ but find that it _must_ be under build\net35.

Thanks,