Runtime: AssemblyLoadContext crash when collectible assembly use XmlSerializer

[uffebjorklund]

[EDIT]
Investigation of this issue is complete and has resulted in 5 additional issues being opened.
See this comment below for links to each of those issues.

We will keep this issue open and update it as those other issues are fixed.
Please add any additional comments to those issues.

---

When creating a collectible AssemblyLoadContext and then use System.Xml.Serialization.XmlSerializer inside of a collectible assembly you will get a crash.

I have a repro project: https://github.com/uffebjorklund/AssemblyLoadContextBug

Have tried a few different things

• If creating the ALC as collectible = false there is no crash
• If the XmlSerializer code is removed from the collectible assembly there is no crash
• The crash will occur regardless of using netstandard2.0 or netstandard2.1 on the collectible assembly

Exception

A non-collectible assembly may not reference a collectible assembly

Stacktrace

at System.Reflection.Emit.ModuleBuilder.GetTypeRef(QCallModule module, String strFullName, QCallModule refedModule, String strRefedModuleFileName, Int32 tkResolution)
at System.Reflection.Emit.ModuleBuilder.GetTypeRefNested(Type type, Module refedModule, String strRefedModuleFileName)
at System.Reflection.Emit.ModuleBuilder.GetTypeTokenWorkerNoLock(Type type, Boolean getGenericDefinition)
at System.Reflection.Emit.ModuleBuilder.GetTypeTokenInternal(Type type, Boolean getGenericDefinition)
at System.Reflection.Emit.ILGenerator.Emit(OpCode opcode, Type cls)
at System.Xml.Serialization.CodeGenerator.Castclass(Type target)
at System.Xml.Serialization.CodeGenerator.InternalConvert(Type source, Type target, Boolean isAddress)
at System.Xml.Serialization.SourceInfo.InternalLoad(Type elementType, Boolean asAddress)
at System.Xml.Serialization.XmlSerializationWriterILGen.WriteElement(SourceInfo source, ElementAccessor element, String arrayName, Boolean writeAccessor)
at System.Xml.Serialization.XmlSerializationWriterILGen.WriteElements(SourceInfo source, String enumSource, ElementAccessor[] elements, TextAccessor text, ChoiceIdentifierAccessor choice, String arrayName, Boolean writeAccessors, Boolean isNullable)
at System.Xml.Serialization.XmlSerializationWriterILGen.WriteMember(SourceInfo source, String choiceSource, ElementAccessor[] elements, TextAccessor text, ChoiceIdentifierAccessor choice, TypeDesc memberTypeDesc, Boolean writeAccessors)
at System.Xml.Serialization.XmlSerializationWriterILGen.GenerateTypeElement(XmlTypeMapping xmlTypeMapping)
at System.Xml.Serialization.XmlSerializationWriterILGen.GenerateElement(XmlMapping xmlMapping)
at System.Xml.Serialization.TempAssembly.GenerateRefEmitAssembly(XmlMapping[] xmlMappings, Type[] types, String defaultNamespace)
at System.Xml.Serialization.TempAssembly..ctor(XmlMapping[] xmlMappings, Type[] types, String defaultNamespace, String location)
at System.Xml.Serialization.XmlSerializer.GenerateTempAssembly(XmlMapping xmlMapping, Type type, String defaultNamespace, String location)
at System.Xml.Serialization.XmlSerializer..ctor(Type type, String defaultNamespace)

dotnet --info

.NET Core SDK (reflecting any global.json):
 Version:   3.0.100
 Commit:    04339c3a26

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.13
 OS Platform: Darwin
 RID:         osx.10.13-x64
 Base Path:   /usr/local/share/dotnet/sdk/3.0.100/

Host (useful for support):
  Version: 3.0.0
  Commit:  7d57652f33

.NET Core SDKs installed:
  2.1.301 [/usr/local/share/dotnet/sdk]
  2.1.302 [/usr/local/share/dotnet/sdk]
  2.1.400 [/usr/local/share/dotnet/sdk]
  2.1.401 [/usr/local/share/dotnet/sdk]
  2.1.403 [/usr/local/share/dotnet/sdk]
  2.1.500 [/usr/local/share/dotnet/sdk]
  2.2.100-preview3-009430 [/usr/local/share/dotnet/sdk]
  2.2.101 [/usr/local/share/dotnet/sdk]
  2.2.102 [/usr/local/share/dotnet/sdk]
  2.2.103 [/usr/local/share/dotnet/sdk]
  2.2.105 [/usr/local/share/dotnet/sdk]
  2.2.106 [/usr/local/share/dotnet/sdk]
  2.2.203 [/usr/local/share/dotnet/sdk]
  2.2.300 [/usr/local/share/dotnet/sdk]
  3.0.100-alpha1-009708 [/usr/local/share/dotnet/sdk]
  3.0.100-preview-010184 [/usr/local/share/dotnet/sdk]
  3.0.100-preview3-010431 [/usr/local/share/dotnet/sdk]
  3.0.100-preview8-013656 [/usr/local/share/dotnet/sdk]
  3.0.100-preview9-014004 [/usr/local/share/dotnet/sdk]
  3.0.100 [/usr/local/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.1 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.2 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.3 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.5 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.6 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.0-preview3-35497 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.0 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.1 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.3 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.4 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 3.0.0-alpha1-10062 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.1 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.2 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.3 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.5 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.6 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.0-preview3-35497 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.0 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.1 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.3 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.4 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0-alpha1-10062 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0-preview-19075-0444 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0-preview3-19153-02 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0-preview8.19405.7 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0-preview9.19424.4 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.1 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.2 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.3 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.6 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.0-preview3-27014-02 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.1 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.3 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.4 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0-preview-27324-5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0-preview1-27029-03 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0-preview3-27503-5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0-preview9-19423-09 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

[csalat]

Hello,

Is any update on this?

Thanks,
Csaba

[StephenBonikowsky]

@uffebjorklund Thanks for reporting this, sorry for the slow response. It looks like all the information we need to do further investigation is here. We will look into it.

[mconnew]

I did a bit of digging in to this and I believe that the problem is due to XmlSerializer creating a dynamic assembly in the default load context which makes the dynamic assembly non-collectible. When we call ILGenerator.Emit(OpCodes.Castclass, type) with a type which is a collectible assembly, we get an exception. I did some digging around the api's and I can't see a way to specify the AssemblyLoadContext to generate a dynamic assembly into.
@sdmaclea, is this scenario supported? If so, can you point me to the api's to create a dynamic assembly in a specific AssemblyLoadContext?

[sdmaclea]

@sdmaclea, is this scenario supported? If so, can you point me to the api's to create a dynamic assembly in a specific AssemblyLoadContext?

@mconnew I think this was a feature we missed in 3.0. There is an API proposal to close this gap dotnet/corefx#38426

/cc @vitek-karas @elinor-fung @jeffschwMSFT

[curia-damiano]

Hi,
I share the same supposition, that it can be because of the dynamic generated assemblies.
But: I have tried using the XmlSerializerGenerator, with the idea that it wouldn't generated dynamic assemblies anymore.
But still I can't make it to work, see attached solution.
Do I miss anything?!?
Test_ALC with XmlSerializerGenerator.zip

[curia-damiano]

Btw this was my first project with the issue, without the XmlSerializerGenerator
Test_ALC.zip

[vitek-karas]

@curia-damiano How do you generate the serialization code? I tried to turning it on via VS on the CL.Implementation.Problem assembly and that failed (which is probably an issue on its own, but not directly related to this issue).

Also - I noticed that your "plugins" are built as netstandard2.0 - this is not recommended as SDK will not always produce the right assets. Ideally plugins should not target a standard but a specific framework - for example netcoreapp3.1 or so.

[curia-damiano]

Hi,
I have a ItemGroup in the project that "should" invoke the generation of the serialization code during build time. I haven't found any additional documentation saying I should do something more.
FYI I have used VS 2019 Preview (16.4.0 Preview 5 or 6).

[vitek-karas]

I tried again - Release build this time - and I spotted this in the output of the build:

3>Method 'System.Xml.Serialization.XmlSerializer.GenerateSerializer' was not found. This is likely because you are using an older version of the framework. Please update to .NET Core v2.1 or later.
3>Sgen utility failed to pregenerate serialization code for E:\AppModel\repro\Test_ALC\CL.Implementation.Problem\obj\Release\netstandard2.0\CL.Implementation.Problem.dll.

@StephenMolloy for the XML serializer specific portions of this question, like the above failure of sgen.

[mconnew]

@curia-damiano, I haven't tried running your code yet but I can see a problem in Test_ALC with XmlSerializerGenerator.zip. This is the problem code:
c# var assembly = TLC.LoadFromAssemblyPath(dllPath); labelUnloaded.Text = "Loaded"; var type = assembly.GetType("CL.Implementation.Easy.Multiplier"); var constructor = type.GetConstructor(new Type[] { }); ICompute compute = (ICompute)constructor.Invoke(new object[] { });
When using a different AssemblyLoadContext, the same type in the same assembly is a different runtime type. So this means the ICompute interface that CL.Implementation.Easy.Multiplier implements is a different runtime type that the same interface in the default load context. So the cast on the last line will fail.

[mconnew]

@sdmaclea, in this comment it says it uses the callers AssemblyLoadContext. You can see here that we do use AssemblyBuilder.DefineDynamicAssembly so I would expect the dynamically created assembly to use the custom AssemblyLoadContext. Is the problem that we pass AssemblyBuilderAccess.Run instead of AssemblyBuilderAccess.RunAndCollect? What's stopping the dynamically generated assembly from being created in the correct context? Also, is this going to be a problem for DispatchProxy too?

[sdmaclea]

@sdmaclea, in this https://github.com/dotnet/corefx/issues/38426#issuecomment-500723986 it says it uses the callers AssemblyLoadContext. You can see https://github.com/dotnet/corefx/blob/e99ec129cfd594d53f4390bf97d1d736cff6f860/src/System.Private.Xml/src/System/Xml/Serialization/CodeGenerator.cs#L1297 that we do use AssemblyBuilder.DefineDynamicAssembly so I would expect the dynamically created assembly to use the custom AssemblyLoadContext.

@mconnew Looking at the link. It looks like System.Xml.Serialization.CodeGenerator.CreateAssemblyBuilder is the caller of AssemblyBuilder.DefineDynamicAssembly so the dynamic assembly would be created in the default ALC, where the CoreFX runtime is loaded.

[sdmaclea]

When using a different AssemblyLoadContext, the same type in the same assembly is a different runtime type. So this means the ICompute interface that CL.Implementation.Easy.Multiplier implements is a different runtime type that the same interface in the default load context. So the cast on the last line will fail.

I haven't looked at the code. So I am not sure this statement above is correct.

Presuming the cited code is loaded and running in AssemblyLoadContext.Default, the ICompute cast will attempt to cast to AssemblyLoadContext.Default version of the type.

For the cast to succeed:

• the CL.Implementation.Easy.Multiplier constructor will need to return an object which implements ICompute.
• ICompute will need to be defined in a different assembly than CL.Implementation.Easy.Multiplier
• The assembly which defines ICompute should only be loaded into AssemblyLoadContext.Default. So that there is only one version of the type.

[curia-damiano]

Hi,
Thank you for the comments above.
So, a good point is that using the XmlSerializerGenerator should be a walkaround for the problem.
The bad point is that XmlSerializerGenerator is badly (or almost no) documented.

@csalat: Following the comments above:

• I have converted the dll from netstandard2.0 to netcore3.1;
• I have removed the project ItemGroup to invoke the generator (now it's in the build.props of the NuGet package, as it comes out from multiple comments);
• still, I get a compile time error saying that it can't find any time to be serialized.

@mconnew : Also, about the cast at runtime: I get the point that there is another instance of the dll in memory and the cast doesn't work. Still I have to guess how to solve this after I am able to make the serializer generator work. My idea is that I should be able to tell the dynamic dll loader to use the Interface dll already loaded in memory. Any tip about this second point?

I kindly ask all of you if you can please help me in making this simple POC to work.

[curia-damiano]

I attach here the modified POC after the changes discussed in the previous post
Test_ALC.zip

[vitek-karas]

@curia-damiano please consider using the same patterns as shown in this sample https://docs.microsoft.com/en-us/dotnet/core/tutorials/creating-app-with-plugin-support. It shows how to write an app with plugins where the problem with casting the interface types doesn't exist.

Namely I would suggest:

• Use AssemblyDependencyResolver if possible - in the sample you shared this is definitely a typical case where it should be used. Using mechanisms like enumerating files in current directory is problematic (it's not guaranteed that the current directory will be what you expect for example, or if there's a .dll which should not be present it might get picked up and so on).
• Build the plugins as "plugins" - that is mark the project dependencies which should be provided by the host with the <Private>false</Private> so that the plugins don't carry copies of host assemblies with them.
• Unless necessary, avoid duplication of assembly resolution logic - in your sample you implement both the AssemblyLoadContext.Load method as well as a handler for the AssemblyLoadContext.Resolving event. You should not need both for typical cases.
• Ideally the plugins should be self-descriptive, that is carry only the files they need to keep in isolation and rely on the host to provide the rest. If that is the case, then you don't need to implement any logic to search for assemblies which are already loaded - instead, just return null from AssemblyLoadContext.Load and the runtime will fallback to the AssemblyLoadContext.Default which has the host (the app).
• Avoid usage of AppDomain.CurrentDomain.GetAssemblies - this returns list of all assemblies loaded in the process, across all AssemblyLoadContext instances. This means that if you load two plugins which both carry some private dependency, you might end up using the dependency from another plugin. It might work, but it would for example prevent unloading from working correctly.

In the sample app I fixed the path to the "problem" dll to use netcoreapp3.1 and now it fails to find the XmlSerializers.dll, which is expected given the failure during build.
If I comment out loading the XmlSerialziers.dll the plugin itself works correctly (so casting to ICompute works, since the custom ALC does end up sharing the Interface assembly correctly), but it fails in the XmlSerializer due to the wrong load context used by the dynamically generated code.

[curia-damiano]

Hi Vitek,
Thank you very much for your extensive explanation.
Tomorrow I will go through the sample you have suggested me and adapt my
small POC accordingly.
Only, I think I still miss guidance about how to use the
XmlSerializerGenerator...
Regards, Damiano

On Tue, Nov 26, 2019 at 4:43 PM Vitek Karas [e-mail]
wrote:

@curia-damiano https://github.com/curia-damiano please consider using
the same patterns as shown in this sample
https://docs.microsoft.com/en-us/dotnet/core/tutorials/creating-app-with-plugin-support.
It shows how to write an app with plugins where the problem with casting
the interface types doesn't exist.

Namely I would suggest:

• Use AssemblyDependencyResolver if possible - in the sample you
  shared this is definitely a typical case where it should be used. Using
  mechanisms like enumerating files in current directory is problematic (it's
  not guaranteed that the current directory will be what you expect for
  example, or if there's a .dll which should not be present it might get
  picked up and so on).
• Build the plugins as "plugins" - that is mark the project
  dependencies which should be provided by the host with the
  false so that the plugins don't carry copies of
  host assemblies with them.
• Unless necessary, avoid duplication of assembly resolution logic -
  in your sample you implement both the AssemblyLoadContext.Load method
  as well as a handler for the AssemblyLoadContext.Resolving event. You
  should not need both for typical cases.
• Ideally the plugins should be self-descriptive, that is carry only
  the files they need to keep in isolation and rely on the host to provide
  the rest. If that is the case, then you don't need to implement any logic
  to search for assemblies which are already loaded - instead, just return
  null from AssemblyLoadContext.Load and the runtime will fallback to
  the AssemblyLoadContext.Default which has the host (the app).
• Avoid usage of AppDomain.CurrentDomain.GetAssemblies - this returns
  list of all assemblies loaded in the process, across all
  AssemblyLoadContext instances. This means that if you load two plugins
  which both carry some private dependency, you might end up using the
  dependency from another plugin. It might work, but it would for example
  prevent unloading from working correctly.

In the sample app I fixed the path to the "problem" dll to use
netcoreapp3.1 and now it fails to find the XmlSerializers.dll, which is
expected given the failure during build.
If I comment out loading the XmlSerialziers.dll the plugin itself works
correctly (so casting to ICompute works, since the custom ALC does end up
sharing the Interface assembly correctly), but it fails in the
XmlSerializer due to the wrong load context used by the dynamically
generated code.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dotnet/corefx/issues/41286?email_source=notifications&email_token=AEN72M72A5KUZPNCNR7M2O3QVU7ZLA5CNFSM4I2BJI4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFGOTYI#issuecomment-558688737,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEN72M45JJKS6QJJ2DIR44TQVU7ZLANCNFSM4I2BJI4A
.

--
Damiano Curia
Swiss mobile: [phone]
Italian mobile: [phone]
skype: curia.damiano
email: damiano.[email protected]
www: http://curia.me

[vitek-karas]

I'm not an expert in the XmlSerializerGenerator, so hopefully others on this thread will be able to help with that aspect of the issue.

[mconnew]

I've worked out the problem with XmlSerializerGenerator. There's a temporary workaround until we work out a proper fix. The issue is being tracked in issue dotnet/corefx#40730. Unfortunately that doesn't solve this problem. XmlSerializer loads the pre-generated serialization assembly into the default AssemblyLoadContext. It then calls the method CanSerialize in the pre-generated serializer which references the types CL.Implementation.Problem.Adder and CL.Implementation.Problem.MyEntity. These types are attempted to be loaded in the default AssemblyLoadContext which is then unable to find them. We should be able to fix that problem by getting the AssemblyLoadContext from the passed in type and loading the pre-generated assembly into it, but I suspect this might end up being an onion problem where we might ultimately need new capabilities from the AssemblyLoadContext feature. We won't know until we get there.

[vitek-karas]

I tried the workaround and ran into two other issues:

Doesn't work with netcoreapp3.1

If I set the TFM to netcoreapp3.1 (since I'm compiling the component in questions as netcoreapp3.1) the SGEN itself works and produces the .cs file, but then compiling that fails with:

2>CSC : warning CS1703: Multiple assemblies with equivalent identity have been imported: 'C:\Windows\Microsoft.NET\Framework64\v4.0.30319\System.Data.DataSetExtensions.dll' and 'C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\3.1.0\ref\netcoreapp3.1\System.Data.DataSetExtensions.dll'. Remove one of the duplicate references.

This in itself shows two problems:

• We obviously don't de-dupe the ref assemblies for some reason.
• More alarmingly, we're using .NET Framework assemblies - something in the build decided that it should look for ref assemblies in the .NET Framework installation. This seems really weird for .NET Core tools.

The generated assembly is not in .deps.json

Once I switched to netcoreapp2.2 it worked and generated the serialization assembly. But I noticed that while we do put that assembly to the output folder, we don't put it into the .deps.json for the app/component. As it now becomes part of the app, it should be listed in the .deps.json as well.
Additionally, the .deps.json now contains entry for dotnet-Microsoft.XmlSerialzier.Generator.dll. If I understand it correctly that assembly should not be needed/used at runtime at all - so it should not be listed in .deps.json. In 3.0 (if it worked) I would actually expect this to be copied to the output (since all NuGet dependencies are now copied to build output).

Loading the generated code

The code which loads the generated assembly is not only not AssemblyLoadContext aware, but to make things worse, it uses Assembly.LoadFile - this will load the assembly into a new "private" instance of AssemblyLoadContext. Ideally the loader would first try to load it via Assembly.Load (or even better AssemblyLoadContext.LoadAssemblyByName) so that the custom ALC logic can kick in. Note that Assembly.Load normally loads into the ALC of the caller, in this case that would be Default, but this can be changed by using contextual reflection - for example AssemblyLoadContext.EnterContextualReflection. Only if that fails I think we should fall back to searching for the file ourselves, by looking next to the assembly (the way it works now). In that case it should definitely load the file into the same ALC as the assembly for which the serializer was generated - by using AssemblyLoadContext.LoadAssemblyByPath.

[curia-damiano]

Thank you Vitek for your further investigations.
So the current state is that XmlSerializerGenerator should work with AssemblyLoadContext, but due to various issues it doesn't.
FYI I have also used the NuGet preview package (available in the public NuGet), but I've got the same results.

[mconnew]

@vitek-karas, the set of reference assemblies that is passed to csc when compiling the generated serializer is just copied from the list of reference assemblies passed to csc when compiling your application assembly. We're not changing that list at all. If full framework assemblies are being added to the set of reference assemblies, something has changed in how the .NET Core SDK compiles that list. That's a breaking change in the .NET SDK.

[vitek-karas]

I'm not trying to put blame on anybody 😄 I'm just pointing out potential issues.
@nguerrera can you please comment on the presence of .NET Framework assemblies in assembly refs list for the compilers?

[nguerrera]

@vitek-karas Can you supply the steps to reproduce this error:

2>CSC : warning CS1703: Multiple assemblies with equivalent identity have been imported: 'C:\Windows\Microsoft.NET\Framework64\v4.0.30319\System.Data.DataSetExtensions.dll

I cloned, retargeted to netcoreapp3.1 and got the original error on run, nothing like that on build. You mention steps to get past this, but it's hard to follow. Do you have a repo in this state you could share?

[nguerrera]

Catching up a bit here, I'm using https://github.com/dotnet/corefx/issues/41286#issuecomment-558503976

Compiliation succeeds, and there are no .NET Framework references passed to the compiler. Later build step fails with:

C:\Users\nicholg\.nuget\packages\microsoft.xmlserializer.generator\2.2.0-preview3.19551.4\build\Microsoft.XmlSerializer.Generator.targets(37,5): warning MSB3073: The command "dotnet Microsoft.XmlSerializer.Generator "obj\Debug\netcoreapp3.1\CL.Implementation.Problem.dll" --force --quiet obj\Debug\netcoreapp3.1\sgen.rsp" exited with code 1. [C:\Users\nicholg\Desktop\Test_ALC\CL.Implementation.Problem\CL.Implementation.Problem.csproj]
C:\Users\nicholg\.nuget\packages\microsoft.xmlserializer.generator\2.2.0-preview3.19551.4\build\Microsoft.XmlSerializer.Generator.targets(38,5): warning : SGEN: Failed to generate the serializer for CL.Implementation.Problem.dll. Please follow the instructions at https://go.microsoft.com/fwlink/?linkid=858594 and try again. [C:\Users\nicholg\Desktop\Test_ALC\CL.Implementation.Problem\CL.Implementation.Problem.csproj]

[nguerrera]

Got a little further looking at https://github.com/dotnet/corefx/issues/41286#issuecomment-558864438 and munging runtime config.

But I still can't get the serialization to succeed in a build.

[nguerrera]

If full framework assemblies are being added to the set of reference assemblies, something has changed in how the .NET Core SDK compiles that list

There are no .net framework assemblies on @(ReferencePath).

Although I can't get that far, I suspect the problem is here by inspection:

https://github.com/dotnet/corefx/blob/ab01852685e659a685af2ace33af5b3cefdf8430/src/Microsoft.XmlSerializer.Generator/pkg/build/Microsoft.XmlSerializer.Generator.targets#L73

Copied below with line breaks:

    <Csc Condition="Exists('$(IntermediateOutputPath)%(_ReferenceSerializationAssemblyName.Identity).cs') == 'true'" 
         ContinueOnError="true" 
         OutputAssembly="$(IntermediateOutputPath)%(_ReferenceSerializationAssemblyName.Identity).dll" 
         References="@(ReferencePath);@(IntermediateAssembly)" 
         EmitDebugInformation="$(DebugSymbols)" 
         Sources="$(IntermediateOutputPath)%(_ReferenceSerializationAssemblyName.Identity).cs" 
         TargetType="Library"
         ToolExe="$(CscToolExe)" 
         ToolPath="$(CscToolPath)" 
         DisabledWarnings="$(_SerializationAssemblyDisabledWarnings)"/>

This is missing NoConfig and NoStandardLib:

https://github.com/dotnet/roslyn/blob/8d1995963dc0d7add5f6482e323ced9bae2f971b/src/Compilers/Core/MSBuildTask/Microsoft.CSharp.Core.targets#L96-L98

Which I think is causing the framework assemblies to be implicitly pulled in by Csc

cc @jaredpar

[nguerrera]

Yet another example where a first class source generation feature would help avoid the peril of trying to invoke csc yourself in the build. Failing that, https://github.com/dotnet/roslyn/issues/35835

[nguerrera]

Also, I'm not seeing anything different about netcoreapp3.1 in this. I suspect the bad behavior would be seen with msbuild or VS and hence .NET Framework csc as opposed to dotnet build and hence .NET Core csc.

[nguerrera]

See also https://github.com/dotnet/sdk/pull/1629 and https://github.com/dotnet/sdk/issues/1447, where we had the same issue with satellite assembly generation and fixed it with /noconfig /nostdlib.

[vitek-karas]

I tried again with the repro project and it still fails on the .NET Framework reference. Project:
Test_ALC.zip. I also went to C:\Users\<alias>\.nuget\packages\microsoft.xmlserializer.generator\2.2.0-preview3.19551.4\lib\netstandard2.0 and modified the dotnet-Microsoft.XmlSerializer.Generator.runtimeconfig.json like this:

{
  "runtimeOptions": {
    "tfm": "netcoreapp3.1",
    "framework": {
      "name": "Microsoft.NETCore.App",
      "version": "3.1.0"
    }
  }
}

I then open the solution in the repro folder with VS (16.5 Preview 2.0) and rebuild the CL.Implementation.Problem project - will produce a repro of the issue. Running dotnet --info I see that I have 3.1.0 runtime and 3.1.100 SDK installed. So I assume it's using 3.1.100 SDK from inside VS to build.

[nguerrera]

Thanks. I can repro now and it's what I predicted above.

1. Caused by lack of /noconfig and /nostdlib in csc invocation
2. Repros in VS / msbuild
3. Does not repro with dotnet build

So I assume it's using 3.1.100 SDK from inside VS to build.

It depends what you mean by using 3.1.100 SDK. VS will use tasks and targets from Microsoft.NET.Sdk (and friends), but it has its own copies of msbuild, roslyn, nuget (and friends) that run on .NET Framework. This particular issue repros only with the full framework csc, because it adds default references to full framework without /noconfig and /nostdlib. .NET Core csc doesn't do that. The usual csc invocation passes /noconfig /nostdlib and is immune to this, but the custom csc invocation in sgen targets does not.

You can see the csc parameters using http://msbuildlog.com. There are no .NET Framework assemblies being passed to csc. It is missing /noconfig and /nostdlib

Another trick is to set the environment variable MSBUILDPRESERVETOOLTEMPFILES=1 to tell msbuild to keep the temporary *.rsp repsonse files in %TEMP%.

repro.rsp.txt

> csc @repro.rsp
Microsoft (R) Visual C# Compiler version 3.5.0-beta2-19577-04 (9b6c6104)
Copyright (C) Microsoft Corporation. All rights reserved.

error CS1703: Multiple assemblies with equivalent identity have been imported: 'C:\Windows\Microsoft.NET\Framework64\v4.0.30319\System.Data.DataSetExtensions.dll' and 'C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\3.1.0\ref\netcoreapp3.1\System.Data.DataSetExtensions.dll'. Remove one of the duplicate references.

> csc /noconfig /nostdlib @repro.rsp
Microsoft (R) Visual C# Compiler version 3.5.0-beta2-19577-04 (9b6c6104)
Copyright (C) Microsoft Corporation. All rights reserved.

[nguerrera]

In summary, https://github.com/dotnet/corefx/blob/ab01852685e659a685af2ace33af5b3cefdf8430/src/Microsoft.XmlSerializer.Generator/pkg/build/Microsoft.XmlSerializer.Generator.targets#L73

does not work with VS or full framework msbuild, because it does not pass /noconfig and /nostslib, and that probably warrants its own bug.

[curia-damiano]

Hi @nguerrera, would it make sense to open a dedicated issue for the XmlSerializerGenerator that is not compiling in .netcore 3.0 and 3.1?
As it seems you are on the piece, would you be able to do it, or do you prefer I do it myself?
If you'll do it, could you please update us about the issue number, so we can subscribe to it?
I still hope fixing this could be a temporary walkaround for this issue.
Thanks!

[mconnew]

We have investigated and discussed this in our team and identified 4 issues. Two with XmlSerializerGenerator (one of which nobody has reported hitting but it's in the same area of launching the tool), and two with XmlSerializer.
Due to lack of API support to generate a dynamic assembly in a specific ALC, we can't get runtime serialization working until .NET 5 when hopefully we'll have the API's available. We are going to fix the pre-generated serializer scenario case as there are api's available to control where an assembly is loaded. We'll update this issue with links to each of the issues we'll open so you can track the progress. The issues and initial changes will live in the new dotnet runtime repo as that's where new changes are made, but as soon as we've got things working we will back-port to the servicing branch(s) in corefx and release a servicing fix.

[curia-damiano]

Hi Matthew,
thank you very much for your information.
Can you please confirm if I understand correctly:

• ALC with XmlSerializer definitely won't work on .NET Core 3.1, we need to wait for .NET 5;
• you are working to make the XmlSerializerGenerator workaround to be possible on .NET Core 3.1 and we can wait for patches for this alternative approach.
  Is it right?

[mconnew]

@curia-damiano, you understand correctly. The issues have been opened. They show the milestone as 5.0 as that's the milestone that the changes will be targeting in the runtime repo. The appropriate ones will be back ported to 3.1, it's just that there's no 3.1 servicing branch in dotnet/runtime so that's not an appropriate milestone over there. We also added a 5th issue around the duplicate types error on 3.1.

Microsoft.XmlSerializer.Generator package does not install correctly as a tool.

Microsoft.XmlSerializer.Generator failing to compile when targeting netcoreapp3.0 or 3.1

Microsoft.XmlSerializer.Generator fails to generate serializer for assemblies not targeting netstandard2.0

Add support to specify the ALC in which to generate a dynamic assembly.

Allow a pre-generated serialization assembly to load in the same ALC the type is in.

[zengqinglei]

Hi Matthew,
thank you very much for your information.
Can you please confirm if I understand correctly:

• ALC with XmlSerializer definitely won't work on .NET Core 3.1, we need to wait for .NET 5;
• you are working to make the XmlSerializerGenerator workaround to be possible on .NET Core 3.1 and we can wait for patches for this alternative approach.
  Is it right?

I also encountered the same problem. Is there a temporary solution?
My application is running .net core 3.1, .netstandard2.1

[StephenBonikowsky]

@mconnew is there a workaround for @zengqinglei ?

[StephenBonikowsky]

@zengqinglei I reviewed your question with @mconnew and unfortunately there is no short-term workaround.

[cpkuo]

I'm having the same issue as @uffebjorklund and @zengqinglei. This is one of the last tasks in porting a Full Framework project. Is there a known alternative to XmlSerializer that will work with AssemblyLoadContext?

[MatthewLymer]

I'm having the same issue with our plugin system, a plugin may reference XmlSerializer and we can't load it into a collectable assembly context.

Is there an env-var or something that instructs XmlSerializer to behave better in these scenarios, perhaps at the expense of performance?

[KallDrexx]

Just encountered this as we tried to update our app to allow dynamic loading and unloading of plugins using the AssemblyLoadContext, and unfortunately this may put the whole effort on hold (as some existing plugins make use of the XmlSerializer).

It appears that the fixes for this is no longer targeting .net 5 but now .net 6, am I understanding that correctly from the corresponding issues?

[drocx]

Why has this bug not been fixed yet?
It has been known since September 2019 ... At that time the bug fix was moved to the .Net 5.0 framework. Now it is out and XML files still cannot be deserialized.

Here is a simple example that has crashed since 2019:

using System;
using System.IO;
using System.Xml.Serialization;

// This is the class that will be deserialized.
public class OrderedItem
{
    [XmlElement(Namespace = "http://www.cpandl.com")]
    public string ItemName;
    [XmlElement(Namespace = "http://www.cpandl.com")]
    public string Description;
    [XmlElement(Namespace="http://www.cohowinery.com")]
    public decimal UnitPrice;
    [XmlElement(Namespace = "http://www.cpandl.com")]
    public int Quantity;
    [XmlElement(Namespace="http://www.cohowinery.com")]
    public decimal LineTotal;
    // A custom method used to calculate price per item.
    public void Calculate()
    {
        LineTotal = UnitPrice * Quantity;
    }
}

public class Test
{
    public static void Main()
    {
        Test t = new Test();
        // Read a purchase order.
        t.DeserializeObject("simple.xml");
    }

    private void DeserializeObject(string filename)
    {
        Console.WriteLine("Reading with Stream");

        // Create an instance of the XmlSerializer.
        // NotSupportedException: A non-collectible assembly may not reference a collectible assembly.
        XmlSerializer serializer =
        new XmlSerializer(typeof(OrderedItem));

        // Declare an object variable of the type to be deserialized.
        OrderedItem i;

        using (Stream reader = new FileStream(filename, FileMode.Open))
        {
            // Call the Deserialize method to restore the object's state.
            i = (OrderedItem)serializer.Deserialize(reader);
        }

        // Write out the properties of the object.
        Console.Write(
        i.ItemName + "\t" +
        i.Description + "\t" +
        i.UnitPrice + "\t" +
        i.Quantity + "\t" +
        i.LineTotal);
    }
}

XmlSerializer.Deserialize Method