Runtime: System.Reflection.Emit.AssemblyBuilder.Save

[jonorossi]

AssemblyBuilder.Save and AssemblyBuilderAccess.RunAndSave isn't available in .NET Core, however coreclr seems to have the code to implement it but I looks conditionally compiled out.

https://github.com/dotnet/coreclr/blob/bc146608854d1db9cdbcc0b08029a87754e12b49/src/mscorlib/src/System/Reflection/Emit/AssemblyBuilder.cs#L1690-L1711

Our use case in Castle DynamicProxy is to write out dynamically created assemblies to disk so we can run peverify over the assembly in unit tests. It also greatly helps writing out the assembly and opening it in ildasm to manually verify IL.

Is there any chance we can get this functionality that .NET Framework has introduced into .NET Core, or an alternative way of doing the same thing. This might not sound like a big deal if you aren't familiar with Castle DynamicProxy's internals, but it really is as it heavily reduces our confidence in both our code and the .NET runtime as well as reducing our ability to track down defects.

[joshfree]

Related issue https://github.com/dotnet/coreclr/issues/1709

[bartdesmet]

Another interesting omission is the lack of LambdaExpression.Compile accepting a MethodBuilder which tends to be very useful when combined with AssemblyBuilder.Save support.

[jkotas]

Large chunks of Reflection.Emit code in the runtime are Windows-specific. It is not easy to just enable them for CoreCLR because of it would not work on Unix. In particular, the support for emitting debug information depends on unmanaged PDB writer that is very complex Windows-specific component that we have no plans to open source and bring x-plat.

We are also not happy about Reflection.Emit being integrated deeply into the runtime. Reflection.Emit should be a independent higher level component that generates PE file that the runtime loads as any other assembly (note that runtime supports loading assemblies from byte array, so the PE file does not have to be persisted on disk). IKVM.Reflection.Emit done by Jeroen Frijters proved that it is pretty doable and can actually work much better than the current Reflection.Emit built into the runtime.

The plan to address the Reflection.Emit is:

1. (Done) Create fast standalone managed metadata, IL and portable PDB writer that can be used by Roslyn, and anybody else to produce managed assemblies. @tmat has been working on it: https://github.com/dotnet/roslyn/pull/7683.
2. Build Reflection.Emit object model as a layer on top of the standalone managed metadata writer to make it easier for existing projects that use Reflection.Emit to migrate to .NET Core.

[jonorossi]

@jkotas many thanks for the explanation here, it makes complete sense. I'm glad you guys are planning to provide a more general purpose object model around the heart of .NET, IL. I'd seen the portable PDB reader/writer and some metadata object model appear in Roslyn, they look great. In the past I've used Cecil for the exact reason the .NET Framework API is so heavily tied into the runtime, glad to see more stuff being pulled out of the runtime.

[tmat]

FYI @ManishJayaswal @jaredpar

[karelz]

Next step: We need API proposal with deep analysis how implementable it is. The old Desktop API might be an option, likely not a good one.
This is quite hard to do and requires lots of experience.

[KevinRansom]

@karelz @jkotas --- What is the status of this feature? When can we expect to see dynamic codegen with persistence on the coreclr?

[karelz]

There is currently no plan (milestone Future + my comment above) - area owners @AtsushiKan @DnlHarvey @joshfree can provide more details.

[stakx]

@AtsushiKan @DnlHarvey @joshfree, one question about the plan that @jkotas laid out above:

We are also not happy about Reflection.Emit being integrated deeply into the runtime. Reflection.Emit should be a independent higher level component that generates PE file that the runtime loads as any other assembly [...].

The plan to address the Reflection.Emit is:

1. [...]
2. Build Reflection.Emit object model as a layer on top of the standalone managed metadata writer to make it easier for existing projects that use Reflection.Emit to migrate to .NET Core.

I do not fully understand the plan, in particular with regard to dynamic assemblies (AssemblyBuilder.DefineDynamicAssembly) and their ability to be incrementally updated / augmented with new dynamic types. How would this still be possible if assemblies must first be fully written out as a byte stream or file, then be loaded as a whole?

I imagine one would have to write each dynamically generated type to a single assembly... but wouldn't this in turn interfere with internal type visibility and type member accessibility between generated types that should all be in the same assembly, and with stuff like [assembly: InternalsVisibleTo("DynamicallyGeneratedAssembly") (on which e. g. DynamicProxy currently depends to be able to proxy internal user types)?

Would it be possible to load more than one assembly having the exact same name (whether weak or strong), so that each such single-type assembly can have a [assembly: InternalsVisibleTo] attribute for its own name, such that types in the separate assemblies can reference each other's internals?

[jkotas]

their ability to be incrementally updated / augmented with new dynamic types

Yes, there would need to be a runtime hook for this.

Conceptually, each type addition in Reflection.Emit produces a new assembly that contains all existing metadata (with all existing tokens preserved) plus the new type. Then the runtime is asked to update the metadata of the already loaded assembly to the new version.

Sending the whole assembly over for each edit is not efficient for large assemblies with small edits. It would be better to just send the metadata delta with the new stuff over. Fortunately, this exists in the runtime for edit and continue already. Look for https://github.com/dotnet/coreclr/search?q=ApplyEditAndContinue.

So exposing the ApplyDelta metadata API for the standalone Reflection.Emit implementation should do the trick here.

[masonwheeler]

Large chunks of Reflection.Emit code in the runtime are Windows-specific. It is not easy to just enable them for CoreCLR because of it would not work on Unix. In particular, the support for emitting debug information depends on unmanaged PDB writer that is very complex Windows-specific component that we have no plans to open source and bring x-plat.

The existence of cross-platform Roslyn suggests that this isn't a particularly intractable problem, at a high level at least.

Could this please be made a high priority? The inability to use Reflection.Emit is currently my dotnet/corefx#1 blocker preventing me from migrating existing work to Core, and as such it's far more urgent, from my perspective at least, than the development of new features.

[zgramana]

Here's a good starting point for a non-Windows implementation (and it's rather mature): https://github.com/mono/mono/tree/master/mcs/class/corlib/System.Reflection.Emit

[stakx]

@AtsushiKan: Out of curiosity, is there any documentation / spec at all about the data format used for EnC metadata deltas, other than the actual CoreCLR source code?

[ghost]

Out of curiosity, is there any documentation / spec at all about the data format used for EnC metadata deltas, other than the actual CoreCLR source code?

Not that I've heard of but EnC is not an area where I have any knowledge.

[stakx]

@AtsushiKan - Thanks for the quick reply. :+1: I almost expected this to be a really arcane area, but I thought I'd give it a try and ask. I'll see what I can gather together on my own.

[karelz]

@stakx EnC MetaData format was never thoroughly documented (neither publicly nor internally). It was always defined just in the implementation.

[tmat]

@stakx Curious, what do you need it for?

[stakx]

@karelz - I almost suspected so... Thanks for this info! 👍

@tmat - Mostly out of curiousity.

I have spent a lot of time with System.Reflection in the past few months, and noticed its various limitations or bugs. There are sometimes situations where you'd rather just see plain, uninterpreted metadata instead of the semi-high-level view that Reflection provides—which is neither original metadata, nor as high-level as e.g. C#, but something of its own sitting somewhere in-between.

I have also discovered System.Reflection.Metadata (SRM). I have been thinking about how nice it would be if the (Core)CLR had the necessary runtime hooks and extension points so that both Reflection and Reflection.Emit could be rebuilt on top of SRM, outside of mscorlib / System.Private.CoreLib. (At the same time, this would also enable folks to build their own custom reflection / dynamic code emission libraries.) Two things appear to be missing as of today to make this possible:

• Neither the CLR nor the CoreCLR runtimes allow user code to get at the metadata stream & IL of loaded assemblies/modules, so it could be inspected using e.g. SRM. (This assumes that the metadata as per ECMA-335 is still available unaltered after a module has been loaded by the CLR, which likely isn't the case.)

• Neither the CLR nor the CoreCLR runtime allows altering/updating an assembly that's been loaded via Assembly.Load(byte[]) (which is what you can use today to load an assembly generated with SRM). Perhaps if EnC was exposed in some way, updating a loaded assembly would become possible.

I realise that having such runtime hooks closely allied with System.Reflection.Metadata is probably a pipe dream at this stage. But it might allow some super-interesting scenarios. So I'm trying to learn how this part of the runtime works to better gauge how big of a project this would be.

If there are any plans or efforts in this direction, I'd love to learn about it & be directed to the appropriate places. Otherwise... I don't intend to hijack this issue, which is just about AssemblyBuilder.Save. :wink:

[jkotas]

Neither the CLR nor the CoreCLR runtimes allow user code to get at the metadata stream & IL of loaded assemblies/modules, so it could be inspected using e.g. SRM

Have you seen https://github.com/dotnet/coreclr/blob/85374ceaed177f71472cc4c23c69daf7402e5048/src/System.Private.CoreLib/src/System/Reflection/Metadata/AssemblyExtensions.cs#L25 ?

[stakx]

@jkotas - Yes, I have, and that's a really nice extension point! But it comes with some limitations: It's a Try... method and so may fail, e.g. for dynamically generated assemblies (AssemblyBuilder); something that today's Reflection (IIRC) can deal with just fine. So it seems Reflection couldn't be built on top of that to work as reliably as it does today. But I was definitely thinking of something like this. :+1:

[vermorel]

We have finally rolled our own open-source version of AssemblyBuilder.Save based on System.Reflection.Metadata (and no further dependencies). It targets .NET Standard and works under .NET Core. It does support dynamically generated assemblies.

Check-out https://github.com/Lokad/ILPack Feedback will be most welcome.

[masonwheeler]

@vermorel Looks interesting. Does it generate PDBs too?

[vermorel]

@masonwheeler ILPack does not generate PDB.

[arlm]

We have finally rolled our own open-source version of AssemblyBuilder.Save based on System.Reflection.Metadata (and no further dependencies). It targets .NET Standard and works under .NET Core. It does support dynamically generated assemblies.

Check-out https://github.com/Lokad/ILPack Feedback will be most welcome.

I tried it on a Sigil branch of mine ( https://github.com/arlm/Sigil-vNext ) nad could not make it work because it does not install on .NET Standard, only on .NET Core solutions.

[AArnott]

I tried the Lokad/ILPack project and hit a couple of blocking issues.

[AArnott]

We need this in VS as well, as we have several libraries that use Ref.Emit and debugging such code during development is absolutely vital. We debug it primarily by writing the dynamic assembly out to disk then using peverify and ILSpy on it.

[tmat]

@AArnott Any reason not to generate .cs files and use Roslyn to build the assemblies?

[AArnott]

Yes, a few reasons @tmat:

1. Perf
2. we generate one type at a time, on demand. We don't want to compile a distinct assembly every time we need to generate a proxy. Dynamic assemblies let us add types without creating a new assembly.
3. We absolutely need to access non-public members of assemblies our dynamic code references. C# won't let us do that, but the CLR will if we write our own IL.

[tmat]

Makes sense. I assume this is for RPC proxies? Would it make sense to pre-generate them as part of VS build (e..g in VS SDK build task)?

[AArnott]

Yes, RPC proxies is the main scenario, but so is generation of serializers in MessagePack.

Pregenerating them is something we're likely to look at, but only as an optimization. We don't want to require that.

And anyway, pregenerating or not doesn't solve our problem that we access non-public members so C# won't do, therefore we use Ref.Emit, and therefore we need to be able to debug our output and therefore need AssemblyBuilder.Save. :) Q.E.D.

[tmat]

The non-public access can be solved. We don't want to expose such option to csc.exe but an API can be added to support the proxy generation scenario. We have other scenarios in C# scripting where this is needed.

The benefit of requiring it would be that you have just one code path to maintain. What's the downside of requiring it? If it is a default build step in VS SDK that is enabled by default, it can "just work".

[AArnott]

I think we should discuss whether we pregenerate or JIT-generate these proxies on another thread, as IMO it's orthogonal to this issue.

[steveharter]

Based on discussion with @AArnott and @GrabYourPitchforks this is not a must-have feature for 5.0. However, this feature is completely valid and important but since it is not on the roadmap for 5.0 this is being moved to Future.

[jmglogow]

I'm currently porting LibreOffice (LO) to Windows Arm64. LO offers a controlling API / RPC service with various bindings, one of it .NET / C# on Windows. Currently it's based on .NET Framework 4 with a C++/CLI implementation + generated assembly code.

The only option for a native implementation on Arm64 is - AFAIK - to use .NET 5.
The LO API is described in an IDL with an extensible type system, so there are binding specific tools to convert LO's IDL into "code". For .NET this tool is called climaker, which uses this .NET interface to generate the assembly code (https://api.libreoffice.org/docs/tools.html#climaker / https://github.com/LibreOffice/core/tree/master/cli_ure).

The port is in early development and rather experimental, so I currently disabled building the .NET bindings for Arm64, but LO will need a way to generate assembly code again. Or implement some completely new approach with dynamic generated runtime assemblies. All the other approaches I tried, didn't work, but I have never used .NET before and might simply be missing some info to implement a different working solution.

FWIW: more information on the port, including my various approaches on using .NET 5, is in https://bugs.documentfoundation.org/show_bug.cgi?id=137143
P.S. LO has it's own build system and could probably integrate ILPack, either by calling MSBuild or integrating the sources in the build, but I would like to avoid that.

[weltkante]

There are different libraries (including third party) that can be used to generate IL code by generators, you don't have to use the reflection-emit API which is limited through various historic constraints in what it can actually emit. Especially if you do offline IL generation as part of the build step I wouldn't chose reflection-emit, these days this API mostly exists for generating executable code into the current process, the offline generation usage has been gotten a replacement already.

The modern framework API for writing offline metadata is MetadataBuilder in System.Reflection.Metadata which is what compile-time tools should be using.

[isral]

The modern framework API for writing offline metadata is MetadataBuilder in System.Reflection.Metadata which is what compile-time tools should be using.

I try to create a compiler (for hobby, not job).
I use reflection emit. I decided to migrate to .NET 5 only later realize I can't save assembly.

There is no documentation on MetadataBuilder. MSDN only show method's parameter list without explanation at all.
I can't find tutorial on MetadataBuilder.

Do you have any suggestion?

[tomhicklingnc]

@isral I have used https://github.com/Lokad/ILPack before in .NET Core projects, it's very easy to use and has worked fine for what I needed it for.

[jinek]

Guys, just to understand. Isn't it possible to save assembly in dotnetcore currently out of the box? It's just I've been hitting missing parts since 1.0.
I also don't understand the problem. .NET compiler works somehow both on Windows and Linux.

[AArnott]

@jinek The problem is that existing .NET APIs under System.Reflection.Emit that can be used to generate dynamic assemblies works on .NET Core except for saving the dynamic assembly to disk. No 3rd party solution addresses this problem as it would require rewriting all your Ref.Emit code to use the new library. Even if this could be done, The Ref.Emit APIs are exclusive in being able to add members to a dynamic assembly after it has already been loaded.

[isral]

@jinek
.net (core) 5 also doesn't have emit methods related to debug information.
https://github.com/Lokad/ILPack mentioned above also can not set entry point.
If you don't need entry point and debug information, give ILPack a try.

[davidfowl]

This would be super useful for debugging, for my usage, it's used to dump the IL to disk and open it in ILSpy 😄

[steveharter]

(updated based on feedback)

One approach for 6.0 is to unblock just the diagnostic scenarios (per @AArnott and @davidfowl) where the emitted IL can be saved to a file, and then opened with PEVerify, ILSpy, or decompiling the saved IL to C# and inspecting or use VS to debug. This could mean:

• Address issues in ILPack preventing common scenarios from working.
• A lighter-weight API (out-of-band) that may only save for diagnostics (not runnable) such as from runtimelab.

The requirements for the full approach (in rough priority order):

• Add AssemblyBuilder.Save and the corresponding enum value AssemblyBuilderAccess.RunAndSave.
• Support all platforms (not just Windows).
• Add APIs to emit debugging\symbolic information.
• Ability to use Reflection.Emit against reference assemblies to help prevent versioning issues in certain scenarios.
• Add APIs to support EnC (Edit-and-Continue) on dynamic assemblies. These are new APIs and also do not exist in .NET Framework.
• Add AssemblyBuilder.SetEntryPoint and other missing APIs that are necessary for specific scenarios.

[jkotas]

EnC

This is internal implementation detail from AssemblyBuilder.Save point of view. I fully agree that it would be a very good idea to build Reflection.Emit as a layer over EnC , but I would not call it a requirement.

Add APIs to emit debugging\symbolic information.

We have requests to add symbols in other issues. Symbols are more important than AssemblyBuilder.Save for some of the users.

Support all platforms (not just Windows).

It would look pretty bad to have Ref.Emit Save on Windows only given our cross platform promise.

I would add more requirement: Ability to use Reflection.Emit against reference assemblies. Otherwise, we will end up with very bad versioning problems when Reflection.Emit is used to build compilers. These versioning problems turn into migration blockers to new .NET Runtime and SDK versions. We had this in .NET Framework and it was not pretty.

[steveharter]

Thanks @jkotas for the feedback. I updated the requirements above.

[steveharter]

If anyone is truly blocked by the issue for 6.0, please describe the scenario and what requirements from https://github.com/dotnet/runtime/issues/15704#issuecomment-730528965 are needed, and what is missing.

This is a very large feature and we need to justify the cost : benefit for 6.0.

[perfolenta]

My project (compiler) is based on Reflection.Emit and I cannot translate it to Net 5 due to the lack of the Save method.
I don't think I'm the only one.
There is no strength to rewrite the entire project, because wrote it for several years.
I hope that at least in 6.0 it will be possible to save a ready-to-run assembly to disk.

[slozier]

In the IronPython project, it's used to save compiled Python code to an assembly (to run and/or debug). However it also relies on LambdaExpression.CompileToMethod (https://github.com/dotnet/runtime/issues/20270) which was never brought over from .NET Framework so it's unclear if/when it'd be able to use AssemblyBuilder.Save again.

[isral]

* Add `AssemblyBuilder.Save` and the corresponding enum value `AssemblyBuilderAccess.RunAndSave`.

Is it possible to have collect and save and run?
For language server in vs code that compile every user type, is collect needed here?
I don't know about debug or run in vs or vs code.

BTW LocalBuilder.SetLocalSymInfo in .net472 doesn't work in VS 2019 (open exe file).

[ahartmanns]

If anyone is truly blocked by the issue for 6.0, please describe the scenario and what requirements from #15704 (comment) are needed, and what is missing.

We use Reflection.Emit in a simulation tool to compile models into IL that implements a specific interface, which our tool then calls to do the simulation runs. The lack of AssemblyBuilder.Save currently blocks the following for us:

• Debugging during development: We cannot save the IL we generate to file in order to run ildasm or peverify equivalents on it. This is a major nuisance that slows down our development significantly, but not a blocker.
• Distributed simulation: To run on multiple machines or a cluster, we compile the code on the master machine, write it to file using AssemblyBuilder.Save, read the bytes of the file back, transfer them to the slave machines, and load the assembly there (via Assembly.Load(byte[])). Then every machine can simulate the same model code and transmit results back to the master. Lack of AssemblyBuilder.Save completely blocks this scenario; we currently need to revert back to an old version running on .NET Framework/Mono for distributed simulation.

Thus what we'd need from https://github.com/dotnet/runtime/issues/15704#issuecomment-730528965 is support for AssemblyBuilder.Save and AssemblyBuilderAccess.RunAndSave on all platforms (Windows, macOS, Linux).

We don't _need_ debug info etc., but it could be helpful for the development part (if it gets us nice variable names in ildasm, for example). We tried ILPack (some time ago), but it was buggy and incomplete (at that time); we generate apparently "complex" code with generic types parameterised by value types with interface constraints on interfaces that are in our main assembly (not emitted) etc. that appeared to trigger lots of scenarios not expected by ILPack.

[jmglogow]

I was told to port the LibreOffice's climaker (just ~2.5k LOC) code to MetadataBuilder in System.Reflection.Metadata in this thread, instead if using this API. I found the documentation a btt lacking with quite a few "Blob" parameters. I'm not really proficient with .NET, so this might be mainly my problem. But it seems that a lot of (equivalent) helper classes, like Emit::FieldBuilder or Emit::ConstructorBuilder, aren't available in the MetadataBuilder API. In the end I came to the conclusion, the result would be a lot like re-implementing ILPack; now having to account for a lot more stuff myself, which was previously done by the AssemblyBuilder interface. Also stuff like creating a type derivated from some existing System type seem rather tedious and the ILPack code also seems to explicitly ignore some cases (I can't remember exactly - this was two months ago).

Maybe a "less flexible, but easier" API on top of MetadataBuilder would help, if people are assumed to use this API now for their previous System.Reflection.Emit.AssemblyBuilder.Save use cases.

I guess at some point someone (me?) has to bite the bullet and do the .NET 5.0 port, not just for Arm64, but currently I decided I don't have the time and it's not clear, if this is really the best way implement a new climaker (also as a DLL run by the dotnet.exe and it uses LibreOffice C++ libraries, so already depends on native code).

[jonorossi]

As stated in the issue description:

Our use case in Castle DynamicProxy is to write out dynamically created assemblies to disk so we can run peverify over the assembly in unit tests. It also greatly helps writing out the assembly and opening it in ildasm to manually verify IL.

With .NET Core/5 getting new runtime features that aren't in .NET Framework we've skated by testing some of them using their underlying representation in IL via attributes and other metadata, but we are not performing the same testing using the .NET Core/5 runtime that we are with .NET Framework's runtime.

We've got just about all the mocking libraries and other projects including Entity Framework depending on Castle DynamicProxy, but as the years pass the .NET Core/5 implementation diverges from .NET Framework so the bar to make changes to DynamicProxy continues to gets higher to remain confident in the generated IL.

@steveharter if you've got any specific questions, I'm happy to answer them.

[masonwheeler]

My project (compiler) is based on Reflection.Emit and I cannot translate it to Net 5 due to the lack of the Save method.
I don't think I'm the only one.

You're not. This is a very noticeable omission in the API for exactly that reason.

[bartdesmet]

We've been affected by the lack of Save as well (and CompileToMethod in expression trees) for a high-density large scale event processing system. I'll sketch the basics of it.

The service hosts standing queries which are shipped from clients to services in the form of some query language that gets translated down to .NET expression trees which get compiled and evaluated to instantiate the query. A typical service replica hosts a few 1000s of these, and each service host process runs 100s is service replicas, so we're seeing up to a million queries per process. These queries can run for seconds to several years (literally!).

To achieve reliability, we employ a (differential) checkpointing strategy whereby the queries (as well as runtime state, e.g. a query may compute an average which requires storing a sum and a count) get persisted such that they can be rehydrated upon a failover onto a different node in the cluster. The base implementation of this stores the expression trees in a normalized form using an expression tree serializer framework we built.

From a performance point of view, deserializing and recompiling expression trees is quite costly and is a significant cost for recovery (it involves a lot of object allocation, going through expensive reflection code paths to type check the expression trees behind the factory methods, invoking compilation, and then the whole native side of JIT). This gets mitigated in a few ways, including detecting common subexpressions, generating templates by hoisting out constants, etc.

However, to get to the next level of performance, we built an "always on" system that leverages saving compiled expression trees to disk (as well as auxiliary types built using Reflection.Emit). It's based on a generational assumption akin to garbage collectors. As a standing query gets checkpointed (say every minute), its age is increased. When it reaches a certain age and a certain number of queries have accumulated in an age group, we go through the motions of compiling all of these into an assembly. Upon recovery, we simply reload the assemblies of these older generations (with a goal of it covering > 95% of the queries), and only recover the youngest generations using expression tree deserialization.

For older queries that got deleted, we keep a tombstone table (used to prevent recovering them), and when that table exceeds a certain percentage, we compact the generation (we still save the original expression trees to cold storage, so we can get them back and apply new rounds of common subexpression elimination with newer ones, prior to recompiling them into a MethodBuilder) and coalesce it with younger generations (so upon a future recovery, the smaller compacted assembly gets loaded instead).

In this scheme, recovery is simply loading a few assemblies, discovering all query methods in it (using reflection), excluding the tombstoned items, and invoking the methods that instantiate the query operator graphs. It cuts out all off the expression deserialization, factory invocations, reflection emit, etc. There's an obvious tradeoff in that these assemblies cannot be unloaded so queries that get deleted remain resident (as IL as well as JITted code), but queries tend to live long, create/delete churn is limited, and services rebalance frequently causing the primary replicas to move to other hosts, causing a recovery. Each such recovery will see new assemblies due to the collection and compaction logic we run periodically in the background, so the alive/dead ratio is kept at bay.

The key missing pieces to re-enable this mode are Save and CompileToMethod. Debug info generation would be a nice to have given that we did implement an expression tree instrumentation mechanism using DebugInfo nodes to emit sequence points against a ToString/DebugView like pretty-printed form of expression trees. We have used this in certain environments to store a .exp expression tree "human-readable code file" alongside these persisted compiled expressions, so we can get some symbolic info (file, line, column) when something blows up inside a query expression being evaluated.

[bartdesmet]

One more thing. Just a few days ago I did have to fall back to .NET 4.8 to figure out why a piece of Reflection.Emit generated code was causing an InvalidProgramException. The inability to Save the assembly to disk and run it through PEVerify.exe or open it in ILSpy was what caused me to switch back temporarily. (This was for a different project as the one mentioned above.)

[leppie]

Author of IronScheme. I use Save to persist libraries to disk (at almost zero effort to me). RE plays a dual role as IronScheme is an incremental compiler. Also it would be silly to have another backend to emit IL, they are never going to be the same, nevermind the huge effort.

Also to note, IronScheme is a single DLL that runs on .NET 2+ and .NET Core 2.1+, MS/Mono, any OS, any bitness, any CPU (even on an ARM TV). So getting a solution for .NET Core would involve backporting it to .NET too. Again, too much effort.

[mjsabby]

@steveharter do you know why this would be very large? Is it the sum of all parts need to be delivered in one release? Could we not bring back the implementation that existed?

[jkotas]

The implementation that existed does not fulfill these requirements from https://github.com/dotnet/runtime/issues/15704#issuecomment-730538328:

• Support all platforms (not just Windows).
• Ability to use Reflection.Emit against reference assemblies to help prevent versioning issues in certain scenarios

[mjsabby]

Which all reference assemblies? The ones that ship in box?

Has a design been thought of for the second point? That seems like the most work at least in my head.

To be able to do that we'd need some way of being able to take live-types and walk their dependencies in a different load context where the ref assemblies are? And if we say no that's not possible then users would have change load these types in the metadata load context where the ref assemblies would be, and while it would still be a lot less work by the developers, it wouldn't be as turn key.

Bringing back the old code has the benefit of it works, but doesn't solve the versioning problem which admittedly is a lot worse on .NET Core/.NET X+ although it is something of value if people care. And the Windows-only part (PDBs I imagine?) could be made optional or something.

Yeah no good answer, just thinking out loud to see if there would be any interest in these lesser solutions.

[masonwheeler]

The implementation that existed does not fulfill these requirements from #15704 (comment):

• Support all platforms (not just Windows).
• Ability to use Reflection.Emit against reference assemblies to help prevent versioning issues in certain scenarios

Fair enough. But it did actually work, within the parameters of what it was designed to do. That makes it a good starting point. Could that code be added in, perhaps on a branch, to give the community something to iterate on?

[jkotas]

The code is included in https://github.com/dotnet/coreclr/commit/ef1e2ab328087c61a6878c1e84f4fc5d710aebce. You can cherry pick it from there if you believe that it has future. I believe that trying to iterate on that code is a dead end.

[steveharter]

To recap the current status for those wanting to use AssemblyBuilder.Save():

Currently there are no concrete plans to add this for 6.0; currently evaluating cost:benefit.

Workaround include:

• Continue to use .NET Framework (desktop).
• For static scenarios where compilation can occur ahead-of-time:
  
  
  
  • Roslyn.
  
  
  • Supports outputting debugging information (PDB files).
  
  
  • Since used internally by the C# compiler, supports newer language features.
  
  
  • An area of continuing investment including Source Generators.
  
  
  • System.Reflection.DispatchProxy
  
  
• https://github.com/Lokad/ILPack
  
  
  
  • A recent PR unblocked ILPack for @AArnott diagnostic scenarios.
  
  
  • Diagnostic tools can then be run on this saved IL including:
  
  
  • PEVerify
  
  
  • https://github.com/icsharpcode/ILSpy
  
  
  • Dynamic debugging \ decompile is also possible with third-party tools.
  
  

[AArnott]

@steveharter Perhaps updated your guidance above to replace PEVerify with ILVerify. It turns out PEVerify only works on .NET Framework dll's. For .NET Core dll's, ILVerify works. See https://github.com/Lokad/ILPack/issues/131 for more.