Edge: .NET Core RC2 support

[lstratman]

Putting this in here as a placeholder to track support for RC2 against. Microsoft changed a ton of stuff with regard to building and assembly loading between RC1 and RC2, so it's going to take me a bit to untangle everything. I'll post additional information/blockers as I progress.

[tjanczuk]

Perfect, thank you!

[lstratman]

Good progress so far, no real blockers. Despite Microsoft making huge changes between RC1 and RC2 to the way that the framework directory and packages are resolved, the managed and unmanaged code has been updated, compiles on Windows, and passes all of the pre-compiled (i.e. non-edge-cs-based) tests. As part of this support, I also copied over the PAL code that Microsoft uses in https://github.com/dotnet/cli, which has almost completely eliminated the platform-specific #ifdefs that I was using previously.

Dynamic compilation is up next and then a bunch of refactoring and fit-and-finish work and testing on the other OSes. All of this work is in https://github.com/medicomp/edge/tree/RC2. I'm traveling this weekend and then have my company's user group conference all next week, so I should be able to pick it back up the week after next.

[tjanczuk]

Wonderful, thank you for all the hard work.

[lstratman]

Dynamic compilation is in now and all tests are green on both Windows and Linux. I'm going to take care of OS X (which shouldn't involve too much work) later this week. I'm also going to make sure that standalone apps (ones with their own copy of the CLR via dotnet publish) can also be used with Edge.js.

The fit-and-finish work is also done, but I may propose a change to the way that CoreCLREmbedding.dll is referenced and loaded. It would be a breaking change, but may result in a cleaner codebase in CoreCLREmbedding. It would also remove the requirement that the .NET Core SDK be installed: it would be possible to use Edge.js with only standalone (i.e. published) .NET Core apps, which could be very useful for folks who don't want to install the CLI on their production systems. Anyway, I'm going to do a bit more research and then I may write something up.

[tjanczuk]

This is fantastic, looking forward to it. I will probably want to pile up a few long outstanding fixes onto this release, and also add support for Node 6. Just FYI.

[lstratman]

OK, OS X support is done and all tests are green there as well. There is one important caveat: I had to up the MACOSX_DEPLOYMENT_TARGET value to 10.7 and use libc++ as the standard library. This is due to the fact that the PAL layer that I'm using from Microsoft's dotnet CLI codebase uses some C++11 features that are either not supported or different in the default gcc compiler (4.2.1) that ships with xcode. It _might_ be possible to get this compiled and linked for older deployment targets using more recent versions of gcc from homebrew, but someone with more OS X dev experience than myself would need to take a look at that.

Next, I'll be enabling support for dotnet published apps and updating the documentation. After that, this should be ready for a PR.

[tjanczuk]

Sounds good. I don't think the C++ 11 on OS X requirement is an issue, not many people use edge on OS X beyond doing development like you and me.

[gingters]

Just wanted to tell you guys to keep up the good work. I'm eagerly waiting to be able to use dotnet published tools from within an Electron app via Edge.

[Konard]

I also wait for .NET Core RC2 support to be able to make a high quality bridges between https://github.com/linksplatform/Crawler and https://github.com/Konard/LinksPlatform

[lstratman]

@gingters and @Konard, thank you for sharing your plans, it's very useful in determining what I should be including in .NET Core support. Given the interest (from at least one person) in being able to use dotnet published apps, it may be useful to hash out and implement the alternate way to load the Edge.js "managed runtime" that I mentioned earlier.

At issue is the fact that, in my current implementation, the dotnet CLI and command line tools still need to be installed on servers that will be using Edge.js, even if you're only going to ever be running dotnet published applications. This is due to the fact that we are currently shipping the Edge.js runtime (CoreCLREmbedding.dll) and its list of dependencies (project.json) with the actual NPM package. When Edge.js is installed by NPM, it runs dotnet restore to bring down the dependencies that it requires (Newtonsoft.Json, Microsoft.CodeAnalysis.CSharp if you're doing dynamic C# compilation, etc.) into the local NuGet package cache. So, when you run Edge.js, we merge your application's .deps.json file with the .NET runtime's shared .deps.json (Microsoft.NETCore.App.deps.json, this only happens if you are running an app that is not dotnet published) and with the Edge.js runtime's .deps.json (generated from its project.json) to generate the full list of dependencies that we need to load when executing the app.

This sort of violates the principal of dotnet publish: you're supposed to get a fully self-contained .NET application, including the runtime and all dependencies, that you can then run on your server without having to have the .NET SDK or CLI toolchain also installed there. The solution is to ship the Edge.js runtime (CoreCLREmbedding.dll) not with the NPM package, but instead implement it as a NuGet package (something like EdgeJs.Runtime) that you reference in your .NET project's package.json. That way, when you run dotnet restore for your own application prior to building or publishing it, the CLI toolchain will pull down all the required dependencies for Edge.js and the Edge.js runtime itself and, if you publish your application via dotnet publish, all of those Edge.js dependencies and the runtime will be included in the set of published assemblies that you would then copy up to the server.

Pros:

1. No need to have the .NET SDK or CLI toolchain installed on the server that's ultimately going to be running Edge.js.
2. Going along with the above, there's no need for the NPM install process to run dotnet restore.
3. Significantly simplifies the code that is needed to get the paths to all of the application's dependent assemblies at runtime: Edge.js will ONLY ever have to look at the shared runtime's .deps.json file (if you didn't dotnet publish) and the application's .deps.json file. It would never have to merge in its own dependency manifest.
4. dotnet published apps get a true, self-contained distribution package.

Cons:

1. Requires action on the part of developers: .NET projects that you want to use with Edge.js will need to reference this still-to-be-created EdgeJs.Runtime NuGet package.
2. Requires additional effort from me :).

Please chime in with your thoughts or comments.

[tjanczuk]

@lstratman I want to make sure you are aware there is already a NuGet package for Edge.js: https://www.nuget.org/packages/Edge.js/, which is the primary way in which Edge.js can be used to enable scripting Node.js from within _desktop CLR_ apps _on Windows_? I believe there is a way to make the construction of this NugetGet package platform and CLR-flavor agnostic, but when I looked at this some months ago it was lacking any form of documentation beyond talking to @davidfowl on Twitter.

The NuGet package is built from https://github.com/tjanczuk/edge/blob/master/tools/build_double.bat

[lstratman]

@tjanczuk Yep, and we can either merge the code for CoreCLREmbedding.dll in with that or create a new NuGet package with just the stuff from CoreCLREmbedding.dll. The former option would be less confusing when people are searching NuGet for the package to install so I imagine that's the route that we'll take.

[tjanczuk]

I agree, having a single Nuget package that covers all platforms is preferable. My head hurts every time I come back to CLR after a break. The entire which-runtime-needs-what-dependencies-on-what-platform-using-what-tools is confusing enough to confuse people with a further choice of NuGet package flavor.

Are you saying the Nuget package for CoreCLR would have to contain the _source code_ of CoreCLREmbedding? The Nuget for full CLR contains a precompiled assembly.

[lstratman]

No need for any of the source, just the precompiled assembly.

[lstratman]

This actually ended up being much more straightforward than I anticipated. I've gotten the initial implementation fully working on Windows and will test it on all of the other OSes next week. Once everything checks out, I'll do a writeup on what I did in this thread, update the docs, and then this should finally be ready for a PR.

[gingters]

Just a few more information about our use-case:
We want to dotnet publish a library, that should run all platforms defined in its project.json. It would be no problem at all from our side to reference an additional edge.js runtime package in our library to have this packaged in a way that it will run via edge.

The idea is that we take the output of the publish process of our library, and just copy that 1:1 into our Electron application. In the best case we would simply include the folder in an asar package. In our electron application we use the electron-edge npm package (as we need to use that as the normal edge won't run with the electron-modified node, as far as I understood that) to load our .NET Core library and call methods there.

It would be great, if (electron-)edge would be able to automatically detect which runtime to load (regarding the platform we are currently at) to load and execute the library, so that I as a developer would just have to point it to the root folder of the published application and it would fetch the correct platform itself.

[lstratman]

This is finally DONE. Or at least I am declaring it done :+1:

What was formerly CoreCLREmbedding.dll that was in each of the OS/architecture/Node.js version directories under lib has been officially integrated into the Edge.js NuGet package. This NuGet package must be referenced in your project.json dependencies list. tools/build_double.bat will build this NuGet package and output it to the same place that it does currently. The change is that the actual package itself (which includes both the .NET 4.0 and .NET Core assemblies) is built with dotnet pack. So, any changes to the package metadata should be done in src/double/Edge.js/project.json; tools/nuget/edge.nuspec isn't used anymore, feel free to delete it. For the moment, the .NET 4.0 assembly in the NuGet package will contain the "scripting Node.js from .NET" functionality and the .NET Core assembly will contain what was formerly CoreCLREmbedding.dll, but once we get "scripting Node.js from .NET" working in .NET Core, that assembly will contain that functionality as well.

All of the compilers that we want to support on .NET Core should also have NuGet packages for them; I have updated my fork of edge-cs to do this and will send you a PR for it.

The requirement that the .NET Core SDK and CLI be installed on servers running Edge.js has been removed. The NPM install process on non-Windows boxes will now always compile edge_coreclr.node since it no longer looks at the presence of the dotnet executable as a gate for whether to build the .NET Core functionality or not. If the .NET Core SDK and CLI are present, the NPM install process will build a small .NET Core bootstrap application that will enable to jump into node directly and start dynamically scripting C# code without a project.json (satisfying #351). You can also use project.json applications, just like on RC1. If the SDK and CLI are not installed, however, you will only be able to run dotnet publish'ed applications, where the entire runtime and all of the dependencies are packaged with the application itself.

With the requirement that a NuGet package be referenced in order for a .NET Core application to work, @tjanczuk can either build and push the NuGet packages with a pre-release version or you can install .NET Core first and build the package using tools/build_double.bat, copy the resulting .nupkg file to a directory somewhere, and then update your user's NuGet.config file to point to this local directory as an alternative package source.

I've updated the documentation with all of the stuff for .NET Core that I could think of, but if I've missed anything or you want me to elaborate on something, please let me know. In addition, the Docker images still need to be updated: I didn't do anything with them.

Feel free to cut loose with any comments or questions. Otherwise, @tjanczuk let me know which branch you would like me to submit my PR under.

[tjanczuk]

Woo hoo! This is great work @lstratman! Can you submit a PR against master?

Let me make sure I understand a few things about how this hangs together:

1. My understanding is that CoreCLREmbedding.dll enables scripting Core CLR from Node.js, and that "scripting Node.js from Core CLR" is not implemented yet. In that case, why did you choose NuGet as a distribution mechanism for this assembly? The dev workflow for scripting Core CLR from Node.js typically starts with NPM - at which point would the NuGet package be installed?
2. Regarding edge-cs, what exactly does the requirements that it is packaged as NuGet imply? If a developer wants to embed literal C# in a Node.js application and use CoreCLR to compile/run it, do they now have to install edge-cs separately? How? Is .NET Core SDK/CLI required to compile literal C#?
3. Regarding this:

The requirement that the .NET Core SDK and CLI be installed on servers running Edge.js has been removed.

On machines without .NET Core SDK/CLI, can I script literal C# from Node.js?

1. Regarding this:

If the .NET Core SDK and CLI are present, the NPM install process will build a small .NET Core bootstrap application that will enable to jump into node directly and start dynamically scripting C# code without a project.json

How does this work exactly? Is the bootstrap application limited to a specific set of .NET references?

1. Finally, just to make sure: the distribution mechanism for windows binaries that allows scripting C# from Node.js has not changed, right? Thay are still part of NPM?

[lstratman]

You are correct that CoreCLREmbedding was the .NET runtime code that acted as the interop layer between Node.js and whatever .NET code that you wanted to run from Node.js. There were several reasons that I moved this over to a NuGet package. First, CoreCLREmbedding.dll depends on several packages (chief among which is Newtonsoft.Json) that a .NET application may specify a different version of. The initial code that I had for RC2 support handled this in a very non-transparent way: it had its own list of dependencies for CoreCLREmbedding.dll that it would load up and then it would merge in the list of dependencies for the application that were specified in the application's project.json to form the complete list of dependencies that we need to load when executing the .NET code. The problem is that the user would be unaware that any of this was happening and might wonder why a different version of Newtonsoft.Json or some other package was loaded since Edge.js might have required a higher version or something similar. The solution was to require the application's project.json to include a dependency on the Edge.js .NET Core runtime (the assembly formerly known as CoreCLREmbedding.dll) so that when the user ran dotnet restore it would take care of producing the complete dependency list (instead of Edge.js doing this behind the scenes at runtime). This way the user has complete visibility into the version of each dependency that will be loaded when they want to use their application with Edge.js. This dovetails nicely into the second requirement that we be able to support scenarios where folks like @gingters want to be able to run dotnet published applications that contain their application assemblies as well as a full set of framework assemblies and native libraries required to run their applications in a completely standalone fashion and not have to rely on the .NET Core SDK and CLI being installed on the target machine. The only way for that to happen is for dotnet publish to be able to pick up not only the dependencies for the application but also the dependencies for Edge.js, hence the NuGet package and requirement that it be listed in the project.json. So, when they publish their app, the resulting list of libraries and assemblies that they can deploy includes not only what they need for their app to function but also what Edge.js requires as well.

The above is all well and good if you're developing a fully-formed .NET Core application (with its own project.json) and then using that application with Edge.js. But, if you just want to:

~$ npm install edge
~$ EDGE_USE_CORECLR=1 node
> var dynamicFunc = require('edge').func('..');

then we support that as well, it just requires you to have the .NET Core SDK and CLI installed on the machine that you're doing it on. During NPM install, we detect if the dotnet executable is present in the path and, if so, we run dotnet restore and dotnet build on a small bootstrap application that is basically just a project.json with the bare minimum dependencies to run a Edge.js .NET Core app with C# dynamic compilation support. So, when you start up Edge.js and we don't find any dependency manifest in EDGE_APP_ROOT, we just use that bootstrap application which enables you to do C# scripting without having a formal project.json (satisfying #351). This bootstrap application only references the NuGet package for the C# compiler (since that's all I've done so far), but it's trivial to update it to include references to the F#, TSQL, etc. compiler NuGet packages once they are created. Once a compiler NuGet package is created and added to the bootstrap project.json, the user should be able to just run:

~$ npm install edge
~$ npm install edge-fs
~$ EDGE_USE_CORECLR=1 node
> var edge = require('edge');
> var fSharp = require('edge-fs');
> var fSharpFunc = edge.func('[some F# code here]');

without any issues. I believe that covers #1, #2, and #4. As for #3, yes, in order to script C# literals without a formal project.json, you will need to have the .NET Core SDK and CLI installed. Finally, for #5, you are correct in that the distribution model for Edge.js desktop CLR and for Mono has not changed and does not require NuGet in any way, shape, or form. You can also script C# literals in Edge.js desktop CLR or in Mono without requiring NuGet; all of the changes that I have made affect running under .NET Core only.

Sorry that was so long-winded, I hope that answers your questions.

[tjanczuk]

Perfect, I think I get it now. Can you PR?

Can you remind me what EDGE_APP_ROOT defaults to?

[lstratman]

EDGE_APP_ROOT will default to the current directory if not specified in the environment variable.

[tjanczuk]

I wonder if it would make sense to default this to __dirname to enable predictable file layout?

[tjanczuk]

Actually it would have to be __dirname of the code calling edge.func... Hmmm...

[tjanczuk]

@lstratman could you send a PR for the edge-cs nuget package?

[lstratman]

Done.

[tjanczuk]

@lstratman Thanks. I am trying to wrap my head around the implications on the Edge.js dev process that the inclusion of coreclrmebedding in a nuget package has. In order to ship a new version of Edge.js, do we now need to:

1. Build Edge.js native assemblies (edge*native.node) using node-gyp based process
2. Build nuget package for edge and edge-cs using build_double.bat
3. Run all tests (test node->clr and clr->node scripting)
4. Publish the nuget packages
5. Update the references to nuget packages in the bootstrap project
6. Publish Edge to NPM

Is that about right?

If so, does running tests in step 3 require the bootstrap project to temporarily point to the locally built nuget packages?

[lstratman]

Yep, that's basically it. I would actually flip steps 1 and 2 around. For your question about step 3, it's already taken care of: in build.bat, I updated the dotnet restore command line to specify a fallback directory (the -f flag) pointing to the output from the NuGet package build process from step 1/2.

[tjanczuk]

I see, clever. How do you organize the dev flow on Linux and Mac given that #2 needs to happen on Windows?

[lstratman]

I build the NuGet package on Windows, create a directory on my Linux and
Mac boxes, copy the NuGet package there and then update my NuGet.config on
the Linux and Mac boxes (~/.nuget/NuGet.config) to add that local directory
as a package source.

On Thu, Jun 23, 2016 at 6:26 PM, Tomasz Janczuk [email protected]
wrote:

I see, clever. How do you organize the dev flow on Linux and Mac given
that #2 https://github.com/tjanczuk/edge/issues/2 needs to happen on
Windows?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tjanczuk/edge/issues/433#issuecomment-228204397, or mute
the thread
https://github.com/notifications/unsubscribe/AABgDKKBJGqlJt5CzsP5dg1SDHTWt2rBks5qOwgYgaJpZM4IglCF
.

[lstratman]

Just FYI, the .NET Core RTM is supposed to drop at some point today, so I'd hold off on officially merging my PR until I have a chance to update the version numbers of the dependencies and do a final round of testing against the new version. Should be able to take care of that today or tomorrow, depending on when the RTM becomes available.

[tjanczuk]

Yes, good idea.

[ChristianWeyer]

Do you have any estimate @lstratman when this could be ready? ;)
Thanks!

[Konard]

Looks like issue should be renamed to .NET Core 1.0.0 RTM support...

[lstratman]

@ChristianWeyer I was hoping to have everything ready the day after the RTM, but was busy with my day job. Now that I've had some time to get everything ready, something in .NET Core's custom assembly resolution process changed, meaning that Edge.js dies silently trying to resolve assemblies from the .deps.json. I'm not sure what's going on with it, but hope to have everything working in the next day or two.

[tjanczuk]

@ChristianWeyer what is your dependency on this by the way?

[ChristianWeyer]

We have a very large customer project which is by now blocked by this...

[tjanczuk]

I see you like playing it close to the chest ;)

[ChristianWeyer]

Cutting EDGE ;P

[lstratman]

Got the dependency resolution issue sorted out, got everything built, and successfully ran the unit tests on both Windows and Linux. The only thing outstanding is that the bootstrap application (which allows running Edge.js without a project.json) seems to need some additional dependencies, which I will look at tomorrow.

[lstratman]

FINALLY, everything is done. .NET Core RTM support is official, tests and builds are green across Windows, Linux, and OS X. @tjanczuk I updated the version numbers in the various project.jsons for both Edge.js and Edge.js.CSharp; they are in no way meant to be final or official, feel free to change them to whatever makes sense to you.

[tjanczuk]

Perfect, this is going to be great. I will go through the devops cycle today and try to ship.

[Konard]

So... When the next release?
Looks like no info at: https://github.com/tjanczuk/edge/releases
But there is new version of package: https://www.nuget.org/packages/Edge.js/

Did the installation process changed?

[forensicsguy20012004]

Does this include support for node 6.xxx?
On Jul 5, 2016 6:00 PM, "Konstantin Dyachenko" [email protected]
wrote:

So... When the next release?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/tjanczuk/edge/issues/433#issuecomment-230615833, or mute
the thread
https://github.com/notifications/unsubscribe/AMCkZ8ylqMzdCLkfHfsGJMrcopcZxj_2ks5qStP4gaJpZM4IglCF
.

[seertenedos]

Really want this on the the released version of CoreCLR so my c# code can call node functions. Is that what this topic is or the other way around?

[Konard]

Looks like everything is ok except NuGet packages usage.. https://github.com/tjanczuk/edge/issues/451

[tjanczuk]

[email protected] has now shipped. Supports CoreCLR 1.0.0 preview 2 and node 6.x.

Big thanks to @lstratman!