Quartznet: Support for .NET Core or vNext

[frapid]

Hi,

Is there a plan to support .NET Core? Would like to obtain more information on this.

[lahma]

I've had some trials but decided to wait for RC2 as things seem to be quite unstable at the moment. Hopefully v3 will have at least a limited support for it.

[frapid]

Thank you for the quick answer.

Open source projects like us (and many others) depend on Quartz.NET. Since Quartz.NET is a framework (unlike web applications), it would be exciting to see Quartz supporting .NET Core.

Without Quartz, it is nearly impossible for many web applications (like ours) to support .NET Core. Eagerly waiting for v3 release and looking forward to contribute in any way we can.

[mjrousos]

Hi @lahma, I'm a member of Microsoft's .NET Customer Success Team. One of the things my team works on is making sure useful third-party libraries work with .NET Core and the .NET Standard Profile. As part of that, I’d like to help you get Quartz.Net running on .NET Core.

I’ve done some rough prototyping in a forked branch and have basic scenarios working. As an example, here’s a screenshot of a simple scheduler using a SQL Server job store running on Ubuntu:

As you observed in a different thread, a number of things are missing or changed in .NET Core, so there's a fair bit of work remaining to get everything up and running. I think most of the work will be:

• Removing remoting dependencies. The remoting usage in Quartz.Net is pretty straight-forward, so it can probably be replaced with HTTP requests. That will require a fair rewrite of the remotable scheduler, though.
• Replace System.Net.Mail usage with a third-party alternative like MailKit
• Enabling database integration. I’ve already created a new .NET Standard-compliant dbproviders.properties file which enables SQL Server to work. We should also add support for SQLite and PostgreSQL. Others don’t work on .NET Core yet, though some will hopefully be coming soon.
• Review and cleanup some of my changes thus far. I made some quick changes to get the SimpleThreadPool working on .NET Core (since the CLRThreadPool won't work there), but we should go back and create a proper .NET Core-compliant IThreadPool implementation.
• Migrate tests to .NET Core and flush out the inevitable bugs.

Here is a diff between quartznet/quartznet-3 and my development branch. There are a lot of changes, but most have to do either with the serialization changes, miscellaneous API replacements, or turning off features (like remoting) that we’ll have to re-enable some other way in the future. I’ve marked all of those ‘future work’ spots with comments containing “TODO (NetCore Port)”.

Would you be up for me pitching in here? Ultimately Quartz.Net is yours to own and drive, but I'm happy to help do some of the work to get it running on .NET Core.

[frapid]

@mjrousos, it feels great when someone from Microsoft comes up and contributes and supports open source frameworks. I am thankful and glad to see you. Kudos for the job you've done!

[mjrousos]

Thanks, @frapid. I'm really excited to about all the changes in the Framework recently and glad to have the opportunity to help build up the .NET Core ecosystem.

[Jawvig]

On the topic of the remoting interface, I have some feedback. It was great to have it and I still have it in use in production, but I'd much rather an HTTP based API. Having to take a binary dependency on Quartz in a monitoring client was uncomfortable. It feels sensible for this to be a separate package, don't you think?

[lahma]

@mjrousos this is great news! I'm really glad you've taken interest on this and helping out. I'm going to go through your changes and learn from them and possible ask some more questions. I had a cursory look and looks really good.

Some thoughts that I want to share that might or might not have significance:

• I would like to take this in to v3 branch sooner than later, @danielmarbach has been helping with async and there will probably be some internals changing and those changes should be done having these changes in
• Web API based interface has been in the works under the Quartz.Web project, as a plugin now - remoting can probably be removed altogether from v3
• SendMailJob has the System.Net.Mail dependency and tha job that can be taken out from core and offered as an code example that can be re-implemented on library's consumer side
• Binary serialization will be a big pain point here, v2 data is in database in binary format and migration from format to another (binary to JSON) might prove itself tricky
  
  
  
  • just something that needs some pondering on my side
  
  

Do you have some kind of time-box for working on this or some other goal where you are ready sign changes as 'ready for merging'? As we take all the help we can get, I'm also comfortable with pulling in the changes and fixing as we go if you have limited.

And once a again, I'm super grateful for the help and appreciate it. I would have taken probably uncountable hours to figure out the way to handle things with RC2 (with RC1 I gave up after some battling).

[mjrousos]

Getting things merged into the v3 branch (or possibly a quartznet branch off of it) so that we can make sure things work with async changes makes sense. I've tried to make sure that the original quartz.sln solution will still build and work as before, but we should double-check that before merging it in. Eventually, everything will build from the project.json file directly, but it probably makes sense to keep the old build process working in the short-term while we're getting everything moved over.

If remoting (and to a lesser degree, mail) can be removed from the primary quartz project, that will make the port much easier. I don't see any reason that a web API plugin or MailKit plugin couldn't be made to work with .NET Core later.

I think you're right that serialization and threading will be left as the two main areas to make work in the new model. Unfortunately, it's probably not realistic to have the .NET Core implementation of Quartz work with data serialized with the binary formatter. I think there are a couple options -

1. The easiest solution will be to let customers know that the new .NET Core implementation of Quartz.Net can't use database information from previous versions of the library and they should wait to use the .NET Core version until they can start with a fresh database.
2. Json serialization could be enabled very easily in the full desktop version of the library, so maybe a configuration setting could be added for desktop Quartz.Net. If the user chooses binary serialization, they will get some size benefits (and backwards compatibility), but if they choose json serialization, they will be able to share the database between desktop and .NET Core implementations of Quartz.Net.
3. If there's a demand for migrating old database information to work with the new .NET Core Quartz.Net, it may be possible to create a stand-alone desktop .NET tool that does the conversion. We'd have to think more about that scenario, though.

As far as time constraints go, I will probably be able to give half of my time or more to this project for at least the next few weeks. After that, I'll likely have other projects that need my focus but I can continue to be involved with Quartz.Net a few hours a week until the port is done. As far as when we should merge things, I think you're right that sooner is better, but let's double-check that nothing has accidentally regressed in the desktop Quartz.Net implementation first.

Also, please let me know if you have questions as you look through my code. If it would help, I'm happy to setup a phone call to talk through the changes. You can find my email address in my GitHub profile if you'd like to arrange something like that.

[danielmarbach]

Why even have a default serialization? You can easily have the abstraction in the core and the serializer is a plugin which is mandatory as a parameter while setting up.

Am 27.04.2016 um 17:48 schrieb Mike Rousos [email protected]:

Getting things merged into the v3 branch (or possibly a quartznet branch off of it) so that we can make sure things work with async changes makes sense. I've tried to make sure that the original quartz.sln solution will still build and work as before, but we should double-check that before merging it in. Eventually, everything will build from the project.json file directly, but it probably makes sense to keep the old build process working in the short-term while we're getting everything moved over.

If remoting (and to a lesser degree, mail) can be removed from the primary quartz project, that will make the port much easier. I don't see any reason that a web API plugin or MailKit plugin couldn't be made to work with .NET Core later.

I think you're right that serialization and threading will be left as the two main areas to make work in the new model. Unfortunately, it's probably not realistic to have the .NET Core implementation of Quartz work with data serialized with the binary formatter. I think there are a couple options -

The easiest solution will be to let customers know that the new .NET Core implementation of Quartz.Net can't use database information from previous versions of the library and they should wait to use the .NET Core version until they can start with a fresh database.
Json serialization could be enabled very easily in the full desktop version of the library, so maybe a configuration setting could be added for desktop Quartz.Net. If the user chooses binary serialization, they will get some size benefits (and backwards compatibility), but if they choose json serialization, they will be able to share the database between desktop and .NET Core implementations of Quartz.Net.
If there's a demand for migrating old database information to work with the new .NET Core Quartz.Net, it may be possible to create a stand-alone desktop .NET tool that does the conversion. We'd have to think more about that scenario, though.
As far as time constraints go, I will probably be able to give half of my time or more to this project for at least the next few weeks. After that, I'll likely have other projects that need my focus but I can continue to be involved with Quartz.Net a few hours a week until the port is done. As far as when we should merge things, I think you're right that sooner is better, but let's double-check that nothing has accidentally regressed in the desktop Quartz.Net implementation first.

Also, please let me know if you have questions as you look through my code. If it would help, I'm happy to setup a phone call to talk through the changes. You can find my email address in my GitHub profile if you'd like to arrange something like that.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[lahma]

I created a pull request to discuss the changes: #359 . I commented some stuff as they came by, didn't repeat when saw same thing second/third time. Everyone is welcomed to chime in and read my ramblings. General observations:

• We probably can drop serialization for Exceptions
• Remoting parts can be removed, REST'ish Web API can be exposed
• A lot of DataMember attributes.. which I understand for private members but hopefully we can configure on Json.NET's side
• As noted in my review comment and as @danielmarbach did here too, I guess we could just make serialization strategy selection mandatory, user needs to do an informed decision there
• New SimpleThreadPool implementation is a big missing link (controllable thread counts, priorties etc)

We can leave worrying database migrations to later stage. I'd like to see this land soon in v3 branch.

Overall, great work! :+1:

[mjrousos]

Great; thanks, @lahma. I'll take a look at the PR and respond to comments. And I agree that it may make sense to just have serialization be pluggable and have users select what they want to use.

[mjrousos]

One other thing I forgot to mention is that I'm using .NET Core RC2 targets, so to build the project you will need the RC2 SDK. We haven't published RC2 Visual Studio tooling yet (it should be coming soon), so for the moment the .NET Core version of the library can only be built from the command line (dotnet restore, dotnet build, dotnet pack).

[frapid]

We are eagerly waiting for Quartz.net so that our solution ported to vNext builds.

Could we have an alpha build in the nuget so that people could at least reference and compile their projects?

[lahma]

Hopefully we could get something alpha quality out in the next couple of weeks to allow you to kick the tires. API will not stable nor will there be any remote management support though.

[smbecker]

Personally, I think that remote management could stay out of the coreclr version. I would think that people would want to handle that by fronting it with an MVC app anyway. That shouldn't be a quartz concern.

[smbecker]

For IThreadPool, I wrote a version that relies on Tasks that I could contribute back if your interested.

[danielmarbach]

Bring it in. I have plans to rework it but currently in military service. Will go back to it in 2 weeks approx

Am 30.04.2016 um 13:12 schrieb Shaun Becker [email protected]:

For IThreadPool, I wrote a version that relies on Tasks that I could contribute back if your interested.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[mjrousos]

I also have an IThreadPool implementation using tasks. I've added it to the ongoing .NET Core PR, though you can also look here to see just the IThreadPool-related changes. @smbecker, @danielmarbach, please take a look and let me know if you see things that can be improved based on your own work here.

[lahma]

I just pulled in the first batch of changes from @mjrousos . They are available now on quartznet-3 branch. This will probably make collaboration easier as now the "big change" is already in. We can continue to refine the solution and tweak.

I'll try to create working POC of replacing the Remoting bits next.

[danielmarbach]

:+1:

Am 03.05.2016 um 06:53 schrieb Marko Lahma [email protected]:

I just pulled in the first batch of changes from @mjrousos . They are available now on quartznet-3 branch. This will probably make collaboration easier as now the "big change" is already in. We can continue to refine the solution and tweak.

I'll try to create working POC of replacing the Remoting bits next.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[mjrousos]

Thanks, @lahma! Let me know if you have any .NET Core related questions as you work on replacing remoting bits.

My idea for that was to redesign the API so that once a scheduler was setup to receive remote connections, it would create an ASP.NET Core IWebHost that would process incoming HTTP requests (using the Kestrel server). I had initially intended to use web sockets, but it looks like web sockets don't work cross-platform yet. So, I've been prototyping a simpler solution that just accepts a JSON object in the request body specifying the method to invoke and the arguments to pass to the method.

To test this out, I made a simple .NET Core RC2 'shared list' that is just a List that allows remote machines to connect to view/manipulate the list. You can take a look at the (somewhat messy) code here.

Here are the relevant excerpts from that sample:
In my simple prototype, setting up the IWebHost looks like this:

``` C#
host = new WebHostBuilder()
.UseServer("Microsoft.AspNetCore.Server.Kestrel")
.Configure(app =>
{
// Middleware function for processing incoming requests
app.Use(async (HttpContext context, Func next) =>
{
if (context.Request.Path.Value == Path && context.Request.ContentLength > 0)
{
// Deserialize the request's body into an InvokeRequest object (a class we define)
InvokeRequest invokeRequest = default(InvokeRequest);
invokeRequest = (InvokeRequest)invokeRequestSerializer.ReadObject(context.Request.Body);

            object reply = null;
            // Try to dispatch the call
            if (Invoke(invokeRequest, ref reply))
            {
                // I used DCS in my prototype, but probably we should use JSON.NET since Quartz.Net already uses it elsewhere
                var replySerializer = new DataContractJsonSerializer(reply.GetType());
                using (var ms = new MemoryStream())
                {
                    replySerializer.WriteObject(ms, reply);
                    await context.Response.WriteAsync(new string(Encoding.UTF8.GetChars(ms.ToArray())));

                    // Halt the HTTP request processing pipeline
                    return;
                }
            }
        }
        await next();
    });
})
.Start(new[] { $"http://0.0.0.0:{port}" });

``` C#
private bool Invoke(InvokeRequest invokeRequest, ref object reply)
{
    // In my simple test, the methods that could be invoked were just simple APIs to manipulate a list.
    // In the real Quartz.Net product, we will need to support more methods. Because of that, it may make sense to just use reflection to build the list of APIs that can be called dynamically when the host starts.
    switch (invokeRequest.MethodName)
    {
        case "Add":
            Console.WriteLine("Invoking 'Add'");
            list.Add(invokeRequest.Arguments[0] as string);
            return false;
        case "GetEnumerator":
            Console.WriteLine("Invoking 'GetEnumerator'");
            reply = list.ToArray();
            return true;
        default:
            Console.WriteLine($"Unknown method: {invokeRequest.MethodName}");
            return false;
    }
}    

C# public struct InvokeRequest { public string MethodName { get; set; } public object[] Arguments { get; set; } }

To make the remote calls, the remoting client can just use an HttpClient to call the endpoints setup by the server.

[smbecker]

That seems like it would require A LOT of work and A LOT of heavy weight dependencies. As I said above, I would place the remote management pieces of Quartz at a lower priority for .NET Core. Or at the very least, spin it off into a separate library. I always front Quartz with my own ReST api anyway.

[mjrousos]

Agreed, @smbecker. I don't know that it's any more resource-intensive at runtime than old-style remoting, but it's definitely a fair bit of code and would require a lot of changes. If people are ok with Quartz.Net on Core differing from older Quartz.Net products in this way, it probably is a good candidate for separating out into its own library.

[lahma]

@smbecker this is true, I'd like time box some checks about what would be feasible. Remote schedulers are in use and for some who don't have time required to wrap everything rely on having something out-of-the-box. Anyway I don't see lacking remoting for a while as a show stopper for shipping early builds. VS tooling and stability would be nice from Core first :smile:

@mjrousos I was initially leaning towards having contract classes (DTO kind of) hidden as Quartz internals and then building client code into the core (HttpClient dependency) and supporting hosting via separate library which would bring along the needed hosting libraries, something between raw request data and Web API. Your solution looks interesting too though.

[mjrousos]

That sounds like it could work (and would remove ASP.NET dependencies from the core library, which would be good). If you wanted to also not have the HttpClient dependency in Quartz.Net, you could probably have a separate library for the client, also, and have it create a child scheduler derived from the standard one. Contract DTOs make sense but they could be marked protected if we wanted to use them in child implementations.

I think removing the HttpClient dependency is a lower priority than pulling out the http server one, but it would help to have a better layering. No matter how you architect it, the client piece will implicitly depend on the server piece (if the web API changes, you would need to update the client accordingly). If the core Quartz.Net library includes the client code then it will potentially have to update when the server library updates and that seems a little backwards.

[runxc1]

Just saw that RC2 dropped and was looking to see if Quartz.net is .net core ready and am happy to see that it is progressing that way.

[killswitch1111]

Is there any news on this now that (I know only by a week) .Net Core is at 1.0.0?

[robborden]

When I try to run "dotnet restore", I get an error saying that Quartz 2.3.3 does not support framework .NetStandard, Version=v1.6.

Please please please, I really need this to work for .Net Core. :)

[lahma]

There's still no NuGet pre-release available, so requires building from source. I'm hoping to have the time to push pre-release-alpha out soon (TM).

[robborden]

I tried to build from source but it doesn't seem like you can reference external DLLs in a .NET Core project (at least, I can't get it to work) and I can't compile the code from within the .NET Core project, for starters because the Common.Logging Nuget package is also not compatible yet. Perhaps I don't have the requisite experience necessary to pull it off...

[killswitch1111]

@robborden Did you pull the branch quartznet-3? That is the branch that should be buildable in .Net Core. (I am currently playing with it in a project I am building and it works).

[robborden]

Thanks @cjnanda, I will give this a try.

[lahma]

Alpha 1 has landed to NuGet, please see the announcement. A lot still to do but now should be easier to kick the tyres.

Some notes:

• Separated Quartz.Serialization.Json as a separate package, Quartz itself has no longer dependencies like JSON.NET
• Remoting alternative not ready
• There will be dragons

[mjrousos]

Awesome! It's great to see bits out there for people to try out.

[lahma]

Now that 2.4 has been released I have merged the v3 branch to master. All v3 work is now against master.

[tanbchr]

I'ill give it a try too!

[roryprimrose]

Any thoughts on an ETA for v3 or how stable it is for production use? I'm interested in both netstandard and async support.

[Meligy]

Wow. Funny since October last year, the one comment asking about this is on the same day (and hour) I come to search for it!

[SidShetye]

Great work folks. Any idea when we'll have a .NET Core (Quartz.NET 3.0) production release?

[andymac4182]

Any idea how far off this is or if help is needed to get it finished? Would be great to be able to update some of our usages. eg. https://github.com/MassTransit/MassTransit/issues/909

[lahma]

Just to give an update here, I've had some time to iron out current problems with 2.x and 3.0, but next alpa will be mostly about bug fixes and .net core 2 support + API cleanup. Hopefully I'll get next alpha out soon (my summer vacation is reaching its end).

[Layoric]

@lahma Just wanted to say a thank you to this project and work for .net standard support! Been running 3.0.0 alpha 2 in docker + postgres + clustered environment for few months and for the most part works a treat. Looking forward to .net core 2 support. 👏

[lahma]

And 3.0 alpha 3 is out! As usual, all feedback and testing in invaluable, please kick the tires and see how it fares.

[lahma]

I'm closing the issue as .NET Core support is in place, I just made beta 1 available.

[intraz]

Hi,

from: https://github.com/quartznet/quartznet/issues/355#issuecomment-215127050

1. If there's a demand for migrating old database information to work with the new .NET Core Quartz.Net, it may be possible to create a stand-alone desktop .NET tool that does the conversion. We'd have to think more about that scenario, though.

Was this tool ever created? We are facing this scenario.
If it wasn't, is there a recommended way of achieving this?
Would appreciate any info.

Thanks in advance,
Eyal