Masstransit: azure web jobs sdk binding

[stombeur]

I'm looking at azure web jobs and I was wondering if you have considered creating a binding for masstransit to allow an azure web job to be triggered by a message received on the bus.

I know the best answer to this question is "yes, just send us a PR" but before I dive deep into it I want to know if

• it is feasible
• it is an interesting feature
  (I'm not that familiar yet with masstransit)

cheers!

[bakatz]

+1. Hardcoding message queue names with the default service bus trigger isn't ideal. I'm in the process of writing these bindings for work right now actually.

[phatboyg]

This is an interesting idea, the use of MT to process a Web Job. I wonder what the interaction is between Azure Service Bus, Web Jobs (what's the input queue for web jobs), and the services that handle them. Surely they are able to trigger jobs in service fabric now (since classic cloud services are dying and aren't compatible with Azure Resource Manager.

Would they use service fabric, or app services, that's the question. Or Azure Functions. Too many choices!

[bakatz]

@phatboyg

So, this is going to be a wall of text. I wrote my comment before I had much of a chance to evaluate webjobs, but as of now I've evaluated all 4 "host your app on azure" products (webjobs, app services, classic cloud services, and azure functions) and have now written some interop code that allows masstransit messages to be consumed by webjobs, so maybe I can provide some insight into all the different products you mentioned:

1. App Service (synonym: "Azure Web Sites") is a platform used to easily host apps on azure. Multiple App Services can share a single App Service Plan (which defines the CPU, RAM, etc. for your group of services). With just an app service by itself, you can deploy a web app to azure and have it run on IIS with SSL w/o much extra work relative to the work required for Cloud Services. Tradeoff is that you can't remote into the nodes in your app service, but they do give you an in-browser powershell console (along with a normal cmd.exe shell).

2. WebJobs are just console apps that are hosted/managed by App Service. App Service ensures that the webjobs get restarted when they crash, started on boot, etc.

3. Functions are a fork of WebJobs that are also hosted by App Service Plans, except scaling is handled automatically / pricing is based on # of requests to your function, not the specific resources you provision (so the App Service Plan is dynamic and is determined by Microsoft, not you). This is an equivalent offering to AWS Lambda, and functions are very exciting, but they just came out of "preview" and the tooling still isn't production ready (i.e. local development/testing appears to be difficult since all C# functions have to be written as .csx files) so I'm not using them for production.

4. I wouldn't bother with service fabric. It was a buggy mess the last time I used it: my entire cluster essentially exploded when I tried to do an ARM deployment to scale up the number of nodes in the cluster. Also had to use certificates to manage my nodes, which was a pain (each developer had to install a cert on their machine to be able to log in to the service fabric management UI). It seems like it's a half baked attempt at an equivalent of what Kubernetes + Docker gets you.

Anyway, tl;dr: I like web jobs and app services. They're stable and work correctly. For web jobs, in order to receive/send service bus messages, you define a method that looks like this:

public static void DoSomethingOnMessageReceived([ServiceBusTrigger("some-input-topic", "subscriber-name") YourEventClass event, [ServiceBus]("some-output-event-queue-or-topic") ICollector<MessageProcessedEvent> outputQueue) {
// Do some processing on the event
//...
outputQueue.Add(new MessageProcessedEvent()); // send a queue *or topic* message saying that the message was processed
}

As you can see here, the function looks an awful lot like what you would write for a MassTransit consumer, and in fact, webjobs does a lot of the same things that MT does behind the scenes (manages retries, serialization, batching, etc). The problem is that, in order to interop with masstransit, you have to do several things:

1. Write out the entire autogenerated MT topic name as a string i.e. [ServiceBusTrigger("yourcompany.messaging.v1.somenamespace.someevent")]. This is pretty ugly.

2. Since MT uses a non-standard content type, you need to write a custom message processor to force each message sent by MT to have the "application/json" content type instead of the "vnd/masstransit+json" content type or whatever it is, otherwise WebJobs doesn't recognize the message as something it can serialize.

3. You need to wrap each message in an envelope class with a "Message" property, since MT uses a custom envelope for each message.

Like I mentioned, I already wrote some code that gets around the above problems (and I'm not sure what my company's open source policy is, so I can't share right now), but it would be ideal to have an officially supported MassTransitTriggerAttribute and MassTransitOutputAttribute that looks something like this: [MassTransitTrigger(typeof(Message))] and [MassTransitOutput(typeof(SomeOtherMessage))]

[ivanko2000]

Just wanted to add my vote - this is the approach that makes most sense for my requirements (with similar conclusions as what @bakatz described so eloquently)

[bakatz]

@ivanko2000 thanks, I'm rarely eloquent so this is a huge compliment :)

[ivanko2000]

Like I mentioned, I already wrote some code that gets around the above problems (and I'm not sure what my company's open source policy is, so I can't share right now), but it would be ideal to have an officially supported MassTransitTriggerAttribute and MassTransitOutputAttribute that looks something like this: [MassTransitTrigger(typeof(Message))] and [MassTransitOutput(typeof(SomeOtherMessage))]

@bakatz, would you be willing to provide more details regarding the changes you had to introduce to make this work? I am still interested in this approach, and since this might not be a priority for the MassTransit team, my [software development] preference is for 'creative borrowing' over an honest day's work... :)

[bakatz]

@ivanko2000 the approach I implemented does not solve the "hardcoded message name" problem, but basically, what I ended up with was writing a MessageProcessor that understood MassTransit content types, and converted those types to standard ones defined by IANA (application+vnd+masstransit/json to application/json, etc.). Then everything pretty much worked. To send out messages to MT consumers, you have to look at the contract that MT expects for its message envelope and wrap the message in that envelope before it will work. That's pretty much all the details I can give for now

[pvanroos]

@bakatz We are looking into using Azure Functions too. The Function would essentially serve as a MT Consumer much like how they implemented in WebJobs. The problem is that there is no MassTransitTrigger to invoke the Function. In WebJobs, you can get around this (like you said), but Functions must have some trigger defined. MS today has not opened the trigger construct to extension so we cannot create own implementation based on MT. See Stackoverflow.

Many companies are aggressively moving large portions of their IT infrastructure to Azure. A service bus implementation in Azure using MT, with the possibility of easily connecting WebJobs, Logic Apps, Azure Functions and even Flow apps would be very powerful.

@phatboyg Have you guys looked into something like this?

[phatboyg]

Work is already underway. Started it this week and should have it finished this weekend. Supporting service bus and perhaps storage queue triggers. Perhaps even http.

[phatboyg]

Work is done, had it up and running in Azure last night. Need feedback/suggestions.

[bakatz]

@phatboyg which build is this change in? Where can I test it out?

[phatboyg]

There is an azure functions branch. Should
Be in the develop builds soon.

[bakatz]

@phatboyg great, thanks

[ivanko2000]

Thank you very much, @phatboyg !

I am struggling to locate relevant changes among 630+ updated files; any plans for a very minimal "blurb" of what exactly these changes bring in and how to use them?