Aspnetboilerplate: Add CQRS infrastructure

[hikalkan]

We may add generic interfaces to use CQRS patterns. Also we can enable UOW, authorization validation... for they automatically as application services.

See these documents:

https://www.cuttingedge.it/blogs/steven/pivot/entry.php?id=91
https://www.cuttingedge.it/blogs/steven/pivot/entry.php?id=92
http://martinfowler.com/bliki/CQRS.html
...

[a-merson]

In my current project I use ABP heavily and I see that CQRS could reduce complexity very well. Though I can't move it to CQRS now when it on preproduction stage, I'm looking forward to use CQRS in next project (or in next version of current project).

So it's very important feature request!

Few details about current project: it has two parts each with its own web portal and db. One part is in intranet and another in internet. Communiaction between the two is possible only through MQ. There are internal users who do 90% of data modification. And there are external users that primarily read data. Now I need to share repositories between the two but with CQRS I would share only queries while commands will stay on intranet part. I'm not saying it's impossible with current ABP, just voting for CQRS future :)

[hikalkan]

Thanks for voting. I want to implement it before v1.0. We will see it :)

[kegumx]

+1

[brunobertechini]

+1

[johnkattenhorn]

+1 We are doing an integration at the moment between ABP and Project Orleans https://github.com/dotnet/orleans and event sourcing. I'm hoping we can open source some of the work soon.

[cuibty]

+1

[houwenlong789]

+1

[brunobertechini]

+1 I can help if needed.
Bruno

Sent from my Samsung device

-------- Original message --------
From: houwenlong789 [email protected]
Date: 02/03/2016 6:55 AM (GMT-03:00)
To: aspnetboilerplate/aspnetboilerplate [email protected]
Cc: Bruno Bertechini bruno.[email protected]
Subject: Re: [aspnetboilerplate] Add CQRS infrastructure (#88)

+1

—
Reply to this email directly or view it on GitHub.

[Farb]

like this

[PAA81]

+1

[chuncui0207]

+1

[loning]

+1

[aleksandretabatadze]

+1

[loning]

@johnkattenhorn would you like share some of your works?

[atabatadze]

• 1

[leehom0123]

+1

[jfbaudracco]

+1

[hikalkan]

Many people want it. We should consider this as a high priority feature request after v1.0 :)

[brunobertechini]

I'd like to give my 2 cents on this topic.

I believe CQRS is a more architectural concept rather than a infrastructure concern.

Although its very good to have some boilerplate to help those going for CQRS path we should take a look on WHERE this topic should be applied.

Mostly I'm talking about SOA and CQRS is a good fit for a given Bounded Context. Not to the whole application.

Abp is suited for web development and is very tied to DDD (Domain Driven Design) principles (and it does it very well by the way). Even the commercial product AspNetZero (which im using for quite a while is great for what it is meant to).

I would say that if we follow some guidelines and take into consideration words from Greg Young and Udi Dahan (google them and you'll see who they are :) ) we should apply some concepts at the architectural level rather than at the infrastructure level.

For example: I am having a hard time developing a new Distributed application and trying to use Abp whithin one of my microservices. The problem is : As it is designed for web/ddd apps, I cant have fine-grained DbContexts without having them coupled to each other at runtime. I would like to see Abp more fine-grained so I can have more control on which features I can use inside a given microservice without coupling it to other microservices.

I am willing to help to design architecture if you guys need...

I believe those who want this CQRS infrastructure already have knowledge about microservices and SOA and probably know Greg Young (@gregoryyoung) and Udi Dahan (@udidahan).

Just wanna give my 2 cents on the topic

Feel free to take this discussion a little further :)

Bruno

[hikalkan]

Thank you very much @brunobertechini for sharing your opinions. I mostly agree with you. That's why I waited this issue for that long. Adding some ICommand, IQuery... interfaces and base classes was not that hard. But this is an architecture and I think we should provide a complete solution with best practices (actually I'm also not an expert in this pattern yet, still learning it's best practices when I have time).
As you said (and Fowler says: http://martinfowler.com/bliki/CQRS.html) this pattern is good for a part of the application rather than the whole application. And it best works with event sourcing.
Anyway, I will work on that in next versions and I highly appreciate for sharing your ideas on that (and waiting others to contribute that :)).

[brunobertechini]

I believe one very good first step would be able to create a separate solution (like te Sample blog module you did) in order to have a completely independent (autonomous) microservice using Abp

I believe the currently downside is: The sample blog module DEPENDS heavily on Abp Application (master database). Which is a runtime coupling.

May I try to get abp source code and adjust Module-Zero with a new AbpZeroDbContext which does not depend on all features?

What would be your approach to start coding it ?

My main goal is be able to deploy a Abp module completely independent of Abp master Web application still using features like multitenancy, softdelete, webapi dynamic creation etc

Let us shed some light on project requirements and I can start coding it and submit a pull request

Bruno

[brunobertechini]

First thing im trying to test is to change the module-zero a little bit to use an approach like this
https://romiller.com/2012/03/26/dynamically-building-a-model-with-code-first/

I mean: instead of having STATIC IDbSets for feature/settings/permissions/roles etc, create the dbsets/entities (required for database) based on configuration enabled like

Configuration.Modules.Zero().RoleManagement.Enabled = true/false;

And if this setting is enable, go for it and create database tables/migrations etc

This way we can have more fine-grained control on dependencies on main abp app...

The problem I see is : Microservices should not care about things like FEature/Edition etc...this is the main appconcern; But they do have to worry about providing WHICH features they have in order to the main app be able to enable/disable it

[yonglehou]

+1

[brunobertechini]

Please see #1476 and lets first be able to create a separated microservice (standalone/autonomous) still with providers for use in main app.

Bruno

[guilmori]

+1

[Caskia]

+1

[hikalkan]

Implementing CQRS + Event Sourcing is a huge work. But we can create some basic CQRS base classes without event sourcing as a start. What do you think?

[albertdev]

@hikalkan @marcosribj
I think that it can be done without event sourcing or denormalized databases. Those things can still be added when implementing the pattern in your own application.

Basically what we do is that the IAsyncCommandHandler implementations use Abp repositories injected through the constructor while the IAsyncQueryHandlers receive a raw DbContext in their constructor so they can query without any explicit UnitOfWork attribute (meanwhile we're still working on the container lifecycle of that DbContext but oh well). Code creating the commands and queries doesn't need to know what kind of database gets used, it just passes the object and lets the implementation deal with it. If someone wants to use event sourcing, fine as well but they'd have to implement the logic and threading themselves. At any rate, CQRS can be used without Event Sourcing just fine.

The tricky stuff lies in the ICommandDispatcher or IQueryDispatcher. It takes some dynamic magic to get all the type information and Resolve the right implementation. The IAsyncCommandHandler interface is generic to declare the type of command, but it then takes some work to make sure that the dispatcher takes the right registration rather than just any IAsyncCommandHandler<ICommand> - dynamic helps with that.

Another thing which we had to build was a set of Exceptions to throw from the commands to indicate some problem with the command execution and which can't be detected before using simple validation: unknown Ids (ItemNotFoundException), concurrent modifications in the database (ConflictException) or just a state which precludes modifying an item any further (BussinessException or derivative). Most of them allow an IList of Errors to be passed with both an error code and human-friendly string.

Also, since we have WebApi 2 controllers using AutoMapper to convert DTO's into ICommands, we also wanted to map these exceptions into HttpResponses. We hookep up both some OWIN pipeline handler as well as a MVC WebApi 2 ExceptionHandler to intercept exceptions and then pass them to something like a TypeConverter: these take the exception and reformat it to the JSON format we want and may do some logging.

[guilmori]

Here is a very clean implementation of Command/Query architecture using WebApi, provided by dotnetjunkie, the author of the blog posts.

https://github.com/dotnetjunkie/solidservices

[kouweizhong]

@hikalkan Any news?

[hikalkan]

not yet.

[gentledepp]

I just read a bit about that topic (CQRS and Rest)
Jimmy Bogard has a very interesting explanatory post about designing REST interfaces for CQRS here:
https://lostechies.com/jimmybogard/2016/06/01/cqrs-and-rest-the-perfect-match/

It looks to me as if this had quite an impact on the IApplicationService stuff of abp.
At least we'd need quite a bit more control over the API surface that the DynamicApiControllerBuilder generates.

[JbmOnGitHub]

@albertdev

CQRS can be used without Event Sourcing just fine.

Yes, but with "CQRS only" if you want to obtain some real gains, you have to find a way to get very directly datas who fits to your viewmodel/dto used in your UI. (= denormalized datas...)
These denormalized data are like a cache that you must invalidate when necessary. It's hard to manage and that's why you'll use Event Sourcing.
Create Views in Database to obtain directly the readviewmodels may be the easiest way to introduce CQRS (without ES) while getting real performance gains in return... (but i don't very like this option)

[gentledepp]

there is a very, very, very, very nice video about how and why an ordered stream of events helps us to scale systems and why distributed transactions cannot be the solution
https://martin.kleppmann.com/2016/03/07/qcon-london.html
An absolute must see for everyone here!

[gentledepp]

another imput:
we are actually using MediatR to easier support CQRS
see this nice article about how the mediator pattern helps with CQRS

There is also a nice article about having a CQRS REST api with mediatR as handler and Hangfire as workhorse:
mediator-pattern-hangfire

[JbmOnGitHub]

@gentledepp : We have the same readings :)
I will try to implement CQRS and ES as basically as possible:

• using the database as eventstore and memorystream as readModelStore
• using MediatR (definitely!)

A good use case can be the functionality of the "question-answer" sample.

I will share it if the result is not too bad lol

[ZhaoRd]

+1

[rexunderlinehxiao]

+1

[kegumx]

+1

[JbmOnGitHub]

After hugely failed... Here are the steps I will follow now:

1. Modify the "question-answer" example to implement more DDD and CQRS patterns without modifying abp infrastructure

2. Command side - Integrate an existing framework to handle the persistence of events (the final storage will be related to Abp)

3. Read side - Use this existing framework to synchronize the readmodels (in Memory)

Why introduce a framework for that? This is a huge work to implement correctly Event Sourcing.
Some guys have already pointed this issues and i confirm they really exists !

I am really impressed by Orleankka. Some code samples are very cool.

[JbmOnGitHub]

Step 0.5 (!) : Just commit here "CQRS added in QuestionAppService with uniform touch'

IQuestionAppService.cs - AFTER :

// Commands
Task Handle(CreateQuestion cmd);
Task Handle(VoteUp cmd);
Task Handle(VoteDown cmd);
Task Handle(SubmitAnswer cmd);
Task Handle(AcceptAnswer cmd);
Task Handle(RenameQuestion cmd); //not implemented yet
Task Handle(DeleteQuestion cmd); //not implemented yet
//Queries
PagedResultDto<QuestionDto> Query(GetQuestions input);
GetQuestionOutput Query(GetQuestion input);

ABP Dynamic Web API Controllers could evolve to expose like this :

| Verb | Url | Application Method |
| --- | --- | --- |
| POST | api/question/CreateQuestion | Task Handle(CreateQuestion cmd); |
| POST | api/question/SubmitAnswer | Task Handle(SubmitAnswer cmd); |
| PUT | api/question/RenameQuestion | Task Handle(RenameQuestion cmd); |
| DELETE | api/question/DeleteQuestion | Task Handle(DeleteQuestion cmd); |
| GET | api/question/GetQuestion | GetQuestionOutput Query(GetQuestion input); |

With EventSourcing (step 2), Task Handle(CreateQuestion cmd) would be replaced with :

• a method responsible for decisions taking : IEnumerable<Event> Handle(CreateQuestion cmd)
• a second method who apply the decision on the Aggregate : void Appy(QuestionCreated event)

This methods would be just take place in the DomainLayer (project *.Core), as for separation of concerns, i think.

[JbmOnGitHub]

On the CQRS part ONLY, i tested the idea of hikalkan for some Cqrs interfaces.
``` c#
//Command side
public interface ICommandHandler
{
void Handle(T cmd);
}
public interface IAsyncCommandHandler
{
Task Handle(T cmd);
}

//Query side
public interface IQueryHandler
{
TQueryOutput Query(TQueryInput queryInput);
}
public interface IAsyncQueryHandler
{
Task Query(TQueryInput queryInput);
}

And now, in the [Abp Question-Answer Sample](https://github.com/aspnetboilerplate/questions-answers),  IQuestionAppService would be like this :
``` c#
public interface IQuestionAppService : 
                           IApplicationService,

                           //Commands
                           IAsyncCommandHandler<CreateQuestion>,
                           IAsyncCommandHandler<VoteUp>,
                           IAsyncCommandHandler<VoteDown>,
                           IAsyncCommandHandler<SubmitAnswer>,
                           IAsyncCommandHandler<AcceptAnswer>,
                           //Queries
                           IQueryHandler<GetQuestions, PagedResultDto<QuestionDto>>,
                           IQueryHandler<GetQuestion, GetQuestionOutput>
{
}

I find this, very clear and concise !

@hikalkan : Is it close to what you suggested?

[hikalkan]

Yes @JbmOnGitHub, I thought similar. In addition, we can add IQueryHandler<TQueryOutput> for actions those returns result but does not get any parameter (and async pair surely).

[hitaspdotnet]

@hikalkan Any update on this?

[hikalkan]

No update.

[slamj1]

After reading through this thread, I thought I'd comment. I've had quite a bit of experience with the CQRS pattern and with distributed and broker-based Enterprise Service Buses - mainly for integration with 3rd party products.

How I've approached CQRS with respect to ABP / ANZ, is to integrate with Particular's NServiceBus (NSB). NSB is open-sourced and has a commercial license as well (very affordable) but you could do the same with MassTransit for example (another .Net based open-source ESB).

The combination of ABP /ANZ and NSB has been highly effective for us for integration and long running process orchestration.

My 2 cents.

[IsmailQasemNafee]

@hikalkan Any update on this?

[ismcagdas]

@IsmailQasemNafee not yet, sorry. We will consider it after .NET 5 upgrade.