Nopcommerce: Async methods (performance)

[DarthSonic]

Based on discussion http://www.nopcommerce.com/boards/t/41947/performance-testing-nopcommerce-370-on-azure-using-redis-blob-azure-sql.aspx here are some suggestions for better performance in Azure enviroment (and even outside Azure)

also see http://www.nopcommerce.com/boards/t/46712/nopcommerce-400-roadmap-and-estimated-release-date-lets-discuss.aspx?p=2#185433

We have to introduce more async methods (actually covering all request) - controllers, services, database calls, etc

[lintho]

Good work. I hope the team will take this seriously!

[chadwixk]

Hope the Nop Team evaluates this seriously ...and soon!

[almarax]

I'd be more specific in p.4 - do not query DB multiple times when you can easily use your own optimization improvement:
QueryableExtensions.IncludeProperties()
to preload all related data in the first query, This will significantly lower used DTUs in Azure and make pages much more responsible, especially considering latency of every query on lower Azure plans.

[martingust]

Great, I think this is a deal breaker for a lot of people.

[pdunleavy]

Please can we add async calls to blob storage (AzurePictureService) this looks to be using a static reference which is torn down on every request, not sure of the internals of the storage api but getpicturethumb gets a lot of hammering and this could cause a scalability bottleneck if requests are being queued synchronously to access the container.

[pdunleavy]

The category listing page uses a stored procedure to retrieve the paged list of products, I need more than QueryableExtensions.IncludeProperties() to include product picture records almarax. This would need refactoring to be multi result set... http://stackoverflow.com/questions/10314032/entity-framework-5-stored-procedure-with-multiple-result-sets

[pdunleavy]

Just want to say, we have now moved our site to a single Azure VM with the same specification. All pages now load 1 second faster than web app with sql azure. If this work item can be done we can consider moving back.

[lintho]

Are there any updates on this? We host in azure with webapps and sql databases, and we need better performance.

Edit: It seems this was moved

[AndreiMaz]

Please sorry and ignore that it's in Russian (only for internal use). It's in our internal list of points (in Russian). I think we have to translate it for others

Optimization. предлагаю обсудить, что мы можем сделать для версии 4.0

1. предлагаю вернуться к задаче - Use async controller when requesting third-party services. использовать асинхронные методы в наших сервисах, которые взаимодействуют со сторонними сервисами. читай все, что связаны с плагинами, с azure, с redis. а возможно и вообще все сервисы сделать асинхронными, поскольку все они так или иначе взаимодействуют друг с другом.
2. вытекает из предыдущего пункта - сделать все методы контроллеров асинхронными. поддержка этого со стороны mvc есть.
3. соответственно обновить наши плагины на использование async запросов к сторонним сервисам. там, где это невозможно, просто использовать крупномодульную асинхронность вида Task.Run(() => {}).
4. предлагаю реализовать также эту задачу Add async support for events. сделать вызовы и обработку событий асинхронными.
5. подумать об асинхронном выполнении schedule tasks.
6. сделать взаимодействие с бд асинхронным? вроде entity framework это поддерживает

---

Как я понял, основная задача - разгрузить сервер, чтобы при запросе сторонних сервисов (платежки, доставка и прочее) thread сразу возвращался обратно в thread pool, не дожидаясь окончания выполнения запроса к API, так как размер thread pool ограничен, и при большой загруженности и долгих ответах от сторонних сервисов сервер перестанет справляться.

При изучении этой темы я наткнулся на следующие моменты - для того, чтобы сделать действительно асинхронное выполнение действий внутри запроса, при которых thread не будет занят и будет сразу возвращаться (чего мы и хотим) сделать асинхронными только контроллеры, как предлагается в исходной задаче будет недостаточным. Нам нужно будет проваливаться с написанием асинхронного кода глубже и глубже вплоть до кода плагинов, и уже там требовать асинхронных вызовов API, то есть сторонние библиотеки должны предлагать такие методы, либо разработчики плагинов сами должны писать асинхронную обработку запросов к API сервисов. Только в таком случае выполнение кода будет действительно асинхронным.

Другой подход заключается в том, чтобы использовать асинхронные обертки над синхронными методами, тогда будет достаточно обернуть вызовы сторонних сервисов внутри контроллеров. Но такой подход не даст требуемого результата - исходный thread запроса будет возвращаться, но при этом, для обрабоки действий внутри обертки, будет взят другой из thread pool. Единственное, что конечный пользователь сможет совершать другие действия в то время, пока дожидается ответа от сервисов, но выигрыша по количеству свободных thread это не даст.

таким образом, можно:

переписать большинство методов с учетом поддержки асинхронного кода и заставить всех разработчиков плагинов также поддерживать асинхронный код (действительно с применением асинхронных методов обработки вызовов API, либо писать обертки поверх обычного кода)
сделать в контроллерах асинхронные обертки поверх существующего кода в тех местах, где вызываются сервисы.
оставить, как есть, без поддержки асинхронности
назначить задачу на кого-то еще и дать поразбираться, не уверен, что я все понял правильно
ссылки:
http://blogs.msdn.com/b/pfxteam/archive/2012/03/24/10287244.aspx
http://stackoverflow.com/questions/21406973/wrapping-synchronous-code-into-asynchronous-call

[AndreiMaz]

We have only tasks related to async left in this task. so I've just renamed it. For further chane I would suggest to create separate work items

[rreaddev]

Please add more async method support to 4.10 to improve Redis caching scalability.

[fendric]

I second this motion. Redis is unusable in its current state in NOP 4.0.

[kmaklakova]

Confirming that Redis is not usable with NOP 4.

[yaronbsmtx]

Same here, performance drops tens of percents when Redis is in use.
Any plans for 4.1?

[skoshelev]

We shouldn't forget about #1275

[galvesribeiro]

Hey folks!

We've evaluating use nopCommerce to a new project but while reading the code, I realised that all the calls to the database are blocking calls using the non-async methods. In particular the INopDataProvider and its friends...

Is there a plan to make them all async? ASP.Net core is completely non-blocking architecture and those calls are hurting a lot performance...

If there is no plan for that, would a PR be accepted to change that?

Thank you!

[untiedshoes]

Hey folks!

We've evaluating use nopCommerce to a new project but while reading the code, I realised that all the calls to the database are blocking calls using the non-async methods. In particular the INopDataProvider and its friends...

Is there a plan to make them all async? ASP.Net core is completely non-blocking architecture and those calls are hurting a lot performance...

If there is no plan for that, would a PR be accepted to change that?

Thank you!

Agreed! These blocking calls are a serious problem with high concurrent users (1000+ (as low as 265/270 with 4.00)). We've narrowed our problems down to possible thread starvation.

We also managed to increase the load that 4.00 can take (up to around 400), by implementing some performance fixes for 4.20, and also 4.20 branches.

We've only just ported our 4.00 multi-store to 4.30, but haven't made it live as yet as we're still doing some tests, but based on tests with an 'out-of-the-box' 4.30, we're not expecting anything miraculous!

Here's a post a put up a week or so ago: https://github.com/nopSolutions/nopCommerce/issues/4964

If we can't get the performance with high concurrent users, the client is going to be looking at other solutions. For the time being, the client is currently using some 'queuing' software for 'kit launches' (football clubs) to help with the load.

Just to add, the issue with performance isn't under general load, as the site (current 4.00), performs extremely well, with loads times under, or just around 1s.

Regards,
Craig

[galvesribeiro]

Yeah that is exactly our usecase. We managed in a simulation to distribute it across hundreds of containers, but it wasting a lot of resources on the host machines not to mention the resources over Azure SQL itself... It would be really great to have they moved to async since the whole ASP.Net Core runtime is based on it and blocking their threds is something really bad.

I would say that if you want to use those blocking calls, better stick with older versions of nopCommerce with .Net Framework and don't move to .Net Core until this get fixed.

[untiedshoes]

We've done some load testing on our sites again, and thought I'd share the results.

Dev 4.30
25: https://bit.ly/32Cxc6Y
30: https://bit.ly/31AqeA1
40: https://bit.ly/2EsuoS6
1000 load: https://bit.ly/2EIq9BD

Base 4.00 - no products / categories etc
25: https://bit.ly/2YHTPWJ
30: https://bit.ly/31zyMHp
40: https://bit.ly/32CrT7P
1000 load: https://bit.ly/2QtMgya

Live 4.00
25: https://bit.ly/34HXW8S
30: https://bit.ly/3lrCxa4
40: https://bit.ly/31yxeh1

Dev 4.30 can handle more clicks per second. 40 was fine on dev 4.30 (probably take more). 40 failed on the live site pretty quickly.

Dev 4.30
40 load single page: https://bit.ly/34Fw4lH
40 load two page: https://bit.ly/3lrLWyo

50 slow: https://bit.ly/2G0V1O9
60 fail: https://bit.ly/3gwEK04

Looks like 40 is the limit.

Yesterday, on our live 4.00 site, we had a serious issue though, with Almost 1000 anonymous users and 200-250 current connections. Threads over a 1000! This basically killed the server, and we couldn’t restart the site! We had to very quickly implement some queue software to be able to restart the site and limit the connections.

We should have the 4.30 dev rolled over to live by the 7th, and have recently implemented a few 4.40 performances fixes such as https://github.com/nopSolutions/nopCommerce/commit/eda728b37a9b92d8a7c2be4c08bf606e6b3b011e and yesterday, 'IVersionLoader makes many DB requests on resolve #5004'.

Regards,
Craig

Note: You'll to copy and paste the urls above.

[untiedshoes]

Quick question - will implementing #5000 break 4.30, and any plugins? If not, then I'll make these changes to our development of 4.30 straightaway.

[untiedshoes]

More tests again today on our 4.30. We thought we'd warm up the site first prior to the main tests - warm-up, 0 - 200 clients over 10mins on one URL.

At 5 mins in, we start to get timeouts, and the response time, of course, goes through the roof.

By the way, we've pushed #5004 to our 4.30.

We're also using Azure CDN for all static files, and of course, images. Not using Redis as we've always found it to reduce overall performance considerably.

30 clients on 1 url for 2 mins. Fine, with the following response times: Average: 2023ms, min/max: 620 / 4833ms.
40 clients on 1 url for 2 mins. Fine, with the following response times: Average: 2995ms, min/max: 765 / 6553ms.
0 - 1000 clients on 1 url for 2 mins. Fail.

We did 45 clients over 1 url for 2 mins, and although it passed, response times were slow: Average: 3065ms Min/Max: 660 / 7455ms.

[galvesribeiro]

Damn man, that is freaking slow... :/

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: untiedshoes [e-mail]
Sent: Monday, August 31, 2020 8:13:19 AM
To: nopSolutions/nopCommerce [e-mail]
Cc: Gutemberg Ribeiro [e-mail]; Comment [e-mail]
Subject: Re: [nopSolutions/nopCommerce] Async methods (#2287)

More tests again today on our 4.30. We thought we'd warm up the site first prior to the main tests - warm-up, 0 - 200 clients over 10mins on one URL.

At 5 mins in, we start to get timeouts, and the response time, of course, goes through the roof.

[Screen Shot 2020-08-31 at 11 40 16]https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fuser-images.githubusercontent.com%2F12123259%2F91712003-d2026380-eb7e-11ea-8dac-bcb6288141ec.png&data=02%7C01%7C%7C0543dd1e7dcc4a9c7a7b08d84d9ede30%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637344692003967843&sdata=p4PzXbc0pyIQ1Qh%2F5RcT8hmSJiMLXorBAr1wNXRN2Ag%3D&reserved=0
[Screen Shot 2020-08-31 at 11 40 29]https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fuser-images.githubusercontent.com%2F12123259%2F91712009-d4fd5400-eb7e-11ea-88f5-776cdd26e5a7.png&data=02%7C01%7C%7C0543dd1e7dcc4a9c7a7b08d84d9ede30%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637344692003967843&sdata=8K%2FAGlqcWtSl8iy%2B9hb8hIv9HDMObYmzRA7t%2Fd%2FsBYc%3D&reserved=0

By the way, we've pushed #5004https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FnopSolutions%2FnopCommerce%2Fissues%2F5004&data=02%7C01%7C%7C0543dd1e7dcc4a9c7a7b08d84d9ede30%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637344692003977837&sdata=R1UKlpyaGm3YQFsrlvDNC31j391cLZFkv1OrTKDFSo8%3D&reserved=0 to our 4.30.

We're also using Azure CDN for all static files, and of course, images. Not using Redis as we've always found it to reduce overall performance considerably.

30 clients on 1 url for 2 mins. Fine, with the following response times: Average: 2023ms, min/max: 620 / 4833ms.
40 clients on 1 url for 2 mins. Fine, with the following response times: Average: 2995ms, min/max: 765 / 6553ms.
0 - 1000 clients on 1 url for 2 mins. Fail.

[Screen Shot 2020-08-31 at 11 56 24]https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fuser-images.githubusercontent.com%2F12123259%2F91713117-0ecf5a00-eb81-11ea-9c66-897ae901bf70.png&data=02%7C01%7C%7C0543dd1e7dcc4a9c7a7b08d84d9ede30%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637344692003977837&sdata=tboH9pXglUEeQwLlBHOv8LJ2qVhjEKyNJF3Q9v7Zrkg%3D&reserved=0
[Screen Shot 2020-08-31 at 11 56 09]https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fuser-images.githubusercontent.com%2F12123259%2F91713121-11ca4a80-eb81-11ea-84bc-7a09c87a1eb9.png&data=02%7C01%7C%7C0543dd1e7dcc4a9c7a7b08d84d9ede30%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637344692003987829&sdata=YaMJVDdTM4hm204ysplwzw3mx2o5gyFbrXi9EWzvl94%3D&reserved=0

We did 45 clients over 1 url for 2 mins, and although it passed, response times were slow: Average: 3065ms Min/Max: 660 / 7455ms.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FnopSolutions%2FnopCommerce%2Fissues%2F2287%23issuecomment-683715597&data=02%7C01%7C%7C0543dd1e7dcc4a9c7a7b08d84d9ede30%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637344692003987829&sdata=9KzH5A68pM3dzGIkwF%2FOGKIDfPDAACbHtW7MlDuB74A%3D&reserved=0, or unsubscribehttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABD64OD4IR43IBVDNZGK5XLSDOAU7ANCNFSM4DGEZPYA&data=02%7C01%7C%7C0543dd1e7dcc4a9c7a7b08d84d9ede30%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637344692003987829&sdata=mHaKkBG3sTRi%2FRnRpaSEWd8veH5f6FyL19IpvklmIdU%3D&reserved=0.

[lintho]

We did a stresstest in azure and we experienced the same, but that was for 4.1. 200 clients stresstesting with azure tool crashed the site completely. And 200 clients during a sale isnt much tbh. I dont know if the stress test in azure operates as a user does though.

[galvesribeiro]

I will see what I can do when I have some time to convert to async and see what happens...

In meanwhile, it would be good that you guys share a generic benchmark code somewhere so we can use the same tests to measure it...

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Sindre Markussen [e-mail]
Sent: Monday, August 31, 2020 8:25:42 AM
To: nopSolutions/nopCommerce [e-mail]
Cc: Gutemberg Ribeiro [e-mail]; Comment [e-mail]
Subject: Re: [nopSolutions/nopCommerce] Async methods (#2287)

We did a stresstest in azure and we experienced the same, but that was for 4.1. 200 clients stresstesting with azure tool crashed the site completely. And 200 clients during a sale isnt much tbh. I dont know if the stress test in azure operates as a user does though.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FnopSolutions%2FnopCommerce%2Fissues%2F2287%23issuecomment-683721035&data=02%7C01%7C%7Cfebed190719c4ffd5ec008d84da0990a%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637344699433589891&sdata=ZiHI0CtKcarficE5pEYfpi49XKeez%2BNBdcHRy5mIfcA%3D&reserved=0, or unsubscribehttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABD64OBARBOWWXNT2BZFFXDSDOCDNANCNFSM4DGEZPYA&data=02%7C01%7C%7Cfebed190719c4ffd5ec008d84da0990a%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637344699433599882&sdata=6OlIyRQjHPjObobjX5Rtp1HmGLMXpjvC3x9D0pcdKpk%3D&reserved=0.

[untiedshoes]

I will see what I can do when I have some time to convert to async and see what happens ... In meanwhile, it would be good that you guys share a generic benchmark code somewhere so we can use the same tests to measure it. .. Get Outlook for iOS < https://aka.ms/o0ukef >
...
________________________________ From: Sindre Markussen [e-mail] Sent: Monday, August 31, 2020 8:25:42 AM To: nopSolutions/nopCommerce [e-mail] Cc: Gutemberg Ribeiro [e-mail]; Comment [e-mail] Subject: Re: [nopSolutions/nopCommerce] Async methods (#2287) We did a stresstest in azure and we experienced the same, but that was for 4.1. 200 clients stresstesting with azure tool crashed the site completely. And 200 clients during a sale isnt much tbh. I dont know if the stress test in azure operates as a user does though. — You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FnopSolutions%2FnopCommerce%2Fissues%2F2287%23issuecomment-683721035&data=02%7C01%7C%7Cfebed190719c4ffd5ec008d84da0990a%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637344699433589891&sdata=ZiHI0CtKcarficE5pEYfpi49XKeez%2BNBdcHRy5mIfcA%3D&reserved=0, or unsubscribehttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABD64OBARBOWWXNT2BZFFXDSDOCDNANCNFSM4DGEZPYA&data=02%7C01%7C%7Cfebed190719c4ffd5ec008d84da0990a%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637344699433599882&sdata=6OlIyRQjHPjObobjX5Rtp1HmGLMXpjvC3x9D0pcdKpk%3D&reserved=0.

If you check #4964, you'll see some base tests for 4.30.

[untiedshoes]

We did a stresstest in azure and we experienced the same, but that was for 4.1. 200 clients stresstesting with azure tool crashed the site completely. And 200 clients during a sale isnt much tbh. I dont know if the stress test in azure operates as a user does though.

The test basically simulates 200 clients per second, all clicking refresh. The max we've found is 50, but response times were not great. 60, failed!

50 clients per second, one url, 2 mins.

On our current 4.00 multi-store, we've had over 600 users on the site at one time with next to zero impact on performance, with over 700 orders, and 700 new customer registration (genuine) in one day.

Our main issue is the initial load - i.e. say 50 to 100 clients on the site, then a large increase (e.g. when a mailshot goes out / social media announcement about a new kit release), and traffic increases significantly in a very short time, and brings the site down. With that load, it's also extremely difficult to restart! Also, because it's a multi-store, it brings down the other stores too.

It certainly looks like 4.30 can handle this better, but it isn't spectacular considering this is an issue that's been ongoing since 2017.

[untiedshoes]

Personally, I'd like to see some load tests on a load-balanced server (azure, with auto-scaling), to see if this could help? At the moment, I'm not able too, but we have an extremely well specced dedicated server.

Dual E5-2620 v3 CPU’s
Dual 1TB SSD in raid 1
128GB ECC ram
Windows Server 2019 on SSD
IIS 10
MSSQL, with web licence, on the same server
Plesk
1Gb/s port
All files on an NVMe (No difference in performance between using the NVMe, and having files on SSD)

[untiedshoes]

Following on from my previous comments, we've finally moved our 4.00 multi-store to 4.30.

The development version, albeit with only a single store, and fewer products/categories, can handle 25 clients per second, 30 & 40 being the max 'useable'. These tests are shown above.

We've done some testing this morning on the newly upgraded site - it can't even handle 25 clients a second, and falls flat on its face within 10 seconds!!

We've cleaned up the DB, reindexed, but still the same. All settings are the same across the live & dev sites.

What on earth is going on with nopCommerce?! I'll be honest, I think we're getting very close to finding another solution, and joining many others than have moved to a new eCommerce solution.

I'm sorry for the rant, but we are getting so much aggravation from the client over this.

[lintho]

This isnt very good PR for nopcommerce as a solution, and i think it should be adressedcommented on by a nopcommerce employee.

[Leftyx]

Following on from my previous comments, we've finally moved our 4.00 multi-store to 4.30.

The development version, albeit with only a single store, and fewer products/categories, can handle 25 clients per second, 30 & 40 being the max 'useable'. These tests are shown above.

We've done some testing this morning on the newly upgraded site - it can't even handle 25 clients a second, and falls flat on its face within 10 seconds!!

We've cleaned up the DB, reindexed, but still the same. All settings are the same across the live & dev sites.

What on earth is going on with nopCommerce?! I'll be honest, I think we're getting very close to finding another solution, and joining many others than have moved to a new eCommerce solution.

I'm sorry for the rant, but we are getting so much aggravation from the client over this.

@untiedshoes The problem with performance is related to the amount of queries running for each operation and a wrong use of caching. They claim the problem was EF Core but it sits in the way they have defined the db model and the way it is queried.
It works fine if you have a bunch of categories but when your catalog grows NopCommerce simply cannot keep up.

[untiedshoes]

Following on from my previous comments, we've finally moved our 4.00 multi-store to 4.30.
The development version, albeit with only a single store, and fewer products / categories, can handle 25 clients per second, 30 & 40 being the max 'useable'. These tests are shown above.
We've done some testing this morning on the newly upgraded site - it can't even handle 25 clients a second, and falls flat on its face within 10 seconds !!
We've cleaned up the DB, reindexed, but still the same. All settings are the same across the live & dev sites.
What on earth is going on with nopCommerce ?! I'll be honest, I think we're getting very close to finding another solution, and joining many others than have moved to a new eCommerce solution.
I'm sorry for the rant, but we are getting so much aggravation from the client over this.

@untiedshoes The problem with performance is related to the amount of queries running for each operation and a wrong use of caching. They claim the problem was EF Core but it sits in the way they have defined the db model and the way it is queried.
It works fine if you have a bunch of categories but when your catalog grows NopCommerce simply cannot keep up.

Yep, there's certainly something serious going on! Our 'upgrade' for the client from 4.00, to 4.30 has massively reduced the number of connections the site can handle. Because this site is a multi-store, for football clubs, it puts our client in a very difficult position with their own clients.

Our 4.00 could handle 25 clients per second... just! Our current 4.30 fails on 25, and the maximum is 10. With Christmas coming up, this is now quite a serious issue.

The development 4.30 version, as mentioned above could do 40 clients per second (50, at a push), and is identical in every way, apart from the number of categories, customers etc,

25 clients per second:

10 clients per second:

Regards,
Craig
p.s. Also, with our 4.30 site, we get a lot of 400 errors, client disconnected errors, incorrect content type errors - I've raised an issue about these, but not had a response.

p.p.s Category wise, we have 55 published. Products, 1941 published. Customers, 186,781

[tekguy]

@untiedshoes are these performance tests located somewhere? I'd like to run against our configuration to see if we get similar results. Our 4.3 app is deployed in azure.

Thanks

[untiedshoes]

@untiedshoes are these performance tests located somewhere? I'd like to run against our configuration to see if we get similar results. Our 4.3 app is deployed in azure.

Thanks

Hi,

The vast majority of our tests are shown above, and also here #4964

We're using loader.io by the way.

[joekrauss]

I agree with the test results provided by @unitedshoes when it comes to the failure rate. Has anyone tested a generic net core site to see if the issue is with net core itself? What if this issue is a Microsoft problem and we are limiting our test parameters to only a NopCommerce site.

Thanks & Stay Safe!
Joseph Krauss

Sent from my iPhone

On Oct 2, 2020, at 6:48 AM, untiedshoes [e-mail] wrote:


@untiedshoes are these performance tests located somewhere? I'd like to run against our configuration to see if we get similar results. Our 4.3 app is deployed in azure.

Thanks

Hi,

The vast majority of our tests are shown above, and also here #4964

We're using loader.io by the way.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

Defense Solution Group, Inc. / DSG Arms
Office: 817-806-9242
Fax: [phone]
Email:
6100 Conveyor Drive
Cleburne, TX 76031-0998
CAGE: 1L0J5
GSA: GS-07F-6007R
https://dsgarms.com
https://www.dsgholsters.com
http://www.dsginc.com
The content of this email is confidential and intended for the recipient specified in the message only. It is strictly forbidden to share any part of this message with any third party, without the written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future.

[nirupamduet]

Redis will not work.
But all you need to add more caching,
Reduce db call, split GenericAttribute table, stop checking image file
existence and some custom development.

You could use stackify to figure it out.

BTW, we have a store(4.0) with a lots of customization, it could hardly
serve 400+, after 2/3 months of optimization it can easily serve 2000+
(Google analytic).

We are using two c5.xlarge app instances and c5.2xlarge db instances.

On Mon, 31 Aug 2020, 5:13 pm untiedshoes, [e-mail] wrote:

More tests again today on our 4.30. We thought we'd warm up the site first
prior to the main tests - warm-up, 0 - 200 clients over 10mins on one URL.

At 5 mins in, we start to get timeouts, and the response time, of course,
goes through the roof.

[image: Screen Shot 2020-08-31 at 11 40 16]
https://user-images.githubusercontent.com/12123259/91712003-d2026380-eb7e-11ea-8dac-bcb6288141ec.png
[image: Screen Shot 2020-08-31 at 11 40 29]
https://user-images.githubusercontent.com/12123259/91712009-d4fd5400-eb7e-11ea-88f5-776cdd26e5a7.png

By the way, we've pushed #5004
https://github.com/nopSolutions/nopCommerce/issues/5004 to our 4.30.

We're also using Azure CDN for all static files, and of course, images.
Not using Redis as we've always found it to reduce overall performance
considerably.

30 clients on 1 url for 2 mins. Fine, with the following response times:
Average: 2023ms, min/max: 620 / 4833ms.
40 clients on 1 url for 2 mins. Fine, with the following response times:
Average: 2995ms, min/max: 765 / 6553ms.
0 - 1000 clients on 1 url for 2 mins. Fail.

[image: Screen Shot 2020-08-31 at 11 56 24]
https://user-images.githubusercontent.com/12123259/91713117-0ecf5a00-eb81-11ea-9c66-897ae901bf70.png
[image: Screen Shot 2020-08-31 at 11 56 09]
https://user-images.githubusercontent.com/12123259/91713121-11ca4a80-eb81-11ea-84bc-7a09c87a1eb9.png

We did 45 clients over 1 url for 2 mins, and although it passed, response
times were slow: Average: 3065ms Min/Max: 660 / 7455ms.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/nopSolutions/nopCommerce/issues/2287#issuecomment-683715597,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAK4OSYHD3JDTKN7LS2B7ULSDOAVLANCNFSM4DGEZPYA
.

[untiedshoes]

We've managed to make some progress with our 4.30 site. We can now handle 80 clients a second - using various more aggressive caching, and multiple worker processes (between 4 & 6). Not exceptional, but certainly better than we were experiencing.

The downside to this is, of course, we now have instances of the scheduled tasks running more than once - order emails for instance.

I've tried using a mutex, but as Mutex in .net core 3.1 only accepts three arguments, I can't pass through MutexSecurity() - trying without, crashes the app pool.

Does anyone have a quick fix for the timer handler - scheduled tasks in a web garden?

4 x Worker Processes:

6 x worker processes

Thanks,
Craig

p.s. I see progress is being made on the async methods :)

[tekguy]

@untiedshoes Very good progress!

Correct me if I am wrong when you say "worker" I am assuming you mean a node/instance in a web farm. In that case I don't see how mutex would help in this situation if the app is deployed to 4, 6 or X number of workers/nodes. A mutexwill operate on a single app pool (IIS definition)

In our case I couldn't find a solution from NOP itself for preventing scheduled jobs from executing on multiple nodes. So we are planning to roll our own, which involves:

1. Randomly assigning a node in the web farm as the schedule task executioner. In our situation, we are running on Azure (app service). And resources are moved around in the cloud, so your app can be running on one server in azure today and possibly another tomorrow. So we are planning to constantly re-assign a valid task executioner (based on the machine name, https://docs.microsoft.com/en-us/dotnet/api/system.environment.machinename?view=net-5.0). This machine name will be stored as a setting in the setting table, we will run this process as a background job, taking advantage of IHostedService, https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-3.1&tabs=visual-studio

2. We are placing additional logic in the following area (where the task are executed): Task class
   Here we will check the machine name that has executed the task matches what we have stored in the settings table.

We haven't done this work yet, but this was my initial idea. Obviously you don't have to create step 1 if you know the machine name of each node in your webfarm won't change to reduce the complexity. You can assign the machine name in appsettings.json or settings table as an example.

As a side note, also looking forward to the async methods PR!

[untiedshoes]

@tekguy

Thanks for the reply 👍

By worker processes, I mean app pool worker processes. Increasing from the standard 1, to 4/6. We’ve tried NUMA in the past, but had issue with it.

The issue of the scheduled task sending multiple emails when you increase worker processes has been around for a long time, and has never really been addressed.

The performance increase we’ve made is great, but we can’t really push it to the live site of the client / customers are going to get multiple emails for every order placed,

For instance, I put an order through earlier, and I received 2 store owner emails, and three customer emails.

Thanks,
Craig

[mattisax]

@untiedshoes

Curious about your load test. Are all the users making multiple requests to the same url for a minute, or does a user make a single request to the url and then they are done? Also, what type of environment are you testing against?

We recently (few weeks ago) went through some battles with Nop 4.3 performance but ended up making a few changes that gave us a big improvement. Sites performed great through Black Friday/Cyber Monday. Busiest of the sites averages around 3k request per minute during peak hours. That is requests actually hitting the web server. Cloudflare is being used too. These are all single instance sites. We are going to explore horizontal scaling in the near future. Here are the changes. The top two take minutes but gave us big bang for the buck.

1. Port this commit back to Nop 4.3, build, and redeploy the updated Nop.Data.dll: https://github.com/nopSolutions/nopCommerce/commit/56e4594e3419ff672dafe2d9b4f252842934188a
2. Change GC settings in Nop.Web.runtime.config. "System.GC.Concurrent", "System.GC.Server", and "System.GC.RetainVM" should all be set to true.
3. We were trying to override Nop's memory cache implementation since it seems to be clearing itself excessively, but ended up just creating our own cache with EasyCaching that is used by our own plugins.

Ken

[untiedshoes]

@untiedshoes

Curious about your load test. Are all the users making multiple requests to the same url for a minute, or does a user make a single request to the url and then they are done? Also, what type of environment are you testing against?

We recently (few weeks ago) went through some battles with Nop 4.3 performance but ended up making a few changes that gave us a big improvement. Sites performed great through Black Friday / Cyber ​​Monday. Busiest of the sites averages around 3k request per minute during peak hours. That is requests actually hitting the web server. Cloudflare is being used too. These are all single instance sites. We are going to explore horizontal scaling in the near future. Here are the changes. The top two take minutes but gave us big bang for the buck.

1. Port this commit back to Nop 4.3, build, and redeploy the updated Nop.Data.dll: 56e4594
2. Change GC settings in Nop.Web.runtime.config. "System.GC.Concurrent", "System.GC.Server", and "System.GC.RetainVM" should all be set to true.
3. We were trying to override Nop's memory cache implementation since it seems to be clearing itself excessively, but ended up just creating our own cache with EasyCaching that is used by our own plugins.

Ken

Hi Ken,

Load tests are done via loader.io - as far as I'm aware, the load (basically), simulates x amount of clients, hitting refresh every second for the duration of the test.

We've done lots of other tests, including 0 - 1000 / 2000 over 1, 2, 5 minutes etc.

I've already ported 56e4594 thanks :+1: plus, many many others
Also, we saw large memory usage using GC.

My next step is to look at the latest version of EasyCaching - I believe the nopTeam tried it, but it hit a wall. I also believe they are now looking at LazyCaching too.

For more tests, and details of the server, have a look at #4964

One thing to add is that the current tests are being performed on a development site, not the live. The last time we pushed the dev to the live, we saw a massive reduction in performance from the dev to live (even though both reside on the same server).

The live, uses a database which, has over time, been upgraded from 3.00, 3.2, 3.7, 4.00, 4.30 - it, as far as I'm aware, was the first, full response, live multi-store.

Increase the app pool worker processes has yielded the biggest performance increase we've seen so far, but until we can solve the issue of the scheduled task, it's a no go.

We're trying awfully hard to keep the client on board, but I feel we're losing the battle with them.

Regards,
Craig

[untiedshoes]

Update! For now, it looks like I solved the scheduled tasks running more than once when using multiple app pool worker processes.

After some testing, I've not received a single, duplicate order email... I'll continue testing today, and report back this evening.

For anyone wondering:

private void TimerHandler(object state) { try { _timer.Change(-1, -1); Run(); } catch { // ignore } finally { if (RunOnlyOnce) Dispose(); else _timer.Change(Interval, Interval); } }

Sorry for going slightly off-topic here :)

[skoshelev]

Closed #2287.
You may find all related commits by this link