Aspnetcore.docs: SignalR and multi instance applications

[ferronsw]

What are the best practices for SignalR and loadbalanced multi instance application?

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 1387f556-b4b7-70f9-5bd0-f0bccc597689
• Version Independent ID: fb016cdc-d281-6b5b-7466-2c718e7274b4
• Content: Use hubs in ASP.NET Core SignalR
• Content Source: aspnetcore/signalr/hubs.md
• Product: aspnet-core
• GitHub Login: @rachelappel
• Microsoft Alias: rachelap

[anurse]

We strongly recommend the Azure SignalR Service for multi-instance application scenarios (including Azure App Service) as it offloads all the connections to the managed service.

If that service is not available, we do have the ability to configure a backplane based on Redis. Documentation for this is coming soon. In general, you just need to add a reference to Microsoft.AspNetCore.SignalR.Redis and call .AddRedis after .AddSignalR in your ConfigureServices method:

public void ConfigureServices(IServiceCollection services)
{
    // ... other services ...
    services.AddSignalR().AddRedis(yourRedisConnectionString);
    // ... other services ...
}

[davidfowl]

@anurse we need a scale out doc.

[ferronsw]

We've setup a master slave Redis configuration for SignalR. We've use this to connect to Redis:

services.AddSignalR().AddRedis("jaundiced-arachnid-redis-slave.default.svc.cluster.local:6379,jaundiced-arachnid-redis-master.default.svc.cluster.local:6379,password=password,defaultDatabase=2");

It works except for one thing. When we load balance our application containers we saw that the request for establishing the SignalR connection were divided over two containers. When that happens the connection fails.

This is the log of our loadbalancer:

10.244.0.1 - [10.244.0.1] - - [25/Jul/2018:12:04:11 +0000] "GET /lib/signalr/signalr.min.js HTTP/2.0" 200 16127 "https://aks.westeurope.cloudapp.azure.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" 6043 0.009 [default-olw-client-service-80] 10.244.0.166:80 69475 0.008 200 f7d7143eed618c38d138080a22cc3639
10.244.0.1 - [10.244.0.1] - - [25/Jul/2018:12:04:11 +0000] "GET /js/WidgetSignalR.min.js HTTP/2.0" 200 235 "https://aks.westeurope.cloudapp.azure.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" 6041 0.009 [default-olw-client-service-80] 10.244.0.166:80 293 0.008 200 c0e692afe319fe9d329e13abdaae59a7
10.244.0.1 - [10.244.0.1] - - [25/Jul/2018:12:04:11 +0000] "GET /sm/b8427ae79f001842dec31421d2331d1bf01b09f0794a6b49d15ae566d4d6b5b7.map HTTP/2.0" 404 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" 6012 0.002 [default-olw-client-service-80] 10.244.0.166:80 0 0.000 404 4e8e7b716399d75f25b00ca1ced538e7
10.244.0.1 - [10.244.0.1] - - [25/Jul/2018:12:04:11 +0000] "GET /lib/signalr/signalr.min.js.map HTTP/2.0" 404 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" 5984 0.002 [default-olw-client-service-80] 10.244.0.167:80 0 0.004 404 042a979460fecf140c09ff33e1789065

This is the log of the console from the browser:

WebSocket connection to 'wss://aks.westeurope.cloudapp.azure.com/widgetHub?id=SSzJbqP0txnH1jaKl1TTOQ' failed: Error during WebSocket handshake: Unexpected response code: 404
signalr.min.js:16 Error: Failed to start the transport 'WebSockets': undefined

Is this as expected?

[anurse]

@anurse we need a scale out doc.

💯

[anurse]

Filed #7814 for that

[anurse]

Closing this since #7814 is tracking a scale-out doc

[ddydeveloper]

@anurse good morning! I am sorry if it was already discussed and described via documentation #7814 but unfortunately I didn't find any way to cope with my problem

I have a k8s cluster (GKE or local with minikube it doesn't matter) with a number of asp.net core API apps include signalR and redis to enable multiple instance flow. The project repo contains all of the stuff. My signalR configuration:

services.AddSignalR().AddRedis(Configuration.GetConnectionString("Redis"));

The api app deployment file contains replicas param:

spec:
   replicas: 2

When I have 1 or 2 instances all works fine:

But when the number of replicas become more than 2 there are the following problem:

The initial error is:

WebSocket connection to 'wss://35.228.213.90/api/signalR/notifications?id=h3HubreK9uywu1P7xNkSnA' failed: Error during WebSocket handshake: Unexpected response code: 404

So it seems some of the pods can't connect to the asp.net core server via websockets and other transports are not available too.

I am sure that it is not a GKE problem because I've noticed the same behavior in my minikube/docker for windows cluster. Could you provide some thougth what is the reason and how to figure out with it?

Maybe it is better to use the separated signalR server in a single instance mode not to scale it with the server replicas but it looks like a huge workaround.

Any help will be approciated.

[davidfowl]

You need to turn on sticky sessions/client affinity in your load balancer when using signalr.

[ddydeveloper]

@davidfowl thank you for suggestion! Actually I've tried to extend nginx-ingress controller with the following annotations:

kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: "route"
nginx.ingress.kubernetes.io/session-cookie-hash: "sha1"

But the result was the same.

BTW could you please specify why we need to set up sticky session for the load balancer? I supposed that if we use a Redis operational storage for signalR that would be enought to share all the stuff we need?

[ferronsw]

@ddydeveloper you would think that, but that's not the case.

https://docs.microsoft.com/en-us/aspnet/core/signalr/version-differences?view=aspnetcore-2.2#sticky-sessions

[ddydeveloper]

@ferronsw got it, thank you!

In ASP.NET Core SignalR, the client must interact with the same server for the duration of the connection. For scaleout using Redis, that means sticky sessions are required.

So we need sticky sessions to keep client connection with the same server in a farm.

[davidfowl]

Yes, it's required because messages from client to server and connect and disconnect notifications fire on the same server get sent to the same server they originated on.

[gmarokov]

@davidfowl we were able to spin multiple SignalR servers without sticky session instead skipping the negotiate process and establishing a connection directly trough WebSocket.

export function createHubConnection(url: string) : signalR.HubConnection {
  return new signalR.HubConnectionBuilder()
    .withUrl(url, {
      skipNegotiation: true,
      transport: signalR.HttpTransportType.WebSockets
    })
    .build();
}

What drawbacks have this method except the unavailability to pick from the list of transports available and not able to fallback to LongPolling when WS is not supported for example?

[davidfowl]

What drawbacks have this method except the unavailability to pick from the list of transports available and not able to fallback to LongPolling when WS is not supported for example?

That's correct. WebSockets is the only transport that can work this way anyways.