Scaling Out
In-memory groups stop working at two replicas; a Redis backplane and sticky sessions are what fix it.
builder.Services.AddSignalR().AddStackExchangeRedis("redis:6379");A hub keeps its bookkeeping in the memory of one process. The list of connections, and which group each one is in, live there and nowhere else. Start a second replica, a second copy of the same app, and it knows nothing about the first. A client in room-7 on replica A and a client in room-7 on replica B sit in two unrelated groups that happen to share a name. Clients.All on A reaches only A's clients. The symptom is messages that arrive for some people and not others, and it gets worse with every replica you add.
The backplane
A backplane is a shared message bus the replicas talk through. Install the Microsoft.AspNetCore.SignalR.StackExchangeRedis package and chain AddStackExchangeRedis onto AddSignalR. Every send is published to Redis, every replica is subscribed, and each one delivers to its own clients. Give each app its own ChannelPrefix, or two apps sharing one Redis server will deliver each other's messages to the wrong clients. Redis stores nothing here: while it is unreachable, messages sent in that window are lost, and SignalR reconnects to it once it returns.
Sticky sessions
Every HTTP request belonging to one connection must reach the same process. A backplane does not change that. Three situations avoid it: a single process, the Azure SignalR Service, or clients configured for WebSockets only with negotiation skipped. In every other case, including Redis, turn on session affinity, also called sticky sessions, in whatever distributes your traffic.
Example
using Microsoft.AspNetCore.SignalR;
using StackExchange.Redis;
var builder = WebApplication.CreateBuilder(args);
// Null when the connection string is absent, as in local development.
string? redis = builder.Configuration.GetConnectionString("Redis");
ISignalRServerBuilder signalR = builder.Services.AddSignalR();
if (redis is not null)
{
signalR.AddStackExchangeRedis(redis, options =>
{
// Replicas sharing this prefix forward messages to each other.
// A different app on the same Redis must use a different prefix.
options.Configuration.ChannelPrefix = RedisChannel.Literal("chat");
});
}
var app = builder.Build();
app.MapHub<ChatHub>("/hubs/chat");
app.Run();
public class ChatHub : Hub
{
// Unchanged by scaling out. With the backplane it reaches every replica.
public Task Broadcast(string text)
=> Clients.All.SendAsync("Receive", text);
}When to use it
- Keep a chat working after scaling a Deployment from one replica to three.
- Add a Redis backplane so a notice sent by any replica reaches every connected client.
- Turn on session affinity at the ingress so clients that fell back to long polling stay connected.
- Give two apps sharing one Redis server separate channel prefixes so their broadcasts stay apart.
More examples
Session affinity on a Kubernetes Service
This is the minimum that makes server-sent events and long polling survive more than one replica. ClientIP affinity is coarse: everyone behind one office NAT address lands on the same pod. It is also only half the story when an ingress controller sits in front of the Service, because the controller is then the thing choosing a pod, and affinity has to be configured on the controller as well, usually as a cookie.
apiVersion: v1
kind: Service
metadata:
name: chat
spec:
selector:
app: chat
# Send every request from one client IP to the same pod.
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10800
ports:
- port: 80
targetPort: 8080Skipping negotiation to avoid sticky sessions
With negotiation skipped there is one request per connection, the WebSocket handshake itself, so there is no second request to route to the wrong replica and no need for affinity. The price is that there is no fallback left: a proxy or network that blocks WebSockets now means no connection at all, rather than a slower one. Worth it when you control every client, which is why it suits a desktop or service client more than a public website.
using Microsoft.AspNetCore.Http.Connections;
using Microsoft.AspNetCore.SignalR.Client;
HubConnection connection = new HubConnectionBuilder()
.WithUrl("https://chat.example.com/hubs/chat", options =>
{
// SkipNegotiation is only legal with WebSockets as the sole transport.
options.Transports = HttpTransportType.WebSockets;
options.SkipNegotiation = true;
})
.Build();
await connection.StartAsync();Broadcasting from a background service
Shows the trap and the fix together. Every replica runs its own copy of this service, so with three replicas each client gets the message once from its own replica and twice more through the backplane. Either elect one sender, or have the work produce a message in one place and let only that place publish.
using Microsoft.AspNetCore.SignalR;
// Registered with builder.Services.AddHostedService<DepthBroadcaster>();
public class DepthBroadcaster(IHubContext<ChatHub> hub) : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
// Replace 0 with the real measurement. With a backplane this
// reaches clients on every replica, not just this one.
await hub.Clients.All.SendAsync("Depth", 0, stoppingToken);
await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
}
}
}
Discussion