Retries, Idempotency and the Outbox
At-least-once delivery means your handler will run twice, so give messages idempotency keys and publish through an outbox.
if (!handled.Add(messageId)) return; // a redelivery of work already doneQueues and brokers almost all promise at-least-once delivery. That means a message is delivered one or more times, never zero. If your handler is slow and the broker does not see an acknowledgement in time, it delivers the message again. So assume your handler will run twice on the same message, and design for it.
Idempotency
An operation is idempotent when running it twice leaves the same result as running it once. Setting a balance to 135 is idempotent. Adding 25 to a balance is not. For the second kind you need an idempotency key: a string that identifies the work, chosen by whoever is able to repeat it. The broker's message id works. So does a key the caller puts in an HTTP header.
Record every key you have handled, in the same database transaction as the effect, and skip any key you have already seen. One table with a unique constraint on the key is enough. The constraint, not your if, is what keeps you safe when two deliveries are handled at the same moment: the second insert fails, and that failure is your answer.
The outbox
Now the other half. "Save the order and publish OrderPlaced" touches two systems. Save first, then publish, and a crash in between loses the event. Publish first, then save, and a crash leaves an event for an order that does not exist. A database transaction cannot cover a broker.
The outbox pattern fixes this. Write the event as a row in an outbox table in the same transaction as the order. One commit, so both land or neither does. A background service then reads unsent rows, publishes them, and marks them sent. A crash after publishing but before marking means the event is published again, which is at-least-once again, which the consumer's idempotency key already handles.
Example
// A handler that is safe to call twice. The key identifies the message, not the
// attempt, so a redelivery carries the same key.
HashSet<string> handled = [];
Dictionary<string, int> balances = new() { ["acct-1"] = 100 };
void Apply(string messageId, string account, int amount)
{
// HashSet.Add returns false if the item was already there. This is the short
// form of: if (handled.Contains(messageId)) { ... } else { handled.Add(...); }
if (!handled.Add(messageId))
{
Console.WriteLine($"{messageId}: already applied, skipped");
return;
}
balances[account] += amount;
Console.WriteLine($"{messageId}: applied {amount}");
}
Apply("msg-7", "acct-1", 25);
Apply("msg-7", "acct-1", 25); // the same message, delivered a second time
Apply("msg-8", "acct-1", 10);
Console.WriteLine($"acct-1 balance: {balances["acct-1"]}");
// Output:
// msg-7: applied 25
// msg-7: already applied, skipped
// msg-8: applied 10
// acct-1 balance: 135When to use it
- Stop a retried payment request from charging the card a second time.
- Publish an OrderPlaced event without a window where the order exists and the event does not.
- Make a queue consumer safe when the broker redelivers a message it thinks timed out.
- Let a mobile client retry a flaky sign-up POST without creating two accounts.
More examples
Writing the event and the order in one transaction
This is the write side of the outbox, and the point is what is absent: no call to the broker. Use it any time a state change has to be announced to another service. One SaveChangesAsync wraps both inserts in a single transaction, so there is no window in which the order exists without its event.
using Microsoft.EntityFrameworkCore;
using System.Text.Json;
namespace Shop;
public sealed class Order
{
public int Id { get; set; }
public required string Reference { get; set; }
}
public sealed class OutboxMessage
{
public int Id { get; set; }
public required string Type { get; set; }
public required string Payload { get; set; }
public bool Sent { get; set; }
}
public sealed class ShopContext(DbContextOptions<ShopContext> options) : DbContext(options)
{
public DbSet<Order> Orders => Set<Order>();
public DbSet<OutboxMessage> Outbox => Set<OutboxMessage>();
}
public sealed class OrderService(ShopContext db)
{
public async Task PlaceAsync(string reference, CancellationToken cancellationToken)
{
db.Orders.Add(new Order { Reference = reference });
// Same DbContext, same SaveChangesAsync, therefore one transaction.
// Either the order and the event are both saved, or neither is. No
// explicit BeginTransaction is needed for this.
db.Outbox.Add(new OutboxMessage
{
Type = "OrderPlaced",
Payload = JsonSerializer.Serialize(new { reference })
});
await db.SaveChangesAsync(cancellationToken);
}
}The publisher that drains the outbox
The read side. Ordering by Id keeps events in the order they were committed, and Take limits how much one pass holds in memory. Note the honest comment: this design republishes rather than loses, which is the trade the outbox makes, and it is why the next example exists.
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
namespace Shop;
public sealed class OutboxPublisher(
IServiceScopeFactory scopeFactory,
ILogger<OutboxPublisher> logger) : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
using PeriodicTimer timer = new(TimeSpan.FromSeconds(2));
while (await timer.WaitForNextTickAsync(CancellationToken.None))
{
if (stoppingToken.IsCancellationRequested)
{
return;
}
using IServiceScope scope = scopeFactory.CreateScope();
ShopContext db = scope.ServiceProvider.GetRequiredService<ShopContext>();
List<OutboxMessage> pending = await db.Outbox
.Where(m => !m.Sent)
.OrderBy(m => m.Id) // publish in the order they happened
.Take(20)
.ToListAsync(stoppingToken);
foreach (OutboxMessage message in pending)
{
await PublishAsync(message, stoppingToken);
message.Sent = true;
// A crash here republishes this message on the next pass. That
// is at-least-once, and the consumer must tolerate it.
}
if (pending.Count > 0)
{
await db.SaveChangesAsync(stoppingToken);
logger.LogInformation("Published {Count} outbox messages", pending.Count);
}
}
}
private static Task PublishAsync(OutboxMessage message, CancellationToken cancellationToken) =>
Task.CompletedTask; // replace with the real broker client
}An Idempotency-Key header on a money endpoint
POST twice with the same Idempotency-Key header and both calls return 200 with the same receipt string; a request without the header returns 400. Use this on any endpoint whose effect costs money or sends a message, because a client that times out will retry and has no way to know whether the first attempt landed.
using Microsoft.Extensions.Primitives;
using System.Collections.Concurrent;
WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
// In memory only, so this is per replica. In production this is a table with a
// unique constraint on the key, written in the same transaction as the charge.
builder.Services.AddSingleton<ConcurrentDictionary<string, string>>();
WebApplication app = builder.Build();
app.MapPost("/payments", (
PaymentRequest request,
HttpContext http,
ConcurrentDictionary<string, string> seen) =>
{
if (!http.Request.Headers.TryGetValue("Idempotency-Key", out StringValues key)
|| key.Count != 1)
{
return Results.BadRequest("An Idempotency-Key header is required.");
}
// A client that timed out and retried sends the same key, so it gets the
// first attempt's receipt back instead of being charged twice.
string receipt = seen.GetOrAdd(key.ToString(), _ => $"receipt-{request.Reference}");
return Results.Ok(receipt);
});
app.Run();
public record PaymentRequest(string Reference, int Amount);
Discussion