Benchmarking with BenchmarkDotNet

Compare two implementations reliably with BenchmarkDotNet: Release builds, baselines, parameters, MemoryDiagnoser and reading the results honestly.

Syntax

[MemoryDiagnoser]
public class OrderCsvBenchmarks
{
    [Params(100, 10_000)] public int LineCount { get; set; }
    [Benchmark(Baseline = true)] public decimal SplitAndParse() => ...;
}

A Stopwatch around a loop measures the JIT, tiered compilation, the timer's resolution and whatever else the machine was doing. BenchmarkDotNet (package BenchmarkDotNet 0.15.8) controls those: it builds the benchmark project in Release, runs each benchmark in a separate process, executes warm-up iterations until tiered JIT has produced optimised code, picks an invocation count so each iteration is long enough to time, runs many iterations, removes outliers and reports the mean with its error and standard deviation.

The parts

[Benchmark] marks a method to measure; Baseline = true on one of them adds a Ratio column relative to it.
[Params] runs every benchmark for each value: performance often changes shape with input size, so test 100 items and 10,000.
[GlobalSetup] prepares input once per parameter combination, outside the measurement. [IterationSetup] exists, but for fast methods it distorts results; avoid it.
[MemoryDiagnoser] adds Gen0 (collections per 1,000 operations) and Allocated (bytes per operation) columns. Allocation numbers are exact, unlike timings.
BenchmarkSwitcher.FromAssembly(...).Run(args) lets one console project hold many benchmark classes and pick them with --filter.

Reading the output

An illustrative run of the main example on one machine:

| Method        | LineCount | Mean       | Error    | Ratio | Gen0    | Allocated | Alloc Ratio |
|-------------- |---------- |-----------:|---------:|------:|--------:|----------:|------------:|
| SplitAndParse | 10000     | 1,412.3 us | 11.80 us |  1.00 | 228.5156| 2343.8 KB |       1.000 |
| SpanRanges    | 10000     |   871.6 us |  6.02 us |  0.62 |       - |       0 B |       0.000 |

If Error is a large share of Mean, or two means are within each other's error, you have not shown a difference. A Ratio of 0.62 is a 38% saving on that method, which only matters if the method is a meaningful share of the request. Profile the endpoint first, then benchmark the hot method.

Mistakes that produce wrong numbers

Mistake	Effect	Fix
Debug build or debugger attached	BenchmarkDotNet refuses, or numbers several times too slow	`dotnet run -c Release`, no debugger
Result not used	JIT removes the work: impossibly fast results	Return the result from the benchmark method
Constant input	JIT folds the computation	Read input from fields set in `[GlobalSetup]`
Benchmarking I/O	Measures the database or network, not your code	Benchmark CPU-bound code; load-test I/O paths
Laptop on battery, shared CI agent	Noise larger than the effect	Dedicated machine; compare within one run

Where benchmarks fit

A micro-benchmark answers "is A faster than B for this input". It does not answer "will p99 improve": that needs a load test against a deployed build. Keep benchmarks in a benchmarks/ project in the solution, run them on demand, and record results in the pull request. Running them as a CI gate on shared agents produces flaky failures; if you gate, gate on Allocated, which is deterministic.

Example

using System.Globalization;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(OrderCsvBenchmarks).Assembly).Run(args);

[MemoryDiagnoser]
public class OrderCsvBenchmarks
{
    private string[] _lines = [];

    [Params(100, 10_000)]
    public int LineCount { get; set; }

    [GlobalSetup]
    public void Setup() =>
        _lines = Enumerable.Range(1, LineCount)
            .Select(i => $"ORD-{i:D6},CUST-{i % 977:D4},{i * 3.75m:F2},GBP")
            .ToArray();

    [Benchmark(Baseline = true)]
    public decimal SplitAndParse()
    {
        decimal total = 0;
        foreach (var line in _lines)
            total += decimal.Parse(line.Split(',')[2], CultureInfo.InvariantCulture);
        return total;
    }

    [Benchmark]
    public decimal SpanRanges()
    {
        decimal total = 0;
        foreach (var line in _lines)
        {
            ReadOnlySpan<char> span = line;
            var field = 0;
            foreach (var range in span.Split(','))
            {
                if (field++ == 2)
                {
                    total += decimal.Parse(span[range], CultureInfo.InvariantCulture);
                    break;
                }
            }
        }
        return total;
    }
}

When to use it

Prove that replacing string.Split with span parsing in the order import halves allocations before merging the change
Quantify the cost of creating JsonSerializerOptions per request that a code review flagged in the invoice service
Run the same pricing benchmarks on net8.0 and net10.0 to show the gain from the runtime upgrade

More examples

The benchmark project

Keep benchmarks in their own console project under <code>benchmarks/</code> that references the production project, so you measure the code you ship. If the solution uses Central Package Management, drop the <code>Version</code> attribute here and add <code>BenchmarkDotNet</code> 0.15.8 to <code>Directory.Packages.props</code> instead.

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <!-- Benchmarks are never shipped or packed -->
    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.15.8" />
  </ItemGroup>

  <ItemGroup>
    <!-- Benchmark the real code, not a copy pasted into the benchmark project -->
    <ProjectReference Include="../../src/Orders.Api/Orders.Api.csproj" />
  </ItemGroup>

</Project>

Running and filtering from the command line

<code>--filter</code> takes glob patterns over the full benchmark name, so one project can hold dozens of classes. <code>--job short</code> trades precision for speed while exploring; use the default job for the numbers you put in a pull request. <code>--runtimes</code> runs the same benchmarks on each runtime, a cheap way to quantify an upgrade; the benchmark project and everything it references must target each runtime, and <code>-f net10.0</code> picks which one hosts the run.

# Always Release; arguments after -- go to BenchmarkDotNet
dotnet run -c Release --project benchmarks/Orders.Benchmarks -- --filter '*OrderCsv*'

# List what exists without running anything
dotnet run -c Release --project benchmarks/Orders.Benchmarks -- --list flat

# Quick, less precise run while iterating on an idea
dotnet run -c Release --project benchmarks/Orders.Benchmarks -- --filter '*OrderCsv*' --job short

# Compare runtimes before an upgrade: the benchmark project needs
# <TargetFrameworks>net8.0;net10.0</TargetFrameworks>, and every project it references
# must target both too (a multi-targeted library, or code compatible with net8.0).
# dotnet run needs -f to pick the host framework of a multi-targeted project.
dotnet run -c Release -f net10.0 --project benchmarks/Orders.Benchmarks -- --filter '*OrderCsv*' --runtimes net8.0 net10.0

# Results land in BenchmarkDotNet.Artifacts/results as Markdown, CSV and HTML; add JSON for tooling
dotnet run -c Release --project benchmarks/Orders.Benchmarks -- --filter '*OrderCsv*' --exporters json

Comparing JSON serialisation strategies

Input is built once in <code>[GlobalSetup]</code> from a field, and every method returns its bytes so nothing is optimised away. The third benchmark measures a pattern analyzer CA1869 flags: creating <code>JsonSerializerOptions</code> per call. Current System.Text.Json shares metadata between options instances with equivalent settings through a global cache, so the cost is the extra allocation and the equivalence lookup, plus a full metadata rebuild when settings differ or the cache, which holds a limited number of entries, has evicted them. Read the gap the benchmark shows rather than assuming it is large, and keep one static options instance either way.

using System.Text.Json;
using System.Text.Json.Serialization;
using BenchmarkDotNet.Attributes;

public sealed record OrderLine(string Sku, int Quantity, decimal UnitPrice);
public sealed record OrderDto(int Id, string CustomerId, DateTimeOffset PlacedAt, List<OrderLine> Lines);

[JsonSourceGenerationOptions(JsonSerializerDefaults.Web)]
[JsonSerializable(typeof(OrderDto))]
public partial class OrderJsonContext : JsonSerializerContext;

[MemoryDiagnoser]
public class OrderSerializationBenchmarks
{
    private static readonly JsonSerializerOptions Web = new(JsonSerializerDefaults.Web);
    private OrderDto _order = null!;

    [GlobalSetup]
    public void Setup() => _order = new OrderDto(
        1042, "CUST-0042", new DateTimeOffset(2026, 10, 1, 9, 30, 0, TimeSpan.Zero),
        Enumerable.Range(1, 20).Select(i => new OrderLine($"SKU-{i:D5}", i % 3 + 1, 9.99m + i)).ToList());

    [Benchmark(Baseline = true)]
    public byte[] Reflection() => JsonSerializer.SerializeToUtf8Bytes(_order, Web);

    [Benchmark]
    public byte[] SourceGenerated() => JsonSerializer.SerializeToUtf8Bytes(_order, OrderJsonContext.Default.OrderDto);

    // Anti-pattern flagged by CA1869: an options allocation plus a lookup in the shared
    // metadata cache on every call, and a full rebuild if that cache has evicted the entry.
    [Benchmark]
    public byte[] NewOptionsEveryCall() =>
        JsonSerializer.SerializeToUtf8Bytes(_order, new JsonSerializerOptions(JsonSerializerDefaults.Web));
}

Discussion

Be the first to comment on this lesson.

The parts

Reading the output

Mistakes that produce wrong numbers

Where benchmarks fit

Example

When to use it

More examples

The benchmark project

Running and filtering from the command line

Comparing JSON serialisation strategies

Related

Discussion