Java Flight Recorder
Record what a running service is doing with JDK Flight Recorder, read the recording, and add your own events, with overhead low enough for production.
java -XX:StartFlightRecording=duration=2m,filename=rec.jfr -jar app.jar
jcmd <pid> JFR.dump name=<recording> filename=dump.jfr
jfr print --events jdk.GarbageCollection rec.jfrJDK Flight Recorder (JFR) is a profiler and event recorder built into the JVM. It records garbage collections, thread states, lock contention, file and socket I/O, allocation, exceptions and CPU samples of where the code spends its time. Its default settings are designed to cost around one percent of performance, so you can leave it running in production and dump the last few minutes when something goes wrong. It is part of OpenJDK, with no agent to install and no licence to buy.
Starting a recording
Start it with the JVM, or attach to a running process with jcmd:
java -XX:StartFlightRecording=duration=2m,filename=rec.jfr -jar app.jar
jcmd 4321 JFR.start name=incident settings=profile
jcmd 4321 JFR.dump name=incident filename=incident.jfr
jcmd 4321 JFR.stop name=incidentsettings=default is the low-overhead configuration; settings=profile gathers more detail, such as more frequent samples, for a short investigation. For an always-on recording, bound it with maxage or maxsize so it keeps only recent history.
Reading a recording
- The
jfrtool in the JDK prints, summarises and filters files:jfr summary,jfr print --events jdk.GarbageCollection,jfr print --json. jfr view, added in Java 21, prints ready-made tables such ashot-methods;jfr help viewlists them.- JDK Mission Control, a separate download, shows recordings graphically: flame graphs, GC timelines and lock contention.
What to look for in a slow service
- Hot methods in the CPU samples: where the time actually goes, often not where you guessed.
- Allocation by class and by stack trace: what feeds the garbage collector.
- GC pauses and heap after collection: whether pauses or a leak are the problem.
- Monitor waits and thread parks: requests stuck on a lock or an exhausted connection pool.
- Socket reads with long durations: a slow downstream service or database.
Your own events
Extend jdk.jfr.Event to record business operations, such as a checkout or a slow query, next to the JVM's own events. An event that is not enabled costs almost nothing, and enabled events go into the same timeline as GC and thread data. The event streaming API, RecordingStream, lets a service watch events while it runs, for example to export GC pauses as metrics.
Java 25 added more to JFR, including method timing and tracing events (jdk.MethodTiming and jdk.MethodTrace) and an experimental CPU-time profiler on Linux.
Example
# Always-on recording of the last 30 minutes, written out on shutdown
java -XX:StartFlightRecording=name=continuous,maxage=30m,dumponexit=true,filename=/tmp/app-exit.jfr \
-jar app.jar
# Something is slow right now: find the process and save the recent history
jcmd -l
jcmd 4321 JFR.dump name=continuous filename=/tmp/slow-checkout.jfr
# Read it
jfr summary /tmp/slow-checkout.jfr
jfr view hot-methods /tmp/slow-checkout.jfr
jfr print --events jdk.GarbageCollection /tmp/slow-checkout.jfrWhen to use it
- Dump the last ten minutes of a continuous recording when checkout latency spikes and find the lock that request threads were waiting on.
- Record a custom event around each slow database query so its duration lines up with GC pauses in the same timeline.
- Stream jdk.GarbageCollection events into the service's metrics so dashboards show pause times without parsing GC logs.
More examples
A custom event for a business operation
Each checkout appears in the recording with its duration, beside the GC pauses and lock waits that happened at the same moment. Custom events are enabled by default; annotate the class with @Enabled(false) to make one opt-in.
import jdk.jfr.Category;
import jdk.jfr.Event;
import jdk.jfr.Label;
import jdk.jfr.Name;
@Name("shop.Checkout")
@Label("Checkout")
@Category("Shop")
class CheckoutEvent extends Event {
@Label("Order id")
String orderId;
@Label("Line items")
int items;
}
class CheckoutService {
void checkout(String orderId, int items) {
CheckoutEvent event = new CheckoutEvent();
event.begin();
// ... reserve stock, charge the card, save the order
event.orderId = orderId;
event.items = items;
event.commit(); // records the duration since begin()
}
}Streaming GC events while the service runs
RecordingStream delivers events to callbacks within about a second, which suits pushing GC pauses or CPU load into your metrics system. Replace the println calls with your metrics client in a real service.
import java.time.Duration;
import jdk.jfr.consumer.RecordingStream;
class JvmEventsReporter {
RecordingStream start() {
RecordingStream stream = new RecordingStream();
stream.enable("jdk.GarbageCollection");
stream.enable("jdk.CPULoad").withPeriod(Duration.ofSeconds(10));
stream.onEvent("jdk.GarbageCollection", event ->
System.out.println("gc " + event.getString("name")
+ " longest pause " + event.getDuration("longestPause").toMillis() + " ms"));
stream.onEvent("jdk.CPULoad", event ->
System.out.println("machine cpu " + event.getFloat("machineTotal")));
stream.startAsync(); // runs on its own thread
return stream; // call close() on shutdown
}
}Recording in a container
maxage and maxsize bound the disk the recording uses, so it can run for weeks. jcmd needs a JDK in the image, or a jlink image that includes the jdk.jcmd and jdk.jfr modules.
# In the image's entrypoint: keep a rolling recording, bounded in size
java -XX:StartFlightRecording=name=continuous,maxage=15m,maxsize=200m,dumponexit=true,filename=/tmp/exit.jfr \
-jar /app/app.jar
# From your machine, while the pod is running (process id 1 in the container)
kubectl exec my-service-7c9d -- jcmd 1 JFR.dump name=continuous filename=/tmp/now.jfr
kubectl cp my-service-7c9d:/tmp/now.jfr ./now.jfr
Discussion