distinct()

Remove duplicate elements by equals and hashCode while keeping the first occurrence, and de-duplicate by a key when whole objects differ.

Signature
Stream<T> distinct()
  • Added in Java Java 8

Return value

Stream<T>A new stream with duplicates removed according to Object.equals. For ordered streams the first occurrence of each element is kept and encounter order is preserved.

Errors

Nothing directly. Exceptions from equals or hashCode surface when the terminal operation runs.

See also

sorted()count()Collectors.toSet()Collectors.toMap()Sorting, distinct and limit

distinct() drops elements that are equal to one already seen. Equality is decided by equals and hashCode, the same rules a HashSet uses. It takes no arguments, which is both its strength and its main limitation.

Order is kept

On an ordered stream (a List, an array, Stream.of) distinct keeps the first occurrence of each value and preserves encounter order. That makes it a good fit when order matters, for example de-duplicating tags while keeping the order the user typed them. new HashSet<>(list) removes duplicates too but loses the order; new LinkedHashSet<>(list) keeps it.

Equality is the whole story

  • Records get equals and hashCode from all their components, so two records with the same values are duplicates.
  • Ordinary classes without overridden equals use identity: two new Customer("c1") objects are different, and distinct keeps both.
  • Strings compare case-sensitively. Normalise first, for example map(String::toLowerCase) for email addresses, then call distinct.
  • Mutable keys: if an element changes after being seen, its hash changes and the de-duplication becomes unreliable.

Distinct by a key

There is no distinctBy(keyExtractor) on Stream. When you want one element per user ID, per SKU or per email, collect into a map keyed by that field and keep the first value:

events.stream()
    .collect(Collectors.toMap(Event::userId, e -> e, (first, later) -> first, LinkedHashMap::new))
    .values();

The merge function decides which duplicate wins and LinkedHashMap keeps the order. Swap in (first, later) -> later to keep the most recent.

Cost

distinct is stateful: it remembers every distinct element it has seen, so memory grows with the number of unique values. That is fine for a page of results; on a stream of millions of rows, de-duplicate in the query with SELECT DISTINCT or a unique constraint. On parallel ordered streams, keeping first-occurrence order adds buffering; if order does not matter, unordered().distinct() is cheaper.

Example

Try it yourself
Loading editor…
Press Run to execute the code.

When to use it

  • Remove repeated tags from a blog post request while keeping the order the author typed them.
  • Send one notification per recipient after normalising email addresses to lower case.
  • Build the list of distinct product IDs in a batch of orders before loading the products in one query.

Frequently asked questions

How do I remove duplicates from a list with streams?

list.stream().distinct().toList() keeps the first occurrence of each element in the original order. Equality is decided by equals and hashCode, so make sure your element type implements them; records do automatically.

How do I make distinct work on one property of an object?

Stream has no distinctBy. Collect into a map keyed by the property with a merge function that keeps the first value, using LinkedHashMap to keep order: collect(Collectors.toMap(T::key, t -> t, (a, b) -> a, LinkedHashMap::new)).values().

Why does distinct not remove my duplicate objects?

Your class probably does not override equals and hashCode, so each object is only equal to itself. Override both methods, or turn the class into a record, which generates them from its components.

More examples

Classes without equals are never duplicates

The class inherits identity-based equals from Object, so two objects with the same ID are still different. The record generates equals and hashCode from its components, so distinct treats the two as one.

Example · java
class Customer {
    final String id;

    Customer(String id) {
        this.id = id;
    }
}

record CustomerRef(String id) {}

void main() {
    long classes = Stream.of(new Customer("c1"), new Customer("c1")).distinct().count();
    IO.println(classes);

    long records = Stream.of(new CustomerRef("c1"), new CustomerRef("c1")).distinct().count();
    IO.println(records);
}
// Output:
// 2
// 1

One element per key, keeping the first

The two u1 events are not equal as whole records, so distinct would keep both. Keying a LinkedHashMap by userId with a keep-first merge function gives distinct-by-key and preserves the original order.

Example · java
record Event(String userId, String action) {}

void main() {
    List<Event> events = List.of(
        new Event("u1", "login"),
        new Event("u2", "login"),
        new Event("u1", "logout"),
        new Event("u3", "login"));

    Collection<Event> firstPerUser = events.stream()
        .collect(Collectors.toMap(Event::userId, e -> e, (first, later) -> first, LinkedHashMap::new))
        .values();
    IO.println(firstPerUser);
}
// Output:
// [Event[userId=u1, action=login], Event[userId=u2, action=login], Event[userId=u3, action=login]]

distinct keeps order, a TreeSet sorts

distinct and LinkedHashSet both keep first-seen order, while a TreeSet sorts. Choose based on whether the caller cares about the original order, such as the order items were added to a basket.

Example · java
void main() {
    List<Integer> productIds = List.of(7, 3, 7, 1, 3, 9);

    IO.println(productIds.stream().distinct().toList());
    IO.println(new TreeSet<>(productIds));
    IO.println(new LinkedHashSet<>(productIds));
    IO.println(productIds.stream().distinct().count());
}
// Output:
// [7, 3, 1, 9]
// [1, 3, 7, 9]
// [7, 3, 1, 9]
// 4

Related

java sorting distinct and limitstream sortedcollectors tosetcollectors tomapjava set and hashset

Discussion

  • Be the first to comment on this lesson.