Using Java Stream distinct() for Deduplication

java stream distinct requires a clear understanding of the core syntax, runtime behavior, and practical implementation patterns demonstrated in the examples below.

The distinct() method is a stateful intermediate operation available on java.util.stream.Stream. When applied, it returns a stream consisting of the distinct elements of the source stream, where distinctness is determined by Object.equals(Object). For ordered streams, the first occurrence of each element is preserved; for unordered streams, no stability guarantee exists.

List<Integer> numbers = List.of(3, 1, 4, 1, 5, 9, 2, 6, 5, 3);
List<Integer> unique = numbers.stream()
        .distinct()
        .toList();
// [3, 1, 4, 5, 9, 2, 6]

The operation is stateful because it must remember every element it has already seen to decide whether a subsequent element should be passed downstream. This has direct consequences for memory usage and parallel execution, both covered later.

How distinct() Determines Equality

The distinct() operation relies on the equals() and hashCode() contracts. When an element arrives, the stream implementation stores it in an internal set-like structure. A new element is considered a duplicate if the set already contains an equal element according to equals().

This means the behavior of distinct() is only as correct as the equals() and hashCode() implementations of the element type. For primitive wrappers like Integer and String, the default implementations are value-based and work as expected. For custom classes, you must override both methods correctly.

public class Order {
    private final String orderId;
    private final String customer;

    // constructor, getters

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (!(o instanceof Order other)) return false;
        return orderId.equals(other.orderId);
    }

    @Override
    public int hashCode() {
        return orderId.hashCode();
    }
}

With this implementation, two orders with the same orderId are considered equal, so distinct() will collapse them. If you override only equals() without hashCode(), the internal set may fail to detect duplicates because equal objects can land in different hash buckets, and distinct() will return elements that should have been removed.

Deduplicating Custom Objects

When working with domain objects, the default identity-based equality is almost never what you want. Consider a stream of Order objects where the same order appears multiple times because it was loaded from different sources.

List<Order> orders = orderService.loadAll();
List<Order> uniqueOrders = orders.stream()
        .distinct()
        .toList();

The result depends entirely on the equals() implementation of Order. If Order does not override equals(), every instance is distinct, and the stream returns all elements unchanged. This is a common source of confusion: distinct() appears to do nothing, but the real problem is missing equality semantics.

When you cannot modify the element class, an alternative is to map to a key and deduplicate manually:

List<Order> uniqueOrders = orders.stream()
        .collect(Collectors.toMap(
                Order::getOrderId,
                Function.identity(),
                (first, ignored) -> first,
                LinkedHashMap::new
        ))
        .values()
        .stream()
        .toList();

This approach keeps the first occurrence of each order ID and preserves encounter order through LinkedHashMap. It is more verbose than distinct(), but it works when the element class does not define the equality semantics you need.

Memory and Runtime Cost

Because distinct() must track every unique element seen so far, its memory footprint grows linearly with the number of distinct elements in the stream. For small collections this is irrelevant, but for streams that process millions of records, the internal set can become a significant memory consumer.

The runtime cost is dominated by hashCode() and equals() calls on each element. If these methods are expensive, the deduplication step becomes a bottleneck. For example, a hashCode() that performs string concatenation or a database lookup will slow down the entire pipeline.

There is no way to bound the memory usage of distinct() without changing the algorithm. If the stream is known to contain a limited number of unique values, distinct() is fine. If the number of unique values is unbounded, consider whether the stream can be sorted first and deduplicated with a windowed comparison, which uses constant memory but requires a sorted source.

Parallel Streams and distinct()

When a stream is parallel, distinct() must coordinate state across threads. The JDK implementation uses a concurrent set-like structure, which adds synchronization overhead. This means distinct() on a parallel stream is typically more expensive per element than on a sequential stream, and the benefit of parallelism can be reduced or negated.

For ordered parallel streams, the implementation must also preserve encounter order of the first occurrences, which requires additional bookkeeping. If order does not matter, you can call unordered() before distinct() to relax this requirement:

List<Integer> unique = numbers.parallelStream()
        .unordered()
        .distinct()
        .toList();

This allows the implementation to use a cheaper concurrent deduplication strategy. Whether this actually improves throughput depends on the size of the stream and the cost of hashCode() and equals(). For small streams, the synchronization overhead dominates and parallel execution is usually slower than sequential.

Common Mistakes and Edge Cases

One frequent mistake is assuming distinct() preserves the order of the last occurrence. It does not; for ordered streams it always keeps the first occurrence. If you need the last occurrence, you must reverse the stream, apply distinct(), and reverse again, or use a toMap collector with a merge function that keeps the later value.

Another edge case involves null elements. The distinct() operation handles null correctly: the first null is kept, and subsequent nulls are removed. This is consistent with how HashSet treats null.

List<String> values = Arrays.asList("a", null, "b", null, "a");
List<String> unique = values.stream().distinct().toList();
// ["a", null, "b"]

Floating-point values follow the Double.equals() contract, so 0.0 and -0.0 are considered different, while NaN is considered equal to itself. This differs from == semantics and can surprise developers who expect numeric comparison.

When distinct() Is the Wrong Tool

distinct() is the right choice when you need to remove duplicates from a stream while preserving the element type and encounter order. But there are cases where a different approach is better.

If you only need to know whether duplicates exist, distinct().count() compared against the original count is wasteful; a short-circuiting approach that stops at the first duplicate is more efficient, though the standard library does not provide one directly.

If you need deduplication by a key that is not the element's natural equality, distinct() cannot help directly. You must either map to a key and back, or use toMap with a merge function as shown earlier.

For very large streams where memory is a concern, consider sorting the stream and removing adjacent duplicates:

List<Integer> unique = numbers.stream()
        .sorted()
        .reduce(new ArrayList<Integer>(), (acc, n) -> {
            if (acc.isEmpty() || !acc.get(acc.size() - 1).equals(n)) {
                acc.add(n);
            }
            return acc;
        }, (left, right) -> left);

This uses constant extra memory but requires O(n log n) time for the sort, so it is only beneficial when the number of distinct elements is large enough that the set-based approach would exhaust memory. For typical in-memory collections, distinct() remains the simplest and most readable option.