How to Use LINQ DistinctBy in C#

When you need to remove duplicate items from a collection, the standard LINQ Distinct method compares entire objects. If you have a list of Person objects and want to keep only the first occurrence of each Email, Distinct alone won't help unless you override Equals. The DistinctBy method, introduced in .NET 6, solves this directly: it accepts a key selector and returns objects that are unique according to that key. This article focuses on c# linq distinctby usage, its behavior, and when to choose it over alternatives.

Basic Syntax and First Example

DistinctBy is an extension method in the System.Linq namespace. It takes a Func<TSource, TKey> key selector and returns an IEnumerable<TSource> containing the first element from each group of objects that share the same key.

For example, given a list of orders:

public record Order(int Id, string Customer, decimal Amount);

List<Order> orders = new()
{
    new(1, "Alice", 100m),
    new(2, "Bob", 50m),
    new(3, "Alice", 200m),
    new(4, "Carol", 75m)
};

var distinctCustomers = orders.DistinctBy(o => o.Customer).ToList();

The result is the first order for Alice, the order for Bob, and the order for Carol—three orders total. The order for Alice with Id 2 is skipped because Alice already appeared.

The key selector can be any expression that returns a value used for equality comparison. The default equality comparer for the key type is used, so for strings it's ordinal, for integers it's numeric equality, and for custom types it uses the type's Equals implementation.

How Key Selection Works

The key selector determines what counts as a duplicate. If you need to compare by multiple properties, you can use an anonymous type or a tuple:

var uniqueByCustomerAndDate = orders
    .DistinctBy(o => new { o.Customer, o.Date })
    .ToList();

Anonymous types in C# implement value-based equality, so two anonymous objects with the same property values are considered equal. This lets you combine fields without writing a custom comparer.

Alternatively, you can use a tuple, which also has value semantics:

var uniqueByCustomerAndDate = orders
    .DistinctBy(o => (o.Customer, o.Date))
    .ToList();

Both approaches work, but anonymous types are often clearer when you have more than two fields.

Practical Example: Removing Duplicate Records

A common use case is cleaning up a data feed where duplicate records arrive with the same ID but different timestamps. You want the latest record for each ID. DistinctBy alone keeps the first, not the latest, so you need to combine it with sorting:

var latestBySensorId = sensorReadings
    .OrderByDescending(r => r.Timestamp)
    .DistinctBy(r => r.SensorId)
    .ToList();

This sorts the readings from newest to oldest, then DistinctBy keeps the first (newest) for each sensor ID. If you don't sort first, you'll keep the earliest occurrence in the original order, which may not be what you want.

This pattern is useful when processing event streams or log entries where the same logical entity appears multiple times.

Working with Custom Types and Comparers

When the key type is a custom class that doesn't override Equals, DistinctBy uses reference equality by default. This means two objects with identical field values won't be considered duplicates unless they are the same instance.

To control comparison, you can pass a custom IEqualityComparer<TKey> as the second argument:

public class CaseInsensitiveComparer : IEqualityComparer<string>
{
    public bool Equals(string? x, string? y) =>
        string.Equals(x, y, StringComparison.OrdinalIgnoreCase);

    public int GetHashCode(string obj) =>
        obj.ToUpperInvariant().GetHashCode();
}

var uniqueNames = people
    .DistinctBy(p => p.Name, new CaseInsensitiveComparer())
    .ToList();

The comparer is also used for hashing, so you must implement GetHashCode consistently with Equals. If the hash codes differ for values that should be equal, DistinctBy may not work correctly.

DistinctBy vs Distinct vs GroupBy

Distinct compares whole objects. DistinctBy compares a selected key. GroupBy groups objects by key and lets you process each group. Here's a comparison:

DistinctBy is equivalent to GroupBy(key).Select(g => g.First()) but with a simpler syntax and potentially better performance because it doesn't build full groups.

When you need more than the first element per key—for example, the sum or count of each group—use GroupBy instead.

Performance and Memory Considerations

DistinctBy works by maintaining an internal HashSet<TKey> of seen keys. As it iterates through the source, it checks each key against the set and only yields the element if the key hasn't been seen. This gives an average time complexity of O(n) for typical cases, but with extra memory usage proportional to the number of unique keys.

If the key type has expensive hash computation or the collection is extremely large, consider the cost of hashing. For most business objects this is negligible. For very large streams where memory is a concern, you could implement a streaming approach that writes to disk, but that's rarely necessary.

Another point: DistinctBy is lazy—it doesn't materialize the entire result until you call ToList() or iterate it. This means you can chain it with other LINQ operators and defer execution.

Where DistinctBy Is Available

DistinctBy is part of .NET 6 and later. If you're working with .NET Framework or earlier versions of .NET Core, you won't have it built-in. In those cases, you can write your own extension method or use a third-party library. A simple implementation is:

public static IEnumerable<TSource> DistinctBy<TSource, TKey>(
    this IEnumerable<TSource> source,
    Func<TSource, TKey> keySelector)
{
    var seen = new HashSet<TKey>();
    foreach (var item in source)
    {
        if (seen.Add(keySelector(item)))
            yield return item;
    }
}

This replicates the built-in behavior for older runtimes. It's a good learning exercise to understand how the method works internally.

Common Pitfalls

One common mistake is assuming DistinctBy keeps the last occurrence. It keeps the first. If you need the last, reverse the source or sort by a descending key before calling DistinctBy.

Another issue is using a mutable key. If the key object changes after it's added to the hash set, the set may become corrupted because the hash code changes. This is rare but can happen if you mutate a property that is part of the key after the sequence has been created. Avoid mutating key values while DistinctBy is still being iterated.

Also be careful with nullable keys. DistinctBy treats null as a valid key, so all elements with a null key will be collapsed into one. If that's not intended, filter out nulls first.

When to Use DistinctBy in Production

DistinctBy is a clean, readable way to deduplicate data based on a property without writing a custom comparer for every type. Use it when:

• You're on .NET 6 or later.
• You need the first object per key.
• The key is a simple property or a combination of properties.
• You don't need to aggregate the group.

For older frameworks, write a custom extension method or use a compatibility package. The implementation is small and well-understood, so the maintenance burden is low.

In scenarios where you need to control equality precisely—like case-insensitive matching or cultural-specific comparisons—pass a custom comparer instead of relying on the default.

DistinctBy fits naturally into a pipeline of LINQ operations. For example, you can filter, project, sort, and then deduplicate in a single expression, making the data transformation intent explicit. That readability is often worth more than micro-optimizations.

Finally, remember that DistinctBy preserves order: the result contains elements in the same order they appeared in the source, with duplicates skipped. This can be important when the source order carries meaning, such as a chronological event log.

If you're deduplicating a large dataset that doesn't fit in memory, consider a streaming approach or a database-side DISTINCT ON query instead of pulling everything into memory. DistinctBy is best suited for in-memory collections where the number of items is manageable.