Remove Duplicates from a List in C#

When you have a List<T> that contains duplicate values, you need to decide how to remove them while preserving order, handling custom types, and keeping performance acceptable. The c# list remove duplicates problem has several solutions, each with different tradeoffs. The right choice depends on whether you care about element order, whether your type is a simple value type or a complex object, and how large the list is.

The Problem: Duplicates in a List<T>

A List<T> allows duplicates by design. It is an ordered collection that stores elements in the order you add them, and it does not enforce uniqueness. If you receive data from a file, a database, or a user input, duplicates may appear. Removing them is a common data-cleaning step, but the approach you choose affects the result's order and the amount of memory and CPU time used.

Consider a simple list of integers:

List<int> numbers = new List<int> { 1, 2, 3, 2, 4, 1, 5 };

You want a list containing 1, 2, 3, 4, 5 in that order. The simplest way is to use LINQ's Distinct() method.

Using LINQ's Distinct() to Remove Duplicates

The Distinct() method is part of LINQ and returns an IEnumerable<T> containing only the first occurrence of each element. It uses the default equality comparer for the type, which for value types compares the values, and for reference types compares references unless the type overrides Equals and GetHashCode.

List<int> uniqueNumbers = numbers.Distinct().ToList();

This preserves the original order: the first occurrence of each value remains in its original position. For a list of integers, this is straightforward and readable. The method is lazy, meaning it does not execute until you iterate over the result or call ToList(). In practice, you almost always materialize the result back into a list.

For simple value types, Distinct() is the clearest and most idiomatic solution. It works with any IEnumerable<T>, so you can use it with arrays, List<T>, or other collections. The main limitation is that it uses the default equality comparison, which may not be what you need for custom objects.

Preserving Order with HashSet<T>

If you need to remove duplicates while preserving the order of the first occurrences, a HashSet<T> is an efficient alternative. A HashSet<T> is a set that stores unique elements and provides O(1) lookup on average. You can iterate over the original list, add each element to a HashSet, and only add to a new list if the element is new.

List<int> uniqueList = new List<int>();
HashSet<int> seen = new HashSet<int>();

foreach (int number in numbers)
{
    if (seen.Add(number))
    {
        uniqueList.Add(number);
    }
}

The Add method returns true if the element was not already present, and false if it was. This approach gives you explicit control over the process and avoids the overhead of LINQ's internal state machine, though the difference is negligible for most list sizes. It also lets you use a custom equality comparer by passing it to the HashSet constructor.

HashSet<int> seen = new HashSet<int>(customComparer);

This is particularly useful when you want to compare objects based on a subset of their properties.

Using a Custom Equality Comparer for Complex Types

For a list of custom objects, the default reference equality is rarely what you want. Suppose you have a Person class with Id and Name properties, and you want to remove duplicates based on Id.

public class Person
{
    public int Id { get; set; }
    public string Name { get; set; }
}

Calling Distinct() on a List<Person> will not remove objects with the same Id unless Person overrides Equals and GetHashCode. To avoid modifying the class, you can implement an IEqualityComparer<Person>.

public class PersonIdComparer : IEqualityComparer<Person>
{
    public bool Equals(Person x, Person y)
    {
        return x.Id == y.Id;
    }

    public int GetHashCode(Person obj)
    {
        return obj.Id.GetHashCode();
    }
}

Then pass it to Distinct():

List<Person> uniquePeople = people.Distinct(new PersonIdComparer()).ToList();

This keeps the original order and removes objects with duplicate Id values. The same comparer can be used with a HashSet in the manual loop, giving you the same deduplication logic without relying on LINQ.

When you implement a custom comparer, you must ensure that GetHashCode is consistent with Equals. Two objects that are equal must return the same hash code. If they do not, the HashSet or Distinct will not work correctly.

Performance and Memory Tradeoffs

Both Distinct() and the HashSet approach have similar time complexity: O(n) on average, because each element is hashed once and compared against the set. The memory usage is also similar, as both need to store the set of seen elements. The main difference is that Distinct() creates an internal set and returns an iterator, while the manual loop lets you control the output collection.

For very large lists, the hash-based approach is usually faster than a naive O(n²) solution that compares each element with every previous element. You should avoid using List.Contains() in a loop for large lists because Contains() is O(n) and the overall complexity becomes O(n²).

// Inefficient for large lists
List<int> result = new List<int>();
foreach (int number in numbers)
{
    if (!result.Contains(number))
    {
        result.Add(number);
    }
}

This works but scales poorly. For a list of 100,000 elements, the nested comparisons become a bottleneck. The HashSet or Distinct approach keeps the operation linear.

If you do not need to preserve order, you can use a HashSet directly as the output collection:

HashSet<int> uniqueSet = new HashSet<int>(numbers);

This is the fastest way to remove duplicates, but you lose the original order. If order does not matter, this is the most concise and efficient solution.

When to Choose Which Approach

Choosing the right method depends on your specific requirements.

If you are working with a large list and need to keep the original order, the manual HashSet loop is slightly more efficient because it avoids the LINQ iterator overhead, but the difference is often negligible. Distinct() is more readable and is the standard choice for most codebases.

If you need to remove duplicates from an existing list without creating a new one, you can use RemoveAll with a HashSet:

HashSet<int> seen = new HashSet<int>();
numbers.RemoveAll(n => !seen.Add(n));

This modifies the original list and preserves the order of the remaining elements. The lambda returns true for elements that are already in the set, so RemoveAll removes them.

Edge Cases: Nulls, Case Sensitivity, and Reference Types

When the list contains null values, the default equality comparer treats all null references as equal. Both Distinct() and HashSet will keep only the first null. If you are using a custom comparer, you need to handle null explicitly in the Equals and GetHashCode methods to avoid NullReferenceException.

For string lists, the default comparer is case-sensitive. If you want to remove duplicates regardless of case, you can use StringComparer.OrdinalIgnoreCase:

List<string> words = new List<string> { "apple", "Apple", "banana", "BANANA" };
List<string> uniqueWords = words.Distinct(StringComparer.OrdinalIgnoreCase).ToList();

This yields apple and banana, preserving the first occurrence's casing. The same comparer can be passed to a HashSet constructor.

For reference types that do not override Equals, the default behavior is reference equality. Two objects with identical property values are considered different unless you provide a custom comparer. This is a common source of confusion, so always check whether your type implements value equality before relying on Distinct().

When you implement a custom comparer, be careful with mutable properties. If an object's hash code changes after it is added to a HashSet, the set will no longer find it correctly. For deduplication, it is safer to use immutable properties or a snapshot of the values used for comparison.

Finally, consider the size of the list. For tiny lists (fewer than a dozen elements), the performance difference between O(n) and O(n²) is irrelevant, and readability matters more. For production code that processes thousands or millions of records, prefer the hash-based approaches and avoid Contains() in a loop.

Choosing the right deduplication strategy is a balance between readability, order preservation, and performance. Distinct() is the most expressive for most scenarios, while a manual HashSet gives you finer control over the process. Custom comparers extend both approaches to complex types, and understanding how the default equality comparer works prevents subtle bugs.