Remove Duplicates From List In C#

Introduction

When working with collections in C#, developers often encounter the need to remove duplicates from list objects. Duplicate entries can waste memory, skew data analysis, and cause unexpected behavior in algorithms. Whether you are cleaning data imported from external sources, preparing a set of unique identifiers, or simply want to ensure each element appears only once, C# provides several built‑in ways to achieve this goal. In this article we will explore the most common techniques—using LINQ’s Distinct method, employing a HashSet, iterating with List.RemoveAll, and even custom extensions—while discussing performance considerations and best practices. By the end of the guide you’ll have a clear understanding of how to remove duplicates from list in C# efficiently and maintain clean, readable code.

Steps to Remove Duplicates

1. Using LINQ’s Distinct Method

The simplest and most readable approach is to make use of LINQ’s Distinct operator. It returns a sequence that contains only unique elements, preserving the order of first occurrence.

using System.Linq;
using System.Collections.Generic;

List source = new List { 1, 2, 3, 2, 4, 1, 5 };
var unique = source.Distinct().ToList();

• How it works: Distinct internally uses a HashSet to track seen items, automatically filtering out duplicates.
• Pros: Concise, readable, works with any type that implements IEquatable or overrides GetHashCode and Equals.
• Cons: It creates a new collection; if you need to modify the original list in place, you’ll need an extra step.

2. Leveraging a HashSet for High Performance

When performance is critical, converting the list to a HashSet can be faster, especially for large datasets. HashSet guarantees uniqueness by design Nothing fancy..

HashSet hashSet = new HashSet(source);
List uniqueList = new List(hashSet);

• Why it’s fast: Insertion and lookup in a HashSet are O(1) on average, making duplicate detection nearly instantaneous.
• Ordering: HashSet does not preserve insertion order, so if you need the original order, consider using LINQ’s OrderBy after the conversion or switch to the Distinct approach.

3. In‑Place Removal with List.RemoveAll

If you prefer to mutate the existing list without allocating a new one, List.RemoveAll combined with a predicate is a viable option. You can create a temporary HashSet of duplicates and then remove them.

HashSet seen = new HashSet();
for (int i = 0; i < source.Count; i++)
{
    if (!seen.Add(source[i]))
    {
        source.RemoveAt(i--); // remove duplicate and adjust index
    }
    else
    {
        seen.Add(source[i]);
    }
}

• Complexity: This approach runs in O(n²) in the worst case because RemoveAt shifts subsequent elements. For small lists it’s fine, but for larger collections the overhead can be significant.
• Alternative: Use List.RemoveAll with a HashSet of duplicates to avoid repeated shifting:

HashSet duplicates = new HashSet();
HashSet uniqueTracker = new HashSet();
foreach (var item in source)
{
    if (!uniqueTracker.Add(item))
    {
        duplicates.Add(item);
    }
}
source.RemoveAll(item => duplicates.Contains(item));

4. Custom Extension Method for Maximum Flexibility

Sometimes you need more control—perhaps you want to keep the first occurrence of each duplicate, or you want to apply a custom equality comparer. Writing a small extension method can encapsulate this logic.

public static class ListExtensions
{
    public static void RemoveDuplicates(this List list) where T : IEquatable
    {
        var seen = new HashSet();
        for (int i = list.Count - 1; i >= 0; i--)
        {
            if (!seen.Add(list[i]))
            {
                list.RemoveAt(i);
            }
        }
    }
}

• Usage: myList.RemoveDuplicates();
• Benefits: Works in‑place, respects custom equality, and is easy to reuse across projects.

Scientific Explanation

How Distinct Works Internally

Distinct is implemented in the System.Linq assembly and relies on a HashSet under the hood. When iterating over the source sequence, each element is passed to the HashSet’s Add method. If Add returns false, the element is a duplicate and is skipped. This guarantees O(n) time complexity for duplicate detection, assuming hash collisions are minimal Took long enough..

HashSet Collision Handling

A HashSet uses a hash table where each bucket may contain multiple entries (linked list or balanced tree). In .NET, the default HashSet implementation uses linked lists for collision resolution. While this adds a small constant factor, the overall performance remains linear for typical data sizes. Choosing an appropriate IEqualityComparer can reduce collisions for custom types (e.g., case‑insensitive string comparison) That's the whole idea..

Performance Comparison

Empirical tests on lists of 100,000 integers show:

The numbers illustrate that LINQ Distinct and HashSet conversion are generally the fastest, while in‑place removal can be slower due to element shifting Worth knowing..

Frequently Asked Questions

Q1: Does Distinct preserve the order of elements?

A: Yes, Distinct preserves the order of first occurrence. If you need a different ordering, apply OrderBy after the operation Most people skip this — try not to. Nothing fancy..

Q2: Can I remove duplicates from a List<string> case‑insensitively?

**

Q2: You can perform a case‑insensitive deduplication by supplying a custom comparer to the extension method (or to LINQ’s Distinct). For example:

using System.Globalization;

myList.RemoveDuplicates(StringComparer.OrdinalIgnoreCase);

If you prefer to keep the original method signature, add an overload that accepts an IEqualityComparer<T>:

public static void RemoveDuplicates(this List list, IEqualityComparer comparer) where T : IEquatable
{
    var seen = new HashSet(comparer);
    for (int i = list.Count - 1; i >= 0; i--)
    {
        if (!seen.Add(list[i]))
        {
            list.RemoveAt(i);
        }
    }
}

Now myList.RemoveDuplicates(StringComparer.OrdinalIgnoreCase); will treat "Apple" and "apple" as the same element, removing the later occurrence while preserving the first one.

Additional Frequently Ask

Additional Frequently Asked Questions

Q3: Does this work on arrays or other collections?
A: The in‑place extension method targets List<T> because it mutates the existing collection. For arrays, IEnumerable<T>, or read-only collections, use LINQ instead:

var uniqueArray = array.Distinct().ToArray();
var uniqueEnumerable = enumerable.Distinct().ToList();

Arrays cannot be resized, so you must create a new array rather than removing elements in place.

Q4: How are null values handled?
A: HashSet<T> treats null as a valid element for reference types and nullable value types. The first null encountered is kept; any subsequent null values are considered duplicates and skipped But it adds up..

Q5: Is the operation thread‑safe?
A: No. Neither List<T> nor HashSet<T> is safe for concurrent reads and writes. If multiple threads access the collection, protect the deduplication logic with a lock or use a thread‑

safe collection, or ensure exclusive access for the duration of the operation. Concurrent reads and writes can corrupt the HashSet<T>’s internal state or throw InvalidOperationException when the list is modified while being enumerated No workaround needed..

Q6: What is the best approach for large lists?
A: For very large lists, the custom reverse‑loop extension backed by a HashSet<T> is usually the best balance of speed and memory. It avoids allocating a second list and performs a single pass. LINQ’s Distinct().ToList() is concise and fast, but it creates a new collection, which may be prohibitive under tight memory constraints. In‑place RemoveAll should generally be avoided for large datasets because each removal shifts all subsequent elements, turning an O(n) operation into O(n²) in the worst case.

Conclusion

Removing duplicates from a List<T> depends on your priorities. So toList()is hard to beat. But if readability and brevity matter most,Distinct(). Always consider element ordering, null handling, custom comparers, and thread safety before choosing a method. If you must mutate the original list and minimize allocations, the reverse‑loop HashSet extension offers strong performance with a small memory footprint. By matching the technique to your data size and constraints, you can keep your collections clean without sacrificing speed or clarity.